Author: Bill

[Troubleshooting] Error: could not find system library ‘openssl’ required by the ‘openssl-sys’ crate

When using reqwest crate and building docker images with Dockerfile mentioned in this article, you will encounter openssl issues due to missing packages.

Error: could not find system library 'openssl' required by the 'openssl-sys' crate

It turns out that below line is the culprit:

FROM docker.io/rust:1-slim-bookworm AS build

#############################################

rust:1-slim-bookworm misses lots of necessary package to run the application. It is suggested in official Rust docker image document that we should avoid using slim image.

Simple but ugly solution

Well, the simplest approach is using a minimal Dockerfile which will always work:

FROM rust:bookworm

COPY . .

RUN cargo build --release

EXPOSE 8080

CMD ./target/release/your_package

However, it will generate a pretty large image which can take gigabytes of storage.

The better one

We can still use multi stage build to generate an optimal image with a little tweak (using default rust image version rust:bookworm):

FROM docker.io/rust:bookworm AS build

## cargo package name: customize here or provide via --build-arg
ARG pkg=hello-world

WORKDIR /build

COPY . .

RUN --mount=type=cache,target=/build/target \
    --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/usr/local/cargo/git \
    set -eux; \
    cargo build --release; \
    objcopy --compress-debug-sections target/release/$pkg ./main

################################################################################

FROM docker.io/debian:bookworm-slim

WORKDIR /app

## copy the main binary
## add more files below if needed
COPY --from=build /build/main ./

EXPOSE 8080

CMD ./main

Now it builds fine, but you will get a runtime error when running reqwest:

./main: error while loading shared libraries: libssl.so.3: cannot open shared object file: No such file or directory

Furthermore, after installing correct packages either vendored or automatic, you will still see error when sending https requests:

hyper_util::client::legacy::Error(Connect, Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: "SSL routines", function: "tls_post_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 2092 }]))) }, X509VerifyResult { code: 20, error: "unable to get local issuer certificate" }))

To solve these issues, I separately wrote an article for detailed explanation and solutions.

December 20, 2024

[Troubleshooting] libssl.so.3 and certificate Error Running Rust/Reqwest Under Debian Image

If you see below errors while running Rust app using Docker, this article is right for you:

error while loading shared libraries: libssl.so.3: cannot open shared object file: No such file or directory

hyper_util::client::legacy::Error(Connect, Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: "SSL routines", function: "tls_post_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 2092 }]))) }, X509VerifyResult { code: 20, error: "unable to get local issuer certificate" }))

Anatomy

Reqwest document mentioned about requirements when running under Linux OS, where you need OpenSSL installed.

To supply OpenSSL, Rust official document provided 2 ways:

Vendored OpenSSL

Add below dependency to Cargo.toml.

[dependencies]
openssl = { version = "0.10", features = ["vendored"] }

Automatic OpenSSL

Add below line to Dockerfile at stage of running app.

RUN apt-get update && apt-get install -y pkg-config libssl-dev

Either approach works fine and you can run the app successfully. However, when sending https requests, you will get below error:

unable to get local issuer certificate

A Debian image issue

This thread gives a good idea of what happens here. Essentially, the debian official image does not have ca certification package installed. To solve the issue, simply install the package in Dockerfile:

RUN apt-get update && apt-get install -y pkg-config libssl-dev && apt install -y ca-certificates
# or below if using vendored OpenSSL
# RUN apt-get update && apt install -y ca-certificates

Final Dockerfile

To build rust + actix-web + reqwest, below is what works for me:

[package]
name = "hello-world"
version = "0.1.0"
edition = "2021"

[dependencies]
actix-web = "4"
reqwest = "0.12"
serde = { version = "1.0", features = ["derive"] }
openssl = { version = "0.10", features = ["vendored"] }

FROM rust:bookworm AS build

## cargo package name: customize here or provide via --build-arg
ARG pkg=hello-world

WORKDIR /build

COPY . .

RUN --mount=type=cache,target=/build/target \
    --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/usr/local/cargo/git \
    set -eux; \
    cargo build --release; \
    objcopy --compress-debug-sections target/release/$pkg ./main

################################################################################

FROM docker.io/debian:bookworm-slim

WORKDIR /app

## copy the main binary
COPY --from=build /build/main ./

EXPOSE 8080

RUN apt-get update && apt install -y ca-certificates

CMD ./main

December 20, 2024

[Troubleshooting] Error during `gcloud app deploy` for GAE app: “Failed to create cloud build: invalid bucket”
When I was trying to run app with Google App Engine, I followed Hello World example step by step, and no surprise that it didn’t work. You will see below error:
```
ERROR: (gcloud.app.deploy) Error Response: [13] Failed to create cloud build: com.google.net.rpc3.client.RpcClientException: <eye3 title='/ArgoAdminNoCloudAudit.CreateBuild, FAILED_PRECONDITION'/> APPLICATION_ERROR;google.devtools.cloudbuild.v1/ArgoAdminNoCloudAudit.CreateBuild;invalid bucket "staging.avid-shape-445101-a4.appspot.com"; service account avid-shape-445101-a4@appspot.gserviceaccount.com does not have access to the bucket;AppErrorCode=9;StartTimeMs=1734486112961;unknown;ResFormat=uncompressed;ServerTimeSec=0.921583338;LogBytes=256;Non-FailFast;EndUserCredsRequested;EffSecLevel=privacy_and_integrity;ReqFormat=uncompressed;ReqID=4b101eaa0d045332;GlobalID=0;Server=[2002:a05:6670:1585:b0:a5e:1b4f:746d]:4001.
```
Same error has been posted in many places:
- https://www.googlecloudcommunity.com/gc/Serverless/Failed-to-create-cloud-build-no-access-to-bucket/m-p/758310
- https://stackoverflow.com/questions/78742739/error-during-gcloud-app-deploy-for-gae-app-failed-to-create-cloud-build-inv
- https://www.googlecloudcommunity.com/gc/Serverless/Error-during-gcloud-app-deploy-for-GAE-app-quot-Failed-to-create/m-p/778778
- etc.
Based on this doc, starting May 3, 2024, iam.automaticIamGrantsForDefaultServiceAccounts is disabled by default, which caused all of these issues.

Thanks to this answer, it gives me strong lead to the solution, although there are still some extra steps (might be from undocumented recent changes from Google Cloud).

Step by step solution

Give access to storage bucket

There are 2 ways of doing it.

Using Console UI
1. Go to https://console.cloud.google.com/storage/browser, select staging.PROJECT_ID.appspot.com and go to Permissions tab.
2. Click on GRANT ACCESS button.
3. Enter PROJECT_ID@appspot.gserviceaccount.com as New principals.
4. Enter Storage Admin as new Role.
5. Save setting.
Using CLI

Simply using below command:
```
gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:PROJECT_ID@appspot.gserviceaccount.com" --role="roles/storage.admin"
```
Now let’s deploy again:
```
gcloud app deploy
```
No surprise, you will see below error:
```
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build f490b199-1e2e-4fbb-89c7-e0caae57f7bc status: FAILURE
An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=us-central1/f490b199-1e2e-4fbb-89c7-e0caae57f7bc?project=519995341679
Full build logs: https://console.cloud.google.com/cloud-build/builds;region=us-central1/f490b199-1e2e-4fbb-89c7-e0caae57f7bc?project=519995341679
```
Its much better now, at least we have a log. Let’s go to the link provided.

You will very likely see above warning and empty logs.

Grant access to writing logs

Run below command to add role as suggested:
```
gcloud projects add-iam-policy-binding avid-shape-445101-a4 --member="serviceAccount:avid-shape-445101-a4@appspot.gserviceaccount.com" --role="roles/logging.logWriter"
```
Now let’s run deploy again (just to to see actual building logs):
```
gcloud app deploy
```
You will see same error but in the build log, you will see why exactly the build failed:
```
ERROR: failed to create image cache: accessing cache image "us.gcr.io/avid-shape-445101-a4/app-engine-tmp/build-cache/default/ttl-7d:latest": connect to repo store "us.gcr.io/avid-shape-445101-a4/app-engine-tmp/build-cache/default/ttl-7d:latest": GET https://us.gcr.io/v2/token?scope=repository%3Aavid-shape-445101-a4%2Fapp-engine-tmp%2Fbuild-cache%2Fdefault%2Fttl-7d%3Apull&service=us.gcr.io: DENIED: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/avid-shape-445101-a4/locations/us/repositories/us.gcr.io" (or it may not exist)
```
Again, its permission issue.

Add necessary permissions

After adding the permission shown in build logs, you will see more permissions needed. In the end I figured out that below permissions are necessary in addition to above ones:
- roles/artifactregistry.createOnPushWriter
- roles/storage.objectAdmin
Now run gcloud app deploy, and you should be able to deploy the app successfully.
December 18, 2024
[AdBlock Anatomy] Run and Debug AdBlock Locally
This article is starter of an anatomy series for dissecting AdBlock/AdBlockPlus/uBlockOrigin source code and learn how they work. First let’s start from running and debugging AdBlock locally.

Note that this article is for eyeo owned browser extension AdBlock and AdBlockPlus.

Source code

Luckily, AdBlock main source entry is using NX monorepo, where we can easily see the dependency graph within the repo below.

However, there are many core features are not within the monorepo, here is a list of important ones:
- @eyeo/webext-ad-filtering-solution: most important core adblocking features shared across extensions.
- @eyeo/snippets: eyeo developed special content scripts for element hiding and behavioral intercepting. This repo reflects everything from eyeo official developer document.
- Filter list: eyeo developed filter list mostly leveraging @eyeo/snippets for enhanced blocking.
- @adblockinc/rules: an npm package for retrieving, storing and providing filter lists from various sources for use in AdBlock and Adblock Plus.
Installation

Now let’s download the main source code and install packages:
```
git clone https://gitlab.com/eyeo/extensions/extensions.git
cd extensions
npm install
```
In often case if you are using Apple silicon chips, you will often see node-gyp errors like below:
```
> npm ERR! gyp ERR! stack Error: `gyp` failed with exit code: 1
```
The extensions repo provides some potential fixes, but below commends worked for me:
```
python3 -m pip install setuptools
brew install pkg-config cairo pango libpng jpeg giflib librsvg
```
Build extension

Simply run below to build both AdBlock and AdBlockPlus extension:
```
npm run build
```
Add extensions to Chrome
- Go to chrome://extensions/.
- At the top right, turn on Developer mode.
- Click Load unpacked.
- Find and select the app or extension folder.
  - e.g. for Chrome AdBlockPlus with Manifest V2, it is located inside /extensions/host/adblockplus/dist/devenv/chrome-mv2
Now you should be able to see the extension being added and enabled.

Debugging @eyeo/webext-ad-filtering-solution

In a lot of cases, the feature you are looking for is mostly in webext-ad-filtering-solution package. This package can be loaded as extension standalone.

Installation
```
git clone https://gitlab.com/eyeo/adblockplus/abc/webext-ad-filtering-solution.git
cd extensions
npm install
```
Build and run in watch mode

You can simply do npm run build, but I found it convenient to use watch mode:
```
npx webpack --watch
```
The built extensions can be found under dist folder
- performance-mv2 for Manifest V2
- performance-mv3 for Manifest V3
Simply add desired extension to Chrome in dev mode for debugging.

Debugging background script

Let’s go to sdk/background/index.js and add some logging:
```
...
export async function start(addonInfo) {
  console.log("start extension");
  ...
}
...
```
Then go to chrome://extensions/ and click on Inspect views -> background page (Service worker for Manifest V3 version) under eyeo's WebExtension Ad-Filtering Solution Test Extension.

You should be able to see start extension log under Console tab. You can try reload extension if not seeing the log.

Debugging content script

Let’s go to sdk/content/index.js and add some logging:
```
...
async function initContentFeatures() {
  console.log("initContentFeatures");
  ...
}
...
```
Open a new tab and go to any page, then open console and you should be able to see initContentFeatures log.
December 10, 2024

Containerize Actix Web App Using Docker In A Cleaner Way

Recently came across this task to try different Rust web frameworks and Actix Web is one of the most popular frameworks. Its documentation is clean and easy to bootstrap, however, when I was trying to deploy the app using Docker, I could not find a clean Dockerfile example that contains minimal dependencies.

Luckily, when I was trying Rocket, it has a pretty clean documentation about containerize the app. After tweaking it a bit, it just worked for Actix Web app as well.

Personally, I feel more confident to follow official documents for long term maintainability. Even tho the image size coming from the build is not as small as ~60MB like in this guide, but ~145MB image is also acceptable here consider other factors.

Initiate Actix Web Sample App

Create a new Rust app:

cargo new hello-world
cd hello-world

Now add Actix Web dependency into Cargo.toml file:

[dependencies]
actix-web = "4"

Then replace the contents of src/main.rs with the following:

use actix_web::{get, post, web, App, HttpResponse, HttpServer, Responder};

#[get("/")]
async fn hello() -> impl Responder {
    HttpResponse::Ok().body("Hello world!")
}

#[post("/echo")]
async fn echo(req_body: String) -> impl Responder {
    HttpResponse::Ok().body(req_body)
}

async fn manual_hello() -> impl Responder {
    HttpResponse::Ok().body("Hey there!")
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| {
        App::new()
            .service(hello)
            .service(echo)
            .route("/hey", web::get().to(manual_hello))
    })
    .bind(("0.0.0.0", 8080))?
    .run()
    .await
}

Compile and run the program:

cargo run

Test app using http://0.0.0.0:8080

Add Dockerfile and build Docker image

The original Dockerfile from Rocket document has some Rocket specific settings, simply removing those env variables would just work:

FROM docker.io/rust:1-slim-bookworm AS build

## cargo package name: customize here or provide via --build-arg
ARG pkg=hello-world

WORKDIR /build

COPY . .

RUN --mount=type=cache,target=/build/target \
    --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/usr/local/cargo/git \
    set -eux; \
    cargo build --release; \
    objcopy --compress-debug-sections target/release/$pkg ./main

################################################################################

FROM docker.io/debian:bookworm-slim

WORKDIR /app

## copy the main binary
## add more files below if needed
COPY --from=build /build/main ./

EXPOSE 8080

CMD ./main

Note that, pkg ARG need to be the same as package name in Cargo.toml.

Now let’s build the image:

docker build -t hello-world-image .

Check the image size (~145MB):

Finally, run the image:

docker run -d -p 0.0.0.0:8080:8080 hello-world-image

Test the app using http://0.0.0.0:8080

December 9, 2024

Develop and Deploy Rust Rocket App Using Docker Compose (Traefik/Docker Hub)

This article will talk about how to use Docker for quick Rocket web app deployment. This architecture can be scaled with more complicated CI/CD framework and Kubernetes clusters for large scale application and zero downtime deployment.

Prerequisite

Linux server
Local dev machine with Rust installed
Docker with Docker Compose
Follow this guide for setting up Traefik.
Domain name

Getting started

Update rust

Install rustup by following the instructions on its website. Once rustup is installed, ensure the latest toolchain is installed by running the command:

rustup default stable

Initiate Rocket sample app

cargo new hello-rocket --bin
cd hello-rocket

Now, add Rocket as a dependency in your Cargo.toml:

[dependencies]
rocket = "0.5.1"

Modify src/main.rs so that it contains the code for the Rocket Hello, world! program, reproduced below:

#[macro_use] extern crate rocket;

#[get("/")]
fn index() -> &'static str {
    "Hello, world!"
}

#[launch]
fn rocket() -> _ {
    rocket::build().mount("/", routes![index])
}

Finally, we can run below command to test our first app:

cargo run

Build docker image and upload

Add Dockerfile

There are many tutorials out there but it is always a good practice to follow official documents: https://rocket.rs/guide/v0.5/deploying/#containerization

Note that, in order to test Docker image in local, EXPOSE is needed:

FROM docker.io/rust:1-slim-bookworm AS build

## cargo package name: customize here or provide via --build-arg
ARG pkg=hello-rocket

WORKDIR /build

COPY . .

RUN --mount=type=cache,target=/build/target \
    --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/usr/local/cargo/git \
    set -eux; \
    cargo build --release; \
    objcopy --compress-debug-sections target/release/$pkg ./main

################################################################################

FROM docker.io/debian:bookworm-slim

WORKDIR /app

## copy the main binary
COPY --from=build /build/main ./

## copy runtime assets which may or may not exist
COPY --from=build /build/Rocket.tom[l] ./static
COPY --from=build /build/stati[c] ./static
COPY --from=build /build/template[s] ./templates

## ensure the container listens globally on port 8000
ENV ROCKET_ADDRESS=0.0.0.0
ENV ROCKET_PORT=8000

## uncomment below to test in local
## EXPOSE 8000

CMD ./main

Make sure the pkg set to same value as package name in Cargo.toml

Build Docker image

docker build -t your_username/my-rocket-image .

Upload Docker image to Docker Hub

First make sure you have Docker Hub account and then login to Docker:

Docker login

Then upload Docker image to Docker Hub:

docker push your_username/my-rocket-image

Deploy docker image to cloud instance with Docker Compose and Traefik

Assume you already followed this guide and Traefik reverse proxy is up running in your server.

Run uploaded Rocket app Docker image

First let’s add a folder:

mkdir ~/rocket-docker && cd ~/rocket-docker

Then add docker-compose.yml:

vi docker-compose.yml

networks:
  traefik:
    external: true
 
services:
  app:
    image: your_username/my-rocket-image:latest
    networks:
      - traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.rocket-docker.rule=Host(`your-domain.com`)"
      - "traefik.http.routers.rocket-docker.service=rocket-docker"
      - "traefik.http.routers.rocket-docker.entrypoints=websecure"
      - "traefik.http.services.rocket-docker.loadbalancer.server.port=8000"
      - "traefik.http.services.rocket-docker.loadbalancer.passhostheader=true"
      - "traefik.http.routers.rocket-docker.tls=true"
      - "traefik.http.routers.rocket-docker.tls.certresolver=letsencrypt"
      - "traefik.http.routers.rocket-docker.middlewares=compresstraefik"
      - "traefik.http.middlewares.compresstraefik.compress=true"
      - "traefik.docker.network=traefik"
    restart: unless-stopped

Make sure to update your-domain.com

Run below to start service:

docker compose up -d

Verify by visiting www.your-domain.com see if everything works properly.

Develop and update service with latest change

Try change something in your local Next.js app, and run upload script again:

docker build -t your_username/my-rocket-image . && docker push your_username/my-rocket-image

Then go to your cloud instance and run below:

docker pull your_username/my-rocket-image:latest && docker compose -f ~/rocket-docker/docker-compose.yml up -d

Lastly, verify the change by visiting www.your-domain.com.

December 9, 2024

Setup Traefik Reverse Proxy and Add Routing Using Docker Compose

Reverse proxy is essential for any service being accessed publicly. Traefik is a popular reverse proxy and load balancer designed for microservices and containerized applications. Please make sure Docker is installed with Docker Compose.

Setup Traefik with built-in dashboard app

Let’s first create a folder for Traefik reverse proxy.

mkdir ~/traefik && cd ~/traefik

Next we need to create network for Traefik to communicate with other containers, and it should be exposed externally.

docker create network traefik

Now, let’s create docker-compose.yml

vi docker-compose.yml

networks:
  traefik:
    external: true

volumes:
  traefik-certificates:

services:
  traefik:
    image: "traefik:latest"
    command:
      - "--log.level=DEBUG"
      - "--accesslog=true"
      - "--api.dashboard=true"
      - "--api.insecure=true"
      - "--ping=true"
      - "--ping.entrypoint=ping"
      - "--entryPoints.ping.address=:8082"
      - "--entryPoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entryPoints.websecure.address=:443"
      - "--providers.docker=true"
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
      - "--providers.docker.exposedByDefault=false"
      - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
      # For requesting dev cert (if prod cert has issue during development)
      # - "--certificatesresolvers.myhttpchallenge.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.letsencrypt.acme.email=admin@bill-min.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/etc/traefik/acme/acme.json"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - traefik-certificates:/etc/traefik/acme
    networks:
      - traefik
    ports:
      - "80:80"
      - "443:443"
    healthcheck:
      test: ["CMD", "wget", "http://localhost:8082/ping","--spider"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 5s
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`traefik.your-domain.com`)"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.tls.certresolver=letsencrypt"
      - "traefik.http.routers.dashboard.middlewares=authtraefik"
      - "traefik.http.middlewares.authtraefik.basicauth.users=your_username:{SHA}your_hash"
    restart: unless-stopped

Above config also adds Traefik Dashboard and https with letsencrypt.

Replace your-domain with your actual domain name and setup correct DNS.

Replace your_username and {SHA}your_hash with ones generated from https://hostingcanada.org/htpasswd-generator/

Finally, lets start Traefik container.

docker compose up -d

Verify the container is running successfully

docker ps

You can also check Traefik dashboard by visiting https://traefik.your-domain.com. (you will need to enter the username and password used when generating htpasswd)

Add routing to other containerized apps

After setting up Traefik, adding routes is simple. In most cases, there are 2 things get involved:

Communicating with Traffic through network that can be accessed externally, which is traefik from above setting.
Running the container from Docker image behind Traefik load balancer at exposed port.

Traefik will config properly based on labels and below is an example of running Next.js app through Docker Compose.

networks:
  traefik:
    external: true

services:
  app:
    image: your_username/nextjs-docker:latest
    ports:
      - 3000:3000
    networks:
      - traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nextjs.rule=Host(`www.your-domain.com`)"
      - "traefik.http.routers.nextjs.service=nextjs"
      - "traefik.http.routers.nextjs.entrypoints=websecure"
      - "traefik.http.services.nextjs.loadbalancer.server.port=3000"
      - "traefik.http.services.nextjs.loadbalancer.passhostheader=true"
      - "traefik.http.routers.nextjs.tls=true"
      - "traefik.http.routers.nextjs.tls.certresolver=letsencrypt"
      - "traefik.http.routers.nextjs.middlewares=compresstraefik"
      - "traefik.http.middlewares.compresstraefik.compress=true"
      - "traefik.docker.network=traefik"
    restart: unless-stopped

For running different apps, the settings are mostly the same except the router name and service name.

December 3, 2024

How Do Ad Blocking Browser Extensions Work?
Partial of below content were copied from answers using Perplexity AI

Ad blocking browser extensions work by intercepting and filtering web content before it’s rendered in your browser. Here’s a detailed explanation of how they function:

Filter lists and rules

Ad blockers rely on filter lists, which are collections of predefined rules that determine what content should be blocked or hidden on web pages. These lists contain patterns and rules that match known ad servers, tracking scripts, and other unwanted elements.

Content interception process

When you visit a website, the ad blocker extension activates before the page is fully loaded. It performs the following steps:
- HTTPS Request Blocking: The extension listens to outgoing HTTPS requests from your browser. It compares these requests against its filter lists and blocks any that match known ad platforms or tracking services.
- URL Filtering: As the page loads, the ad blocker checks URLs of various elements against its filter lists. If a match is found, the content from that URL is blocked.
- Content Filtering: The extension analyzes the HTML, CSS, and JavaScript of the page, looking for patterns that indicate ads or unwanted content.
- CSS Injection: Ad blockers may inject custom CSS rules to hide elements that couldn’t be blocked at the network level.
- JavaScript Injection: Some ad blockers inject their own JavaScript code to counteract advertising scripts and prevent them from functioning.
What happens behind scene?

Ad blocking browser extensions utilize both content scripts and background scripts to effectively block ads and unwanted content. Let’s delve deeper into how these scripts work together.

Background scripts

Background scripts run continuously in the extension’s background page, separate from any particular web page. They play a crucial role in ad blocking:
- Filter List Management: Background scripts are responsible for downloading, parsing, and updating filter lists. These lists contain rules for blocking ads and are typically updated periodically to stay current with new ad servers and patterns.
- Request Interception: Background scripts use the browser’s webRequest API to intercept and analyze network requests before they are sent. This allows the ad blocker to block requests to known ad servers or tracking domains at the network level, preventing the ads from loading in the first place.
- Communication Hub: The background script acts as a central communication point, receiving messages from content scripts and popup interfaces, and coordinating the extension’s overall behavior.
- Rule Matching: When a request is intercepted, the background script quickly checks it against the loaded filter lists to determine if it should be blocked.
Content scripts

Content scripts are injected into web pages and can manipulate the DOM (Document Object Model) directly. They are essential for handling ad blocking tasks that can’t be accomplished through network request blocking alone:
- Element Hiding: Content scripts can inject CSS rules or modify the page’s existing CSS to hide ad elements that have already loaded. This is useful for ads that are served from the same domain as the main content and can’t be blocked at the network level.
- DOM Scanning: These scripts can scan the page’s DOM structure to identify and remove ad-related elements based on specific patterns or rules.
- Script Injection: Content scripts can inject additional JavaScript into the page to neutralize ad-related scripts or prevent them from executing.
- Cosmetic Filtering: They apply cosmetic filters to remove empty spaces left by blocked ads, improving the page’s appearance.
Interaction between background and content scripts

The background and content scripts work together to provide comprehensive ad blocking:
- Message Passing: Content scripts communicate with the background script via message passing, sending information about the current page and receiving instructions on what to block or modify.
- Dynamic Rule Application: The background script can send updated blocking rules to content scripts in real-time, allowing for dynamic ad blocking that adapts to changes on the page.
- Performance Optimization: By dividing tasks between background and content scripts, ad blockers can optimize performance. Network-level blocking happens in the background script, while page-specific modifications occur in the content script.
This combination of background and content scripts allows ad blocking extensions to provide a comprehensive and efficient ad-blocking experience, handling both network-level blocking and page-specific content manipulation.
November 30, 2024

Develop and Deploy Next.js App Using Docker Compose (Traefik/Docker Hub)

Before Docker, the way to deploy a node app usually requires uploading/downloading latest files using git or file transfer protocol then re-run the app in cloud. With Docker we can build docker image from local development then upload to Docker Hub. With Docker Compose setup in cloud, we can easily fetch latest image and re-run the container. This architecture can be scaled with more complicated CI/CD framework and Kubernetes clusters for large scale application and zero downtime deployment.

Install Docker Desktop

Simply download and install Docker Desktop from: https://www.docker.com/products/docker-desktop/

Register account (if not already)

A Docker Hub account is required in order to upload images.

Login to Docker user CLI

Run below command to login in to Docker for using docker push command.

Docker login

Initiate sample Next.js app

Official Next.js doc has sample app with docker:

npx create-next-app --example with-docker nextjs-docker

After going through prompts, Dockerfile should be already inside project folder.

Add build&upload script inside package.json

Official Next.js doc has instructions for building and running Next.js with Docker, however, let’s do it more intuitively. Add below script to package.json

{
  ...
  "scripts": {
    "upload:docker": "docker build -t your_username/nextjs-docker . && docker push your_username/nextjs-docker"
  }
  ...
}

Replace nextjs-docker with your preferred app’s name.

Build docker image and upload

Simply run below command:

npm run upload:docker

This will build docker image then upload to Docker Hub with latest tag.

Github and Github Action

In case github is being used as source control, above step can be integrated with Github Action and trigger on commit or PR merge.

Deploy docker image to cloud instance with Docker Compose and Traefik

Prerequisite

Linux server
Docker Engine installed with Docker Compose plugin

Setup Traefic (skip if already done)

Let’s make a folder for Traefic:

mkdir ~/traefik && cd ~/traefik

Next we need to create network for Traefik to communicate with other containers, and it should be exposed externally.

docker create network traefik

Now, let’s create docker-compose.yml

vi docker-compose.yml

networks:
  traefik:
    external: true

volumes:
  traefik-certificates:

services:
  traefik:
    image: "traefik:latest"
    command:
      - "--log.level=DEBUG"
      - "--accesslog=true"
      - "--api.dashboard=true"
      - "--api.insecure=true"
      - "--ping=true"
      - "--ping.entrypoint=ping"
      - "--entryPoints.ping.address=:8082"
      - "--entryPoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entryPoints.websecure.address=:443"
      - "--providers.docker=true"
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
      - "--providers.docker.exposedByDefault=false"
      - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
      # For requesting dev cert (if prod cert has issue during development)
      # - "--certificatesresolvers.myhttpchallenge.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.letsencrypt.acme.email=admin@bill-min.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/etc/traefik/acme/acme.json"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - traefik-certificates:/etc/traefik/acme
    networks:
      - traefik
    ports:
      - "80:80"
      - "443:443"
    healthcheck:
      test: ["CMD", "wget", "http://localhost:8082/ping","--spider"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 5s
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`traefik.your-domain.com`)"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.tls.certresolver=letsencrypt"
      - "traefik.http.routers.dashboard.middlewares=authtraefik"
      - "traefik.http.middlewares.authtraefik.basicauth.users=your_username:{SHA}your_hash"
    restart: unless-stopped

Above config also adds Traefik Dashboard and https with letsencrypt.

Replace your-domain with your actual domain name and setup correct DNS.

Replace your_username and {SHA}your_hash with ones generated from https://hostingcanada.org/htpasswd-generator/

Finally, lets start Traefik container.

docker compose up -d

Verify the container is running successfully

docker ps

You can also check Traefik dashboard by visiting https://traefik.your-domain.com. (you will need to enter the username and password used when generating htpasswd)

Run uploaded Next.js docker image

First let’s add a folder:

mkdir ~/nextjs-docker && cd ~/nextjs-docker

Then add docker-compose.yml:

networks:
  traefik:
    external: true

services:
  app:
    image: your_username/nextjs-docker:latest
    ports:
      - 3000:3000
    networks:
      - traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nextjs.rule=Host(`www.your-domain.com`)"
      - "traefik.http.routers.nextjs.service=nextjs"
      - "traefik.http.routers.nextjs.entrypoints=websecure"
      - "traefik.http.services.nextjs.loadbalancer.server.port=3000"
      - "traefik.http.services.nextjs.loadbalancer.passhostheader=true"
      - "traefik.http.routers.nextjs.tls=true"
      - "traefik.http.routers.nextjs.tls.certresolver=letsencrypt"
      - "traefik.http.routers.nextjs.middlewares=compresstraefik"
      - "traefik.http.middlewares.compresstraefik.compress=true"
      - "traefik.docker.network=traefik"
    restart: unless-stopped

Note that, the Next.js app need to run inside same network as Traefic. Also replace your-domain with real domain.

Run below to start service:

docker compose up -d

Verify by visiting www.your-domain.com see if everything works properly.

Develop and update service with latest change

Try change something in your local Next.js app, and run upload script again (or using Github Action):

npm run upload:docker

Then go to your cloud instance and run below:

docker pull your_username/nextjs-docker:latest && docker compose -f ~/nextjs-docker/docker-compose.yml up -d

Lastly, verify the change by visiting www.your-domain.com.

November 29, 2024

Pull and Run Latest Docker Image With One Line Using Docker Compose
In case the docker image with latest tag has been updated, we can use below one-line script to pull the updated version and recreate container.
```
docker pull docker-image:latest && docker compose -f ~/apps/docker-compose.yml up -d
```
After that, we can use below script for cleaning up dangling images.
```
docker image prune
```
November 29, 2024

Author: Bill

Simple but ugly solution

The better one

Anatomy

Vendored OpenSSL

Automatic OpenSSL

A Debian image issue

Final Dockerfile

Step by step solution

Give access to storage bucket

Using Console UI

Using CLI

Grant access to writing logs

Add necessary permissions

Source code

Installation

Build extension

Add extensions to Chrome

Debugging @eyeo/webext-ad-filtering-solution

Installation

Build and run in watch mode

Debugging background script

Debugging content script

Initiate Actix Web Sample App

Add Dockerfile and build Docker image

Prerequisite

Getting started

Update rust

Initiate Rocket sample app

Build docker image and upload

Add Dockerfile

Build Docker image

Upload Docker image to Docker Hub

Deploy docker image to cloud instance with Docker Compose and Traefik

Run uploaded Rocket app Docker image

Develop and update service with latest change

Setup Traefik with built-in dashboard app

Add routing to other containerized apps

Filter lists and rules

Content interception process

What happens behind scene?

Background scripts

Content scripts

Interaction between background and content scripts

Install Docker Desktop

Register account (if not already)

Login to Docker user CLI

Initiate sample Next.js app

Add build&upload script inside package.json

Build docker image and upload

Github and Github Action

Deploy docker image to cloud instance with Docker Compose and Traefik

Prerequisite

Setup Traefic (skip if already done)

Run uploaded Next.js docker image

Develop and update service with latest change

Debugging `@eyeo/webext-ad-filtering-solution`