Category: Troubleshooting

  • [Troubleshooting] Error: could not find system library ‘openssl’ required by the ‘openssl-sys’ crate

    When using reqwest crate and building docker images with Dockerfile mentioned in this article, you will encounter openssl issues due to missing packages.

    Error: could not find system library 'openssl' required by the 'openssl-sys' crate

    It turns out that below line is the culprit:

    FROM docker.io/rust:1-slim-bookworm AS build
    
    #############################################

    rust:1-slim-bookworm misses lots of necessary package to run the application. It is suggested in official Rust docker image document that we should avoid using slim image.

    Simple but ugly solution

    Well, the simplest approach is using a minimal Dockerfile which will always work:

    FROM rust:bookworm
    
    COPY . .
    
    RUN cargo build --release
    
    EXPOSE 8080
    
    CMD ./target/release/your_package

    However, it will generate a pretty large image which can take gigabytes of storage.

    The better one

    We can still use multi stage build to generate an optimal image with a little tweak (using default rust image version rust:bookworm):

    FROM docker.io/rust:bookworm AS build
    
    ## cargo package name: customize here or provide via --build-arg
    ARG pkg=hello-world
    
    WORKDIR /build
    
    COPY . .
    
    RUN --mount=type=cache,target=/build/target \
        --mount=type=cache,target=/usr/local/cargo/registry \
        --mount=type=cache,target=/usr/local/cargo/git \
        set -eux; \
        cargo build --release; \
        objcopy --compress-debug-sections target/release/$pkg ./main
    
    ################################################################################
    
    FROM docker.io/debian:bookworm-slim
    
    WORKDIR /app
    
    ## copy the main binary
    ## add more files below if needed
    COPY --from=build /build/main ./
    
    EXPOSE 8080
    
    CMD ./main

    Now it builds fine, but you will get a runtime error when running reqwest:

    ./main: error while loading shared libraries: libssl.so.3: cannot open shared object file: No such file or directory

    Furthermore, after installing correct packages either vendored or automatic, you will still see error when sending https requests:

    hyper_util::client::legacy::Error(Connect, Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: "SSL routines", function: "tls_post_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 2092 }]))) }, X509VerifyResult { code: 20, error: "unable to get local issuer certificate" }))

    To solve these issues, I separately wrote an article for detailed explanation and solutions.

  • [Troubleshooting] libssl.so.3 and certificate Error Running Rust/Reqwest Under Debian Image

    If you see below errors while running Rust app using Docker, this article is right for you:

    error while loading shared libraries: libssl.so.3: cannot open shared object file: No such file or directory
    hyper_util::client::legacy::Error(Connect, Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: "SSL routines", function: "tls_post_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 2092 }]))) }, X509VerifyResult { code: 20, error: "unable to get local issuer certificate" }))

    Anatomy

    Reqwest document mentioned about requirements when running under Linux OS, where you need OpenSSL installed.

    To supply OpenSSL, Rust official document provided 2 ways:

    Vendored OpenSSL

    Add below dependency to Cargo.toml.

    [dependencies]
    openssl = { version = "0.10", features = ["vendored"] }

    Automatic OpenSSL

    Add below line to Dockerfile at stage of running app.

    RUN apt-get update && apt-get install -y pkg-config libssl-dev

    Either approach works fine and you can run the app successfully. However, when sending https requests, you will get below error:

    unable to get local issuer certificate

    A Debian image issue

    This thread gives a good idea of what happens here. Essentially, the debian official image does not have ca certification package installed. To solve the issue, simply install the package in Dockerfile:

    RUN apt-get update && apt-get install -y pkg-config libssl-dev && apt install -y ca-certificates
    # or below if using vendored OpenSSL
    # RUN apt-get update && apt install -y ca-certificates

    Final Dockerfile

    To build rust + actix-web + reqwest, below is what works for me:

    [package]
    name = "hello-world"
    version = "0.1.0"
    edition = "2021"
    
    [dependencies]
    actix-web = "4"
    reqwest = "0.12"
    serde = { version = "1.0", features = ["derive"] }
    openssl = { version = "0.10", features = ["vendored"] }
    FROM rust:bookworm AS build
    
    ## cargo package name: customize here or provide via --build-arg
    ARG pkg=hello-world
    
    WORKDIR /build
    
    COPY . .
    
    RUN --mount=type=cache,target=/build/target \
        --mount=type=cache,target=/usr/local/cargo/registry \
        --mount=type=cache,target=/usr/local/cargo/git \
        set -eux; \
        cargo build --release; \
        objcopy --compress-debug-sections target/release/$pkg ./main
    
    ################################################################################
    
    FROM docker.io/debian:bookworm-slim
    
    WORKDIR /app
    
    ## copy the main binary
    COPY --from=build /build/main ./
    
    EXPOSE 8080
    
    RUN apt-get update && apt install -y ca-certificates
    
    CMD ./main
  • [Troubleshooting] Error during `gcloud app deploy` for GAE app: “Failed to create cloud build: invalid bucket”

    When I was trying to run app with Google App Engine, I followed Hello World example step by step, and no surprise that it didn’t work. You will see below error:

    ERROR: (gcloud.app.deploy) Error Response: [13] Failed to create cloud build: com.google.net.rpc3.client.RpcClientException: <eye3 title='/ArgoAdminNoCloudAudit.CreateBuild, FAILED_PRECONDITION'/> APPLICATION_ERROR;google.devtools.cloudbuild.v1/ArgoAdminNoCloudAudit.CreateBuild;invalid bucket "staging.avid-shape-445101-a4.appspot.com"; service account avid-shape-445101-a4@appspot.gserviceaccount.com does not have access to the bucket;AppErrorCode=9;StartTimeMs=1734486112961;unknown;ResFormat=uncompressed;ServerTimeSec=0.921583338;LogBytes=256;Non-FailFast;EndUserCredsRequested;EffSecLevel=privacy_and_integrity;ReqFormat=uncompressed;ReqID=4b101eaa0d045332;GlobalID=0;Server=[2002:a05:6670:1585:b0:a5e:1b4f:746d]:4001.

    Same error has been posted in many places:

    • https://www.googlecloudcommunity.com/gc/Serverless/Failed-to-create-cloud-build-no-access-to-bucket/m-p/758310
    • https://stackoverflow.com/questions/78742739/error-during-gcloud-app-deploy-for-gae-app-failed-to-create-cloud-build-inv
    • https://www.googlecloudcommunity.com/gc/Serverless/Error-during-gcloud-app-deploy-for-GAE-app-quot-Failed-to-create/m-p/778778
    • etc.

    Based on this doc, starting May 3, 2024, iam.automaticIamGrantsForDefaultServiceAccounts is disabled by default, which caused all of these issues.

    Thanks to this answer, it gives me strong lead to the solution, although there are still some extra steps (might be from undocumented recent changes from Google Cloud).

    Step by step solution

    Give access to storage bucket

    There are 2 ways of doing it.

    Using Console UI

    1. Go to https://console.cloud.google.com/storage/browser, select staging.PROJECT_ID.appspot.com and go to Permissions tab.
    2. Click on GRANT ACCESS button.
    3. Enter PROJECT_ID@appspot.gserviceaccount.com as New principals.
    4. Enter Storage Admin as new Role.
    5. Save setting.

    Using CLI

    Simply using below command:

    gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:PROJECT_ID@appspot.gserviceaccount.com" --role="roles/storage.admin"

    Now let’s deploy again:

    gcloud app deploy

    No surprise, you will see below error:

    ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build f490b199-1e2e-4fbb-89c7-e0caae57f7bc status: FAILURE
    An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=us-central1/f490b199-1e2e-4fbb-89c7-e0caae57f7bc?project=519995341679
    Full build logs: https://console.cloud.google.com/cloud-build/builds;region=us-central1/f490b199-1e2e-4fbb-89c7-e0caae57f7bc?project=519995341679

    Its much better now, at least we have a log. Let’s go to the link provided.

    You will very likely see above warning and empty logs.

    Grant access to writing logs

    Run below command to add role as suggested:

    gcloud projects add-iam-policy-binding avid-shape-445101-a4 --member="serviceAccount:avid-shape-445101-a4@appspot.gserviceaccount.com" --role="roles/logging.logWriter"

    Now let’s run deploy again (just to to see actual building logs):

    gcloud app deploy

    You will see same error but in the build log, you will see why exactly the build failed:

    ERROR: failed to create image cache: accessing cache image "us.gcr.io/avid-shape-445101-a4/app-engine-tmp/build-cache/default/ttl-7d:latest": connect to repo store "us.gcr.io/avid-shape-445101-a4/app-engine-tmp/build-cache/default/ttl-7d:latest": GET https://us.gcr.io/v2/token?scope=repository%3Aavid-shape-445101-a4%2Fapp-engine-tmp%2Fbuild-cache%2Fdefault%2Fttl-7d%3Apull&service=us.gcr.io: DENIED: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/avid-shape-445101-a4/locations/us/repositories/us.gcr.io" (or it may not exist)

    Again, its permission issue.

    Add necessary permissions

    After adding the permission shown in build logs, you will see more permissions needed. In the end I figured out that below permissions are necessary in addition to above ones:

    • roles/artifactregistry.createOnPushWriter
    • roles/storage.objectAdmin

    Now run gcloud app deploy, and you should be able to deploy the app successfully.