Tech Unblocker

Category: Cloud

  • [Troubleshooting] Error during `gcloud app deploy` for GAE app: “Failed to create cloud build: invalid bucket”

    When I was trying to run app with Google App Engine, I followed Hello World example step by step, and no surprise that it didn’t work. You will see below error:

    ERROR: (gcloud.app.deploy) Error Response: [13] Failed to create cloud build: com.google.net.rpc3.client.RpcClientException: <eye3 title='/ArgoAdminNoCloudAudit.CreateBuild, FAILED_PRECONDITION'/> APPLICATION_ERROR;google.devtools.cloudbuild.v1/ArgoAdminNoCloudAudit.CreateBuild;invalid bucket "staging.avid-shape-445101-a4.appspot.com"; service account avid-shape-445101-a4@appspot.gserviceaccount.com does not have access to the bucket;AppErrorCode=9;StartTimeMs=1734486112961;unknown;ResFormat=uncompressed;ServerTimeSec=0.921583338;LogBytes=256;Non-FailFast;EndUserCredsRequested;EffSecLevel=privacy_and_integrity;ReqFormat=uncompressed;ReqID=4b101eaa0d045332;GlobalID=0;Server=[2002:a05:6670:1585:b0:a5e:1b4f:746d]:4001.

    Same error has been posted in many places:

    • https://www.googlecloudcommunity.com/gc/Serverless/Failed-to-create-cloud-build-no-access-to-bucket/m-p/758310
    • https://stackoverflow.com/questions/78742739/error-during-gcloud-app-deploy-for-gae-app-failed-to-create-cloud-build-inv
    • https://www.googlecloudcommunity.com/gc/Serverless/Error-during-gcloud-app-deploy-for-GAE-app-quot-Failed-to-create/m-p/778778
    • etc.

    Based on this doc, starting May 3, 2024, iam.automaticIamGrantsForDefaultServiceAccounts is disabled by default, which caused all of these issues.

    Thanks to this answer, it gives me strong lead to the solution, although there are still some extra steps (might be from undocumented recent changes from Google Cloud).

    Step by step solution

    Give access to storage bucket

    There are 2 ways of doing it.

    Using Console UI

    1. Go to https://console.cloud.google.com/storage/browser, select staging.PROJECT_ID.appspot.com and go to Permissions tab.
    2. Click on GRANT ACCESS button.
    3. Enter PROJECT_ID@appspot.gserviceaccount.com as New principals.
    4. Enter Storage Admin as new Role.
    5. Save setting.

    Using CLI

    Simply using below command:

    gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:PROJECT_ID@appspot.gserviceaccount.com" --role="roles/storage.admin"

    Now let’s deploy again:

    gcloud app deploy

    No surprise, you will see below error:

    ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build f490b199-1e2e-4fbb-89c7-e0caae57f7bc status: FAILURE
An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=us-central1/f490b199-1e2e-4fbb-89c7-e0caae57f7bc?project=519995341679
Full build logs: https://console.cloud.google.com/cloud-build/builds;region=us-central1/f490b199-1e2e-4fbb-89c7-e0caae57f7bc?project=519995341679

    Its much better now, at least we have a log. Let’s go to the link provided.

    You will very likely see above warning and empty logs.

    Grant access to writing logs

    Run below command to add role as suggested:

    gcloud projects add-iam-policy-binding avid-shape-445101-a4 --member="serviceAccount:avid-shape-445101-a4@appspot.gserviceaccount.com" --role="roles/logging.logWriter"

    Now let’s run deploy again (just to to see actual building logs):

    gcloud app deploy

    You will see same error but in the build log, you will see why exactly the build failed:

    ERROR: failed to create image cache: accessing cache image "us.gcr.io/avid-shape-445101-a4/app-engine-tmp/build-cache/default/ttl-7d:latest": connect to repo store "us.gcr.io/avid-shape-445101-a4/app-engine-tmp/build-cache/default/ttl-7d:latest": GET https://us.gcr.io/v2/token?scope=repository%3Aavid-shape-445101-a4%2Fapp-engine-tmp%2Fbuild-cache%2Fdefault%2Fttl-7d%3Apull&service=us.gcr.io: DENIED: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/avid-shape-445101-a4/locations/us/repositories/us.gcr.io" (or it may not exist)

    Again, its permission issue.

    Add necessary permissions

    After adding the permission shown in build logs, you will see more permissions needed. In the end I figured out that below permissions are necessary in addition to above ones:

    • roles/artifactregistry.createOnPushWriter
    • roles/storage.objectAdmin

    Now run gcloud app deploy, and you should be able to deploy the app successfully.