Docker Build Works Locally and Fails in CI: The Eight Causes

Same Dockerfile, same repo, different result. The difference is nearly always cache, context, architecture, or something on your machine that is not in the image.

Share
Docker Build Works Locally and Fails in CI: The Eight Causes. Abstract docker illustration in orange and dark grey on debugly.dev

The short answer

Your local build is using something CI does not have. In rough order of frequency:

  1. Layer cache hiding a broken step
  2. Files excluded by .dockerignore that exist locally
  3. Architecture mismatch between your Mac and the CI runner
  4. Uncommitted files that are in your working directory and not in git
  5. Different base image because a tag moved
  6. Resource limits on the runner, usually memory during a build step
  7. Network policy blocking a registry or package index
  8. Build secrets present locally as env vars and absent in CI

Reproduce CI's conditions locally with:

docker build --no-cache --pull --progress=plain -t test .

That one command catches causes one and five, which together are most of them.

Tested on Docker 27.5 with BuildKit.

1. The layer cache is lying to you

The most common by a wide margin.

Docker caches each layer. If a RUN step's inputs have not changed, the cached result is reused. Your local machine has months of cached layers. A CI runner usually starts empty.

So a step that is genuinely broken can appear to work locally forever, because it has not actually run since the day it worked.

RUN apt-get install -y libpq-dev     # cached from before you removed apt-get update

Without apt-get update in the same layer, this works from cache and fails on a clean build.

docker build --no-cache -t test .

If that fails and your normal build succeeds, this is your answer. I would run a --no-cache build on a schedule in CI regardless, because it catches this class before it blocks a release.

The related trap is cache invalidation ordering. Put the steps that change least at the top:

COPY package.json pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile
COPY . .                              # source changes do not bust the install layer

Reverse those and every source edit reinstalls all dependencies, which is slow locally and expensive in CI.

2. .dockerignore is excluding something you need

The build context is what gets sent to the daemon. .dockerignore filters it. A file present in your working directory but excluded from context does not exist inside the build.

Locally you might never notice, because a cached layer already contains it from before the ignore rule was added.

Check what is actually in the context:

# what is being sent
docker build --no-cache --progress=plain . 2>&1 | head -5
# look for "transferring context"

To see the file list, build a throwaway image that lists it:

FROM alpine
COPY . /ctx
RUN find /ctx -type f | head -100

The inverse problem is more common and more damaging: .dockerignore missing entries it should have. If node_modules is not ignored, your locally built, architecture specific, possibly stale dependencies get copied into the image. That produces a build that works on your Mac and dies with an exec format error on an amd64 runner.

A baseline .dockerignore:

.git
node_modules
target
.venv
__pycache__
dist
build
.env
.env.*
*.log
.DS_Store

3. Architecture

You are on Apple Silicon. The runner is amd64. Docker builds for the host architecture by default, so you have built and tested an arm64 image and pushed an amd64 one, or vice versa.

Symptoms range from an obvious exec format error to subtler failures where a native dependency compiles fine locally and fails on the runner because a prebuilt wheel exists for one architecture and not the other.

docker buildx build --platform linux/amd64 -t test .

I cover the whole category in the ARM64 and AMD64 post, including how to cross compile rather than emulate, because emulated builds are slow enough to matter in CI.

4. Uncommitted files

CI builds from a clean checkout of a commit. You build from your working directory.

A file you created and never committed, a local config, a generated file that is gitignored, or a .env you have had sitting there since March will all be present locally and absent in CI.

git status --porcelain --ignored | head -30

Then build from a clean clone to confirm:

git clone --depth 1 file://$(pwd) /tmp/clean-build
cd /tmp/clean-build && docker build --no-cache -t test .

This is the single most reliable way to reproduce CI locally and it takes thirty seconds.

5. The base image moved

FROM node:22

That tag is a moving target. Node 22.6 last month, 22.14 today, with a different npm and possibly a different Debian base. Your local machine has the old one cached. CI pulls the new one.

docker build --pull ...     # always fetch the newest base

Pin properly:

FROM node:22.14.0-bookworm-slim

Digest pinning is stricter still, and note the tradeoff: a digest identifies one specific manifest, so it pins the architecture too and breaks multi arch builds. Pin by full version tag unless you have a specific reason not to.

6. Runner resources

CI runners are usually smaller than your laptop. GitHub's standard hosted runners are 2 vCPU with 7GB RAM at time of writing.

A webpack or Vite build, a TypeScript compile of a large project, or a Rust build with high parallelism can exceed that. The failure is often unhelpful:

The command '/bin/sh -c pnpm build' returned a non-zero code: 137

Exit 137 is SIGKILL, and in a build that almost always means the OOM killer. Same mechanism as a container getting OOMKilled at runtime.

Fixes: cap Node's heap with NODE_OPTIONS=--max-old-space-size=3072, reduce build parallelism with -j2 or --max-workers=2, or move to a larger runner.

7. Network policy

Corporate CI often runs behind a proxy or with egress restrictions. A build step that reaches out to a registry, a git URL, or a CDN works locally and times out in CI.

# needs github.com egress, which your CI may not have
RUN pip install git+https://github.com/someone/somelib.git

Symptoms are timeouts rather than refusals, so the build hangs for a while and then fails. Check the runner's egress rules and prefer vendored dependencies or an internal mirror for anything on the critical path.

8. Build secrets

You have NPM_TOKEN exported in your shell. CI does not, or has it under a different name.

Never bake secrets into layers with ARG, because they persist in the image history and anybody who pulls the image can read them:

docker history --no-trunc myimage | grep -i token

Use BuildKit secret mounts, which are not written to any layer:

RUN --mount=type=secret,id=npmtoken \
    NPM_TOKEN=$(cat /run/secrets/npmtoken) pnpm install --frozen-lockfile
docker build --secret id=npmtoken,env=NPM_TOKEN .

Getting useful output from a failing CI build

BuildKit hides output by default, which is unhelpful when you cannot attach to the runner.

docker build --progress=plain --no-cache .

--progress=plain prints every command's full output instead of the collapsed view. Put it in your CI build command permanently. The bandwidth cost is nothing and the debugging benefit is large.

To inspect an intermediate state, build up to a specific stage:

docker build --target builder -t debug-builder .
docker run --rm -it debug-builder sh

That gets you a shell in the environment as it existed at that stage, which is usually enough to see that the file you expected is not there.

For a step that fails, a quick trick is to make it non fatal temporarily and inspect after:

RUN pnpm build || (ls -la && cat /root/.npm/_logs/*.log && exit 1)

Ugly, effective, and it gets the diagnostic output into the CI log where you can read it.

A prevention checklist

  • Run --no-cache --pull builds on a nightly schedule so cache masking surfaces on its own timeline
  • Pin base images to full version tags
  • .dockerignore includes node_modules, .git, .env, and build outputs
  • Set --platform explicitly if any developer is on a different architecture
  • --progress=plain in the CI build command
  • Secrets through BuildKit mounts, never ARG
  • Build from a clean clone before blaming CI