A 1.2 GB Image That Should Have Been 90 MB

The deploy pipeline moved a gigabyte and a half per build, per node, and the cold start pulled it again. The application inside was a ninety megabyte runtime with a binary. The other fourteen hundred megabytes were the compiler that built it, the package manager cache that fed the compiler, and a shell and toolchain the production container would never use, all of it shipped because Docker layers do not know the difference between building and running.

I have lost on-call weeks to this exact shape of slow burn, so the order below is the order I fix it in.

An image is the accumulation of every layer, and a Dockerfile written as a recipe for building, without a stage boundary, produces an image that is a build machine which happens to run your app. The fix is structural, and it is one keyword.

This was Docker 27.5 with a Node 22.14 app and a Go 1.23 service, both offenders, both fixed the same way.

Why every layer ships

A Dockerfile is a list of instructions, and each instruction produces a layer, and the image is the stack of all of them. A RUN apt install build-essential layer adds the compilers to the image permanently. A later RUN npm ci adds node_modules, including the dev dependencies, permanently. Deleting files in a later layer does not remove them from the earlier layer. It adds a new layer that hides them, and the hidden bytes still ship, because the image is the sum.

This is the layering trap in one sentence: you cannot uninstall a layer, you can only stack on top of it. So the size budget must be spent in the stage that ends the image, not cleaned up afterwards.

The multistage shape

The fix is to separate the build machine from the runtime, using stages, and copy only the artefact across:

FROM node:22 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:22-slim
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
CMD ["node", "dist/server.js"]

The final image contains only what the second stage holds. The compilers, the sources and the caches live in a stage that is discarded, and the shipped image collapses to the runtime plus the artefact.

For Go the collapse is more dramatic, because the artefact is a static binary and the runtime can be nearly empty:

FROM golang:1.23 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /out/app ./cmd/app

FROM gcr.io/distroless/static-debian12
COPY --from=build /out/app /app
ENTRYPOINT ["/app"]

Ninety megabytes becomes twenty, and the attack surface shrinks with the size, because the shell and the toolchain that an attacker would use are no longer present. This is the same instinct as debugging a distroless container with no shell, except here the missing shell is a feature you chose rather than a debugging obstacle you met.

The mistakes that keep images fat

Copying the whole context first. COPY . . before the dependency install defeats the layer cache and pulls sources and secrets into a layer. Copy the manifests first, install, then copy sources, so dependencies cache and sources change cheaply.

Shipping dev dependencies. npm ci installs dev by default. Production stages want npm ci --omit=dev or a separate production install, because the dev toolchain is usually the bulk of node_modules.

Using the full base image out of habit. The full language image carries build tools. The slim or alpine or distroless variant is the runtime. The difference is hundreds of megabytes and most of the CVE surface.

One stage for everything because debugging needs a shell. The honest answer is a debug variant built on demand, not a production image that carries a shell forever. Keep the production stage bare and build the instrumented image when you need it.

What the slim image changes operationally

Pulls get faster, which matters on every scale event, because a new node pulls the image before it can serve. Cold starts shrink for serverless, where the image is on the critical path. The registry bill drops with the bytes. And the security review gets shorter, because the package list of a slim image is short enough to actually read, which is the point.

The cache interaction is worth naming too. Multistage builds keep the build stage's layers cached, so rebuilds are fast for the developer even though the shipped image is small. The two goals, fast builds and small images, stop being in tension once the stages are separate.

The takeaway

An image should contain what the process needs to run and nothing that was needed to build it. Enforce that with a stage boundary and a copy of the artefact, not with cleanup commands that only hide bytes in new layers.

Measure the image size in CI and fail the build when it grows without a corresponding change, because image size, like payload size, grows by accretion, one convenient layer at a time, and the budget is the only thing that makes each accretion a decision.

The runtime cost of shipping more than you need shows up again at the network layer in Brotli saved forty percent on text, where the lesson is the same: the bytes you do not ship are the only bytes that cost nothing.

Verifying the boundary did its job

After splitting the stages, prove the image contains what you intend and nothing else, because the whole point is a boundary and boundaries deserve a check. List the final image's filesystem and read the package manifest of the runtime stage, and confirm the compiler and the build cache are absent. A CI step that fails when the image size or the package list grows beyond the declared budget turns the boundary from a hope into an enforced contract, the same discipline as the performance budget, applied to the artefact rather than the page. The first time a well meaning RUN apt install sneaks into the final stage, that step is what catches it, at review time, instead of the registry bill catching it at the end of the quarter.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud