Your CI Cache Is Making Builds Slower and Occasionally Wrong

A cache with the wrong key restores nothing and uploads everything. A cache with too broad a key restores stale artifacts. Both are common.

Share
Your CI Cache Is Making Builds Slower and Occasionally Wrong. Abstract devops illustration in orange and dark grey on debugly.dev

The short answer

Two failure modes, opposite causes, both common.

Cache never hits. Your key includes something that changes every run, so you download nothing and upload a fresh copy every build. Net effect: slower than no cache at all.

Cache hits when it should not. Your key is too broad, so you restore artifacts built from different inputs. Net effect: builds that succeed on stale code, or fail with errors that make no sense.

Check your hit rate first. Most CI systems log Cache restored from key: or Cache not found for input keys:. If you have never looked at that line, look now.

Diagnosing a cache that never hits

Cache not found for input keys: linux-node-a3f9c2e1

Every run. The key changed.

Print your key and compare across two runs:

- run: echo "key=linux-node-${{ hashFiles('**/package-lock.json') }}"

The usual culprits for a key that changes every time:

hashFiles matching a generated file. **/package-lock.json will match lockfiles inside node_modules if node_modules exists at the time the expression is evaluated, and those change. Be specific: package-lock.json or **/package-lock.json with node_modules excluded.

A timestamp or run id in the key. ${{ github.run_id }} guarantees a miss, always. It appears in a lot of copied configs because it is used correctly in restore-keys fallbacks and incorrectly in the primary key.

Hashing the whole repository. hashFiles('**') changes on every commit.

Different runner OS or architecture between runs, if your matrix is not reflected in the key.

Diagnosing a cache that hits when it should not

Harder, because the symptom is a weird build rather than an obvious message.

Signs worth recognising:

  • A build succeeds after you deleted a dependency, because the old one is still in the restored node_modules
  • A build fails referencing a file that no longer exists in the repository
  • Type errors about a version of a package your lockfile does not specify
  • Tests passing in CI and failing locally on a clean checkout, or the reverse

The test that settles it: disable the cache and re-run. If the behaviour changes, the cache was involved.

- uses: actions/cache@v4
  if: ${{ !contains(github.event.head_commit.message, '[no-cache]') }}

A commit message escape hatch is genuinely useful. Being able to force a clean build without editing the workflow saves time during exactly the investigation this post is about.

What to key on

The rule: the key must include a hash of everything that determines the cached content.

- uses: actions/cache@v4
  with:
    path: |
      ~/.npm
      node_modules
    key: ${{ runner.os }}-node${{ steps.setup.outputs.node-version }}-${{ hashFiles('package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node${{ steps.setup.outputs.node-version }}-

Note the Node version in the key. Native modules compile against a specific ABI, so a node_modules built on Node 20 and restored on Node 22 produces errors that look nothing like a cache problem.

restore-keys is the partial fallback. If the exact key misses, it restores the most recent cache matching the prefix, which gives you a warm starting point rather than nothing. This is the right pattern for a package manager cache directory and the wrong pattern for node_modules itself.

Cache the download, not the install

This is the distinction most configs get wrong.

Caching ~/.npm or ~/.cache/pnpm is caching downloads. Restoring it means the install runs but does not hit the network. Safe, because the install still resolves the lockfile and produces a correct tree.

Caching node_modules directly is caching the result of the install. Faster, and it skips the step that would have corrected any drift. If the cache key is even slightly wrong, you get a dependency tree that does not match your lockfile, and nothing will tell you.

I default to caching the package manager's store, not the installed tree. The install with a warm store is usually only a few seconds slower and the failure mode is much less confusing.

Same reasoning for Python:

path: ~/.cache/pip        # yes
path: .venv               # be careful

And for Rust, cache ~/.cargo/registry and ~/.cargo/git, plus target/ keyed carefully. target/ is large and the biggest win, and it is also the most likely to hold stale artifacts. cargo is generally good at invalidation, better than most build tools, which makes this safer than the equivalent in other ecosystems.

Docker layer caching

Different mechanism, same principles.

- uses: docker/build-push-action@v6
  with:
    cache-from: type=gha
    cache-to: type=gha,mode=max

mode=max caches intermediate layers, not just the final one. Without it, multi stage builds cache almost nothing useful, because the build stage's layers are discarded.

The ordering rule matters more than the cache configuration:

COPY package.json pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile
COPY . .
RUN pnpm build

Dependencies before source. Reversed, every source change invalidates the install layer and the cache buys you nothing.

Be aware that a layer cache can hide a broken build step indefinitely. A RUN apt-get install without apt-get update in the same layer works from cache forever and fails on a clean build, which is one of the reasons a build works locally and fails in CI. Run a --no-cache build on a schedule to catch it on your own timeline.

Measure before you optimise

The instinct is to add caching everywhere. Measure first, because a cache that saves 8 seconds and costs 30 seconds of upload is a regression.

Time each step and compare cold against warm:

- run: echo "::group::timing" && date +%s

Or read the timing your CI already records per step. What you want to know for each cache:

  • Restore time
  • Save time, which many people forget counts
  • Time saved in the step it accelerates

A cache is worth it when restore plus save is meaningfully less than the work it replaces. For a large node_modules, restore and save can each take 20 to 40 seconds, which means it only pays if the install would take over 90 seconds.

Cache size matters here. Most CI providers cap total cache storage and evict least recently used entries. A workflow caching 2GB per matrix entry across six entries will evict its own caches, producing a mysterious pattern where the cache works in the morning and misses in the afternoon.

The correctness rule

One principle worth holding: a cache should never change the result of a build, only its speed.

If disabling the cache changes whether the build passes, the build is not reproducible and the cache is not the real problem, it is the thing that revealed it.

Practically that means:

  • Lockfiles committed, and frozen installs (npm ci, pnpm install --frozen-lockfile, uv sync --frozen)
  • Base images pinned to full version tags
  • Tool versions pinned in the workflow, not resolved to latest
  • Caches keyed on a hash of everything that determines their contents

That last one is the whole discipline. If you cannot articulate what inputs determine a cached artifact, you cannot key it correctly, and you will eventually restore something that does not match.

A quick audit

Worth running through once on any workflow you have inherited:

  1. What is the hit rate? Read the log line.
  2. Does the key include the tool version, the OS, and a hash of the lockfile?
  3. Are you caching the download store or the installed tree?
  4. How long do restore and save take, and is the step they accelerate longer than that?
  5. Does the build pass with the cache disabled?
  6. Is there a scheduled clean build to catch cache-masked breakage?

Most workflows fail two or three of those, and fixing them usually makes CI both faster and less mysterious.