Docker layer cache poisoning: the build that kept shipping a stale bundle from a cache mount nobody reset
Problem
A release image built from fresh source serves a JavaScript bundle from nine days earlier. The build log shows every step running — compile included — and the artifact is still old:
prod image contains bundle from 2026-08-19 (hash a3f9c1) — source at build time was 2026-08-28
docker build log: CACHED [builder 5/6] RUN pnpm build (cache hit on /repo/.pnpm-store + dist cache mount)No --cache-from cross-machine magic, no base image confusion. The cache was poisoned locally and every subsequent build inherited it.
Root cause
The Dockerfile used BuildKit cache mounts to speed up the build:
# the poison: 'dist' cached across builds, keyed on nothing
RUN --mount=type=cache,target=/repo/.pnpm-store \
--mount=type=cache,target=/repo/dist \
pnpm install --frozen-lockfile && pnpm buildCache mounts persist their contents across builds by design, keyed only by the mount target path — they have no input fingerprint. Someone once added a script that wrote into /repo/dist directly during a local debugging session (or a build was interrupted mid-write, leaving a partial artifact). Every later build: pnpm build saw the mount already populated, its tooling skipped unchanged-looking inputs or merged with the stale entries, and the final COPY dist picked up the old bundle. Classic COPY-layer caching (the other Docker cache failure) is invalidated by changed inputs; cache mounts are never invalidated by anything, which is why the stale artifact outlived correct source.
# fixed: cache only the package store; the artifact is rebuilt fresh every time
RUN --mount=type=cache,target=/root/.local-share/pnpm/store \
pnpm install --frozen-lockfile
… 3 more lines in the fix🔒 the fix — including 3 code blocks — is members-only. $1/mo unlocks everything.