Permission Denied in a Container When the File Is Right There

The short answer

EACCES: permission denied, open '/app/data/cache.json'

The file exists and your process cannot touch it. Four causes:

1. The container runs as a non root UID that does not own the file
2. A mounted volume is owned by a different UID on the host
3. readOnlyRootFilesystem: true with no writable mount at that path
4. The file was copied by root in the Dockerfile and the runtime user cannot read it

Check what you actually are and what the file actually is:

kubectl exec mypod -- sh -c 'id; ls -la /app/data'

uid=1000 gid=3000 groups=3000
drwxr-xr-x 2 root root 4096 Nov 11 09:14 .

UID 1000 writing to a directory owned by root with mode 755. There is your answer.

Tested on Kubernetes 1.32 and Docker 27.5.

Cause 1: COPY makes files root owned

FROM node:22-slim
WORKDIR /app
COPY . .
RUN npm ci
USER node
CMD ["node", "server.js"]

Everything copied before USER node is owned by root. The node user can read world readable files and cannot write anywhere.

COPY --chown=node:node . .

Or set ownership explicitly before switching:

RUN mkdir -p /app/data && chown -R node:node /app
USER node

The --chown flag on COPY is the cleaner option because it does not add a layer.

Cause 2: volume ownership

A mounted volume brings its own ownership from the host or the storage backend, and it overrides whatever the image had at that path.

volumeMounts:
  - name: data
    mountPath: /app/data

If the PersistentVolume is owned by root and your pod runs as 1000, you cannot write.

Kubernetes has fsGroup for this:

securityContext:
  runAsUser: 1000
  runAsGroup: 3000
  fsGroup: 2000

fsGroup makes the kubelet chown the volume's contents to that group and set the setgid bit, so new files inherit it. This works for most volume types and not for all of them. NFS in particular ignores it, because ownership is enforced server side.

For a large volume, the recursive chown at mount time can take minutes and delays pod startup. fsGroupChangePolicy: OnRootMismatch only does it when the top level does not already match, which usually avoids the cost:

securityContext:
  fsGroup: 2000
  fsGroupChangePolicy: OnRootMismatch

With Docker bind mounts on Linux, the host's ownership passes straight through and there is no fsGroup equivalent. Either match the UID:

docker run -u "$(id -u):$(id -g)" -v "$PWD:/app" myimage

or chown on the host. Docker Desktop on macOS papers over this with its filesystem layer, which is why bind mount permissions work on a Mac and break on a Linux CI runner.

Cause 3: read only root filesystem

securityContext:
  readOnlyRootFilesystem: true

Good hardening, and it breaks anything that writes.

The surprise is how many things write to /tmp that you did not know about. Package managers, image libraries, some HTTP clients buffering large responses, font caches, and several language runtimes.

Mount writable space at every path that needs it:

securityContext:
  readOnlyRootFilesystem: true
volumeMounts:
  - name: tmp
    mountPath: /tmp
  - name: cache
    mountPath: /app/.cache
volumes:
  - name: tmp
    emptyDir: { sizeLimit: 256Mi }
  - name: cache
    emptyDir: { sizeLimit: 512Mi }

Always set sizeLimit on an emptyDir. Without it, a runaway process filling /tmp consumes the node's disk and affects every pod on that node, which turns your problem into everyone's problem.

To find what a process writes, run it without the read only flag and watch:

kubectl debug -it mypod --image=nicolaka/netshoot --target=app
strace -f -e trace=openat -p 1 2>&1 | grep -E 'O_WRONLY|O_RDWR|O_CREAT'

That lists every write attempt with its path, which is faster than discovering them one crash at a time. strace is good at exactly this kind of question.

Cause 4: capabilities, not ownership

Sometimes the file permissions are fine and the operation still fails.

Error: listen EACCES: permission denied 0.0.0.0:80

Binding to a port below 1024 requires CAP_NET_BIND_SERVICE. A non root container does not have it by default.

The right fix is to listen on a high port and map it:

ports:
  - containerPort: 8080

The service maps 80 to 8080. There is no reason for the container to bind a privileged port.

If you genuinely need it:

securityContext:
  capabilities:
    add: ["NET_BIND_SERVICE"]
    drop: ["ALL"]

Other capability related denials: chown on files you do not own needs CAP_CHOWN, raw sockets for ping need CAP_NET_RAW, and attaching a debugger needs CAP_SYS_PTRACE. That last one is why gdb -p and strace fail inside containers with an unhelpful message.

Diagnosing systematically

kubectl exec mypod -- sh -c '
  echo "--- identity"; id
  echo "--- target";   ls -la /app/data 2>&1
  echo "--- parent";   ls -ld /app 2>&1
  echo "--- mounts";   mount | grep -E "app|tmp"
  echo "--- caps";     cat /proc/self/status | grep -i cap
'

Check the parent directory as well as the file. Creating or deleting a file requires write permission on the directory, not on the file. A world readable file in a directory you cannot write to cannot be replaced, which produces a confusing "I can read it but not update it".

Also check for the sticky bit and setgid on shared directories, and whether SELinux or AppArmor is involved:

ls -Z /app/data          # SELinux context
dmesg | grep -i denied   # AppArmor and SELinux denials

On RHEL derived hosts, an SELinux denial produces Permission denied with correct Unix permissions, and the only evidence is in the audit log. Bind mounts need the :Z or :z suffix:

docker run -v /host/data:/app/data:Z myimage

Getting it right from the start

FROM node:22-slim

RUN groupadd -r app -g 1001 && useradd -r -u 1001 -g app app

WORKDIR /app
COPY --chown=app:app package*.json ./
RUN npm ci --omit=dev
COPY --chown=app:app . .

RUN mkdir -p /app/.cache && chown app:app /app/.cache

USER app
EXPOSE 8080
CMD ["node", "server.js"]

securityContext:
  runAsNonRoot: true
  runAsUser: 1001
  runAsGroup: 1001
  fsGroup: 1001
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]
volumeMounts:
  - { name: tmp, mountPath: /tmp }
volumes:
  - name: tmp
    emptyDir: { sizeLimit: 128Mi }

Pin the UID numerically rather than by name. Kubernetes runAsUser takes a number, and if the image's user has a different UID than you assumed, the mismatch produces exactly the errors in this post.

Test the hardened configuration in development rather than discovering it at deploy time. A container that works locally as root and fails in production as UID 1001 is the same category as any other environment difference, and the fix is to make development match.