Docker

Debugging a Container With No Shell, No curl, and No ps

Distroless and scratch images are great for security and miserable when something breaks. Here is how to investigate a container that ships nothing.

The short answer

$ kubectl exec -it mypod -- sh
OCI runtime exec failed: exec: "sh": executable file not found in $PATH

Your image is distroless or scratch. There is no shell to exec into. Use an ephemeral debug container, which attaches a container with tools to the running pod's namespaces without changing the image:

kubectl debug -it mypod --image=nicolaka/netshoot --target=app

For plain Docker:

docker run -it --rm --pid=container:mycontainer \
  --network=container:mycontainer \
  --cap-add=SYS_PTRACE nicolaka/netshoot

Tested on Kubernetes 1.32 and Docker 27.5.

Why your image has nothing in it

Distroless images ship your application and its runtime dependencies, and nothing else. No shell, no package manager, no coreutils, no ps, no curl.

This is genuinely good practice. Most container vulnerabilities are in packages the application never uses, and an attacker who gets code execution in a distroless container has no shell to pivot with. Image size drops dramatically too.

The cost is that every debugging habit you have depends on tools that are no longer there.

Ephemeral debug containers

This is the right answer and it is underused because people do not know it exists.

kubectl debug starts a new container inside the existing pod, sharing namespaces with a target container. Your image stays untouched. The debug container disappears when you are done.

kubectl debug -it payments-7f9c-x2ktp \
  --image=nicolaka/netshoot \
  --target=payments-api

The --target flag is the important part. Without it you share only the network namespace. With it you also share the process namespace, so you can see and inspect the actual application process.

Once inside, the application's filesystem is not directly at /, because mount namespaces are not shared. Reach it through proc:

ls -la /proc/1/root/          # the target's filesystem
cat /proc/1/environ | tr '\0' '\n'    # its environment variables
cat /proc/1/cmdline | tr '\0' ' '     # how it was started
ls -l /proc/1/fd/             # open file descriptors
cat /proc/1/limits            # ulimits

/proc/1/root is the single most useful path here. It gives you the target container's entire filesystem, readable with the tools in your debug image.

If the process is not PID 1, find it with ps aux in the debug container, since you share the PID namespace.

For a copy of the pod with a shell, when you want to change the entrypoint too:

kubectl debug mypod -it --copy-to=mypod-debug --container=app -- sh

That creates a duplicate pod rather than touching the running one, which is the safer option when the pod is serving traffic.

What to actually run once you are in

netshoot is the debug image worth remembering. It bundles curl, dig, tcpdump, strace, ss, iperf, jq, and most of what you would want. nicolaka/netshoot is the standard.

Common investigations:

# is the app listening where you think
ss -lntp

# can it reach a dependency
curl -v http://postgres.default.svc.cluster.local:5432
dig +short api.internal

# what is it doing at the syscall level
strace -f -T -p 1 -e trace=network

# what is the actual traffic
tcpdump -i any -n port 5432 -c 20

# what is using memory, from the cgroup rather than the app
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.stat

That last pair is worth knowing because it works regardless of what your image contains. The cgroup filesystem is the kernel's view, not the container's, and it is the number that actually gets you OOMKilled.

When you cannot use ephemeral containers

Some clusters disable the feature, and some environments are not Kubernetes at all.

Docker, sharing namespaces:

docker run -it --rm \
  --pid=container:myapp \
  --network=container:myapp \
  --cap-add=SYS_PTRACE \
  nicolaka/netshoot

SYS_PTRACE is what lets you attach strace or a debugger to the target process. Without it you can see the process and not inspect it.

Copy files out instead of going in. docker cp works on stopped containers and needs no shell in the image:

docker cp myapp:/app/config.yaml ./config.yaml
docker cp myapp:/var/log/. ./logs/

Same in Kubernetes, though kubectl cp does require tar in the image, which distroless does not have. Use kubectl debug and read through /proc/1/root instead.

Inspect the image without running it. Often the question is what is actually in this image, and you can answer that on your own machine:

# export the filesystem to a tarball
docker create --name tmp myimage
docker export tmp | tar -tv | head -50
docker rm tmp

# or use dive to browse layers interactively
dive myimage

dive is excellent for the question "why is this file missing" and for finding which layer added a surprise.

Temporarily switch the entrypoint. In Kubernetes you can patch a deployment to run something that just blocks, so the container starts and you can attach:

command: ["/bin/sleep"]
args: ["3600"]

Except distroless has no /bin/sleep either. This works only if your runtime has something equivalent, so it is a fallback rather than a plan.

Building a debuggable image without giving up distroless

Two techniques that keep production lean.

Multi stage with a debug target. Keep a second final stage that adds tools, and build it only when needed:

FROM gcr.io/distroless/base-debian12 AS runtime
COPY --from=build /out/server /app/server
ENTRYPOINT ["/app/server"]

FROM runtime AS debug
COPY --from=busybox:1.36 /bin/busybox /bin/busybox
COPY --from=busybox:1.36 /bin/sh /bin/sh

docker build --target debug -t myapp:debug .

Production ships runtime. When something is wrong, you deploy myapp:debug with the same application binary and layers, so you are debugging the real thing rather than an approximation.

Use the distroless debug tags. Google publishes :debug variants of distroless images that include busybox:

FROM gcr.io/distroless/nodejs22-debian12:debug

Same base, same libraries, plus a shell at /busybox/sh. Swapping the tag in an emergency is a one line change.

The things that keep working

Worth remembering that a distroless container has not removed your best sources of information.

Logs. kubectl logs --previous still works and still shows you the run that died.

Exit codes. kubectl describe pod gives you Last State and the exit code, which narrows a CrashLoopBackOff to a handful of causes immediately.

Events. kubectl get events --sort-by=.lastTimestamp shows probe failures, evictions, and image pull problems.

Metrics and traces. If the app is instrumented, that data is leaving the container regardless of what the image contains.

The general lesson from working on isolated environments is that the more locked down a runtime is, the more you have to design its observability up front. If your production containers cannot be shelled into, then logs, metrics, and structured errors are not nice to have. They are the entire interface, and an error message that cannot be acted on becomes very expensive in that setting.

A workable order

kubectl logs --previous and kubectl describe pod first, because they need nothing
kubectl debug with netshoot and --target
Read /proc/1/root, /proc/1/environ, and the cgroup files
If it is a network problem, ss, dig, tcpdump from the shared network namespace
If you need the binary's behaviour, strace -p 1 with SYS_PTRACE
If none of that resolves it, redeploy the :debug variant of the same image