ImagePullBackOff vs ErrImagePull: What Each One Means

Two statuses, one underlying problem, and about six causes. Here is how to read the event message and fix the right thing.

Share
ImagePullBackOff vs ErrImagePull: What Each One Means. Abstract kubernetes illustration in orange and dark grey on debugly.dev

The short answer

ErrImagePull is the first failure. Kubernetes tried to pull and it did not work.

ImagePullBackOff is what happens next. The kubelet is waiting before retrying, with exponential backoff up to five minutes. Same problem, later in its lifecycle.

The status tells you nothing about the cause. The event does:

kubectl describe pod <pod> | tail -20

Read the message on the Failed event. It names the actual problem: not found, unauthorized, no such host, or a timeout.

Tested on Kubernetes 1.32.

Read the event, not the status

Events:
  Type     Reason     Age                From     Message
  ----     ------     ----               ----     -------
  Normal   Pulling    2m                 kubelet  Pulling image "myreg.io/api:v1.4.2"
  Warning  Failed     2m                 kubelet  Failed to pull image "myreg.io/api:v1.4.2":
             rpc error: code = NotFound desc = failed to pull and unpack image
             "myreg.io/api:v1.4.2": failed to resolve reference: not found
  Warning  Failed     2m                 kubelet  Error: ErrImagePull
  Normal   BackOff    30s (x4 over 2m)   kubelet  Back-off pulling image "myreg.io/api:v1.4.2"
  Warning  Failed     30s (x4 over 2m)   kubelet  Error: ImagePullBackOff

The useful line is the first Failed. Everything after it is bookkeeping.

Message to cause mapping:

Message contains Cause
not found, manifest unknown Tag or repository does not exist
unauthorized, authentication required Missing or wrong credentials
no such host DNS cannot resolve the registry
connection refused, i/o timeout Network path to the registry blocked
denied, forbidden Credentials valid, permissions insufficient
toomanyrequests Rate limited, usually Docker Hub
no match for platform Image exists for a different architecture

The causes

1. The tag does not exist

Most common and most embarrassing. A typo, or a CI pipeline that failed to push before the deploy ran.

Verify from your own machine:

docker manifest inspect myreg.io/api:v1.4.2
crane manifest myreg.io/api:v1.4.2      # crane is nicer for this

If that fails with the same message, the image genuinely is not there and the problem is in your pipeline, not your cluster.

The race is worth calling out: a deploy that starts before the image push finishes produces exactly this, intermittently, and only under load when the build is slower. Make the deploy step depend on the push completing.

Never use :latest in production. With imagePullPolicy: Always you get unpredictable rollouts, and with IfNotPresent you get nodes running different versions of latest depending on when they last pulled. Pin a tag or a digest.

2. Missing or wrong pull secret

Failed to pull image: unauthorized: authentication required

Private registry, no credentials.

Create the secret:

kubectl create secret docker-registry regcred \
  --docker-server=myreg.io \
  --docker-username=<user> \
  --docker-password=<token> \
  --namespace=production

Reference it:

spec:
  imagePullSecrets:
    - name: regcred

Three things that catch people here.

Secrets are namespaced. A secret in default does nothing for a pod in production. This is the single most common version of this bug, especially when a deployment gets copied between namespaces.

--docker-server must match the registry host in your image reference exactly. For Docker Hub that is https://index.docker.io/v1/, which is not intuitive.

Attach it to the service account if you want it applied automatically to every pod in a namespace:

kubectl patch serviceaccount default -n production \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'

Verify the secret decodes to what you expect:

kubectl get secret regcred -n production \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq

3. Docker Hub rate limits

toomanyrequests: You have reached your pull rate limit

Anonymous pulls from Docker Hub are limited per IP. In a cluster, every node shares an egress IP, so a busy cluster hits the limit quickly. It presents as intermittent failures across unrelated pods, which is confusing until you recognise the message.

Fixes: authenticate even for public images, which raises the limit considerably; mirror the images you depend on into your own registry; or use a pull through cache.

Mirroring base images is worth doing regardless. Depending on a public registry being available during a deploy is a reliability risk you do not need to carry.

4. Architecture mismatch

no match for platform in manifest: not found

The image exists and not for your nodes' architecture. Built on an Apple Silicon machine and pushed a single arm64 manifest, deploying to amd64 nodes.

docker buildx imagetools inspect myreg.io/api:v1.4.2

If it lists one platform, that is the problem. The full explanation and the multi arch build setup is a separate post, because this shows up in three or four different disguises.

Worth noting the failure mode differs by where the mismatch is caught. If the registry has no matching manifest you get ImagePullBackOff. If the pull succeeds but the binary is wrong, you get CrashLoopBackOff with an exec format error instead.

5. Registry unreachable from the nodes

dial tcp: lookup myreg.io: no such host

Your laptop can reach the registry. The nodes cannot.

Test from inside the cluster rather than from your machine:

kubectl run nettest --rm -it --image=nicolaka/netshoot -- sh
nslookup myreg.io
curl -v https://myreg.io/v2/

Causes: a private registry on a VPC the nodes cannot route to, an egress network policy blocking it, a corporate proxy the container runtime is not configured for, or a cluster DNS problem.

Note that the container runtime's proxy configuration is separate from your pods'. If your nodes need a proxy to reach the internet, containerd needs to be told about it in its own config, not through pod environment variables.

6. Self signed or private CA

x509: certificate signed by unknown authority

An internal registry with a certificate the nodes do not trust. The CA has to be installed on every node, in the container runtime's trust store. This is a node configuration task, not something you can fix in a manifest, which surprises people.

Speeding up the retry

Once a pod is in ImagePullBackOff, the kubelet backs off up to five minutes. After you fix the underlying problem, nothing happens for a while and it looks like the fix did not work.

Delete the pod to force an immediate retry:

kubectl delete pod <pod>

For a deployment, a rollout restart is cleaner:

kubectl rollout restart deployment/<name>

This is the same pattern as CrashLoopBackOff, and it is worth internalising once: any Kubernetes backoff state means your fix will appear not to work for several minutes unless you force a retry.

Prevention

Pin by digest for anything critical. myreg.io/api@sha256:... is immutable and cannot be moved out from under you.

Verify the image exists before deploying. One line in the pipeline, right before the apply:

crane manifest "$IMAGE" > /dev/null || { echo "image $IMAGE not found"; exit 1; }

That converts a confusing cluster state into a clear pipeline failure with the image name in it.

Mirror your base images. Removes the Docker Hub rate limit and the availability dependency at once.

Put pull secrets on the service account so new deployments in a namespace inherit them and nobody has to remember.

Alert on pods not reaching Ready within a few minutes. ImagePullBackOff does not fire a crash alert and can sit unnoticed during a partial rollout, with old pods still serving traffic and the new ones never arriving.