Kubernetes Fundamentals · Module 13 of 14

Troubleshooting Pending Pods, crashes and Service failures

Use a repeatable evidence-first workflow instead of guessing: get state, describe events, inspect logs, validate selectors/endpoints and then change one cause at a time.

Learning objectives

By the end of this module, you should be able to:

Diagnose a deliberately Pending Pod.
Differentiate scheduling, image/runtime and application failures.
Troubleshoot a Service with no matching endpoints.

Start with observed state

kubectl get tells you what state Kubernetes reports. kubectl describe adds events and configuration. logs explains container output. For Services, inspect EndpointSlices to confirm that selectors actually resolve to ready Pods.

Pending often means placement or setup is incomplete

A Pod can remain Pending because the scheduler cannot find an eligible node, required storage is not bound, or image/sandbox setup is still incomplete. Events usually narrow the reason quickly.

CrashLoopBackOff is a symptom

CrashLoopBackOff describes repeated container failure with backoff between restart attempts. The cause comes from logs, previous logs, exit codes, configuration and dependencies.

Service failures are often selector/readiness failures

A healthy Service object with no endpoints cannot route to application Pods. Verify labels/selectors and readiness before debugging DNS or kube-proxy.

Worked examples

See the idea in practice.

Evidence-first triage

kubectl -n gnu-k8s-fundamentals get pods -o wide
kubectl -n gnu-k8s-fundamentals describe pod NAME
kubectl -n gnu-k8s-fundamentals logs NAME --previous
kubectl -n gnu-k8s-fundamentals get svc,endpointslices

State first, then Events, then process output.
--previous is useful after a container restarted.

Hands-on lab

Lab — Diagnose without guessing

Goal: Create one intentionally unschedulable Pod and one Service with a selector mismatch, prove each root cause, then fix/clean up.

Before you start

kubectl is installed and can reach the lab cluster.
Use the dedicated gnu-k8s-fundamentals namespace unless a step explicitly says otherwise.

STEP 1

Create an intentionally Pending Pod

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pending-demo
  namespace: gnu-k8s-fundamentals
spec:
  nodeSelector:
    training.gnugroup.org/nonexistent: "true"
  containers:
  - name: app
    image: busybox:1.36
    command: ["sleep","3600"]
EOF
sleep 3
kubectl -n gnu-k8s-fundamentals get pod pending-demo
kubectl -n gnu-k8s-fundamentals describe pod pending-demo | tail -30

STEP 2

Create backend and mismatched Service

kubectl -n gnu-k8s-fundamentals create deployment troubleshoot-web --image=nginx:alpine
kubectl -n gnu-k8s-fundamentals rollout status deployment/troubleshoot-web --timeout=120s
kubectl -n gnu-k8s-fundamentals create service clusterip broken-svc --tcp=80:80
kubectl -n gnu-k8s-fundamentals patch service broken-svc -p '{"spec":{"selector":{"app":"does-not-exist"}}}'
kubectl -n gnu-k8s-fundamentals get endpointslices -l kubernetes.io/service-name=broken-svc

STEP 3

Fix the selector

kubectl -n gnu-k8s-fundamentals patch service broken-svc -p '{"spec":{"selector":{"app":"troubleshoot-web"}}}'
kubectl -n gnu-k8s-fundamentals get endpointslices -l kubernetes.io/service-name=broken-svc

STEP 4

Clean up

kubectl -n gnu-k8s-fundamentals delete pod pending-demo
kubectl -n gnu-k8s-fundamentals delete service broken-svc
kubectl -n gnu-k8s-fundamentals delete deployment troubleshoot-web

Verify

pending-demo remains Pending because no node matches the impossible selector.
broken-svc initially has no useful endpoints.
After fixing the selector, EndpointSlice addresses appear.

Expected outcome

The lab demonstrates two different control-plane-visible root causes without random restarts.

If it fails

If Events are truncated, use kubectl get events --sort-by=.lastTimestamp.
On shared clusters, do not label production nodes merely to satisfy a lab; delete the intentionally Pending Pod instead.

Real-world connection

Production incident response improves when operators follow a fixed evidence hierarchy instead of changing multiple variables simultaneously.

Avoid these traps

Common mistakes

Restarting everything before collecting evidence.
Treating every Pending Pod as a container image problem.
Debugging DNS before checking whether the Service has endpoints.

Knowledge check

Can you explain it without looking back?

Open each answer only after you have tried to answer the question yourself.

1Where should you look for scheduling failure reasons?

Pod Events, typically through kubectl describe pod.

2What does CrashLoopBackOff tell you?

The container is repeatedly failing and Kubernetes is backing off restarts; it does not identify the root cause.

3A Service exists but has no endpoints. What is an early check?

Compare its selector with Pod labels and readiness.