Debugging production, running incidents and writing them up.
30 items · all topics
An AWS Lambda function is triggered by an Amazon SQS queue. A small number of malformed messages cause the function to fail repeatedly. These messages are retried continuously, which delays processing of valid messages and increases cost. What should a solutions architect do to resolve this?
A message that can never succeed will be retried until it expires, blocking the queue behind it. The fix is a dead-letter queue with a maxReceiveCount redrive policy, which moves the poison message aside after a set number of failures so everything else keeps flowing.
An apply is failing on one broken resource and a colleague suggests routinely using -target to apply the rest. What does HashiCorp say about that flag, and why?
-target is documented for exceptional recovery situations, not routine use. It applies part of the graph, so the resulting state can be inconsistent with the configuration as a whole.
Requests to a ClusterIP Service time out. kubectl get endpointslices shows no endpoints for it, though the Pods are Running. What are the TWO most likely causes?
A Service with no endpoints means nothing matched or nothing is ready. Check that the selector matches the Pod labels, and that the Pods pass their readiness probes.
A PVC with no storageClassName field is applied to a cluster and stays Pending. Which check explains it fastest?
A PVC that omits storageClassName uses the default StorageClass, marked by the storageclass.kubernetes.io/is-default-class annotation. With no default and no matching PV, the claim waits.
A Pod has been Pending for ten minutes. Which single command gives you the reason, and what should you expect to read in it?
kubectl describe pod shows the scheduler's FailedScheduling event, which names the reason node by node: insufficient CPU or memory, an untolerated taint, or no node matching the selector.
kubectl get pods shows a Pod with STATUS Running and READY 1/2, and requests to its Service are failing. What does that pair of columns tell you?
READY counts ready containers out of total containers in the Pod. At 1/2 the Pod is not ready, so it is left out of Service endpoints even though STATUS says Running.
A Pod is in ImagePullBackOff. Which TWO causes are consistent with that status?
ImagePullBackOff means the kubelet cannot fetch the image. A wrong name or tag and a missing registry credential are the two usual reasons, and the Pod's events name which one.
A Pod will not start and you need both the scheduling events and the exact resource requests as submitted. Which two commands give you those, and what does each leave out?
describe gives a human summary plus recent Events, which the YAML never contains. get -o yaml gives the exact stored object including defaults, which describe abbreviates.
A container is in CrashLoopBackOff and kubectl logs returns nothing useful because the container has just restarted. Which command shows the output from the failed run?
kubectl logs --previous returns the logs of the previous instantiation of the container, which is where the reason for the crash actually is.
A Pod is scheduled to a node but its status reads CreateContainerConfigError. The image pulled fine. What class of problem is this, and where do you look?
The kubelet could not assemble the container's configuration, usually because a referenced ConfigMap, Secret or key does not exist. The Pod events name the missing object.
An engineer is unsure whether a for expression over a map produces the shape they expect, and wants to try it against real state without running a plan. Which tool does that?
terraform console opens an interactive shell for evaluating expressions against the current configuration and state. It is read-only, so it changes nothing while you experiment.
kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137, and a restart count climbing. What does that tell you, and what does not follow from it?
The container was killed for exceeding its memory limit, or the node ran out of memory. It is a container-level kill, so the Pod stays and the container restarts in place.
You need a shell alongside a running Pod to test connectivity, but the image is distroless and kubectl exec fails because there is no shell in it. What is the intended approach?
kubectl debug adds an ephemeral container to the running Pod, sharing its network namespace. That gives you a shell and tooling without rebuilding the image or restarting the Pod.
An application takes about 90 seconds to warm up. Its liveness probe uses the defaults with no initialDelaySeconds, and the Pod restarts continuously. What is the best fix?
The liveness probe is failing during startup and killing the container before it is ready. A startupProbe handles slow starts properly, holding the liveness probe off until startup succeeds.
A namespace holds forty Pods with status Evicted and no containers running. Replacement Pods are healthy on other nodes. What do these objects represent?
Evicted Pods are terminated Pod objects the kubelet left behind as a record of node pressure. They consume no resources, and deleting them is safe once you have read why they were evicted.
During a rolling update, some requests fail with connection errors even though every Pod eventually becomes healthy. Which mechanism prevents this, and how?
A readiness probe keeps a Pod out of a Service's endpoints until it can serve. Without one, a Pod receives traffic as soon as its container starts.
You need to check whether the API server is healthy and which of its internal checks is failing. Which approach reflects current practice?
The API server exposes livez and readyz, and healthz is deprecated. Adding verbose lists every individual check, which is how you find the one that is failing.
A worker node shows NotReady and its Pods are being replaced elsewhere. kubectl describe node reports the kubelet has stopped posting status. What do you check on the node itself?
The kubelet is a systemd service on the node. Check systemctl status kubelet and its logs with journalctl -u kubelet, which name the real failure.
After editing the kube-apiserver static Pod manifest, kubectl fails with a connection refused error. Which approach diagnoses this?
With the API server down, kubectl is useless. Use crictl on the control plane node to find the container and read its logs, and check the kubelet journal for manifest errors.
A Pod writes a large volume of logs. kubectl logs returns only recent output, and the earlier lines you need are missing. Why, and what does that imply?
The kubelet rotates container logs and kubectl logs only reads the latest file. Keeping history means shipping logs off the node to a cluster-level logging system.
You are asked to find out what happened in a namespace over the last few minutes. Which command gives the most useful ordered picture, and what limitation should you expect?
kubectl get events --sort-by=.lastTimestamp gives a namespace timeline. Events are namespaced and short-lived, retained for one hour by default, so older history is simply gone.
kubectl top nodes fails with an error saying the metrics API is not available, though every node is Ready and workloads are healthy. What does that indicate?
kubectl top reads the Metrics API, which is served by metrics-server rather than by the API server itself. Without it installed there is nothing to answer, and resource usage has to come from elsewhere.
Pods on one node show status Evicted and the node reports the DiskPressure condition. Which explanation is correct?
The kubelet evicts Pods when a node-level resource crosses an eviction threshold. DiskPressure comes from node or image filesystem thresholds, whose defaults are 10% and 15% available.
A Pod cannot resolve any Service name. Other Pods in the cluster resolve names normally. Which check comes first?
With other Pods resolving fine, CoreDNS is healthy, so look at this Pod. Its /etc/resolv.conf and its dnsPolicy decide which resolver it uses at all.
A Pod has been Terminating for fifteen minutes after a delete. What is the correct sequence of things to consider?
Check whether the process is ignoring SIGTERM within its grace period, whether the node is unreachable, and whether a finalizer is holding the object. Force deletion is a last resort.
On a node using containerd, you need to list containers and read one container's logs without going through the API server. Which tool is intended for this?
crictl is the CRI-compatible command line interface for inspecting containers on a node. It talks to the runtime directly, so it works when the control plane does not.
An application using the ServiceAccount ci in the build namespace gets a 403 listing Pods. Which command confirms the permission gap without deploying anything?
kubectl auth can-i with --as impersonates the identity and answers yes or no against the real authorization layer, so you can test a ServiceAccount's access from your own session.
A Pod that was Running is suddenly gone, and events show the scheduler removed it to make room for a higher-priority Pod. Which mechanism is this, and what protects against it?
This is preemption, driven by PriorityClass. The scheduler removes lower priority Pods so a pending higher priority Pod can be scheduled, and preemptionPolicy: Never opts a Pod out of preempting others.
On a fresh install, kubectl get nodes fails with a message about the connection to the server localhost:8080 being refused. What does that specific message mean?
localhost:8080 is kubectl's fallback when it finds no kubeconfig. The fix is to point it at a real one, usually by copying admin.conf into ~/.kube/config.
A Pod's status is Init:0/1 and it has stayed that way for several minutes. Where is the problem, and how do you read its output?
An init container has not completed, and the app containers cannot start until it does. Read its logs by naming it with kubectl logs -c.