Debugging production, running incidents and writing them up.
15 items at intermediate level · all topics
An AWS Lambda function is triggered by an Amazon SQS queue. A small number of malformed messages cause the function to fail repeatedly. These messages are retried continuously, which delays processing of valid messages and increases cost. What should a solutions architect do to resolve this?
A message that can never succeed will be retried until it expires, blocking the queue behind it. The fix is a dead-letter queue with a maxReceiveCount redrive policy, which moves the poison message aside after a set number of failures so everything else keeps flowing.
An apply is failing on one broken resource and a colleague suggests routinely using -target to apply the rest. What does HashiCorp say about that flag, and why?
-target is documented for exceptional recovery situations, not routine use. It applies part of the graph, so the resulting state can be inconsistent with the configuration as a whole.
Requests to a ClusterIP Service time out. kubectl get endpointslices shows no endpoints for it, though the Pods are Running. What are the TWO most likely causes?
A Service with no endpoints means nothing matched or nothing is ready. Check that the selector matches the Pod labels, and that the Pods pass their readiness probes.
An engineer is unsure whether a for expression over a map produces the shape they expect, and wants to try it against real state without running a plan. Which tool does that?
terraform console opens an interactive shell for evaluating expressions against the current configuration and state. It is read-only, so it changes nothing while you experiment.
kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137, and a restart count climbing. What does that tell you, and what does not follow from it?
The container was killed for exceeding its memory limit, or the node ran out of memory. It is a container-level kill, so the Pod stays and the container restarts in place.
You need a shell alongside a running Pod to test connectivity, but the image is distroless and kubectl exec fails because there is no shell in it. What is the intended approach?
kubectl debug adds an ephemeral container to the running Pod, sharing its network namespace. That gives you a shell and tooling without rebuilding the image or restarting the Pod.
An application takes about 90 seconds to warm up. Its liveness probe uses the defaults with no initialDelaySeconds, and the Pod restarts continuously. What is the best fix?
The liveness probe is failing during startup and killing the container before it is ready. A startupProbe handles slow starts properly, holding the liveness probe off until startup succeeds.
A namespace holds forty Pods with status Evicted and no containers running. Replacement Pods are healthy on other nodes. What do these objects represent?
Evicted Pods are terminated Pod objects the kubelet left behind as a record of node pressure. They consume no resources, and deleting them is safe once you have read why they were evicted.
During a rolling update, some requests fail with connection errors even though every Pod eventually becomes healthy. Which mechanism prevents this, and how?
A readiness probe keeps a Pod out of a Service's endpoints until it can serve. Without one, a Pod receives traffic as soon as its container starts.
You need to check whether the API server is healthy and which of its internal checks is failing. Which approach reflects current practice?
The API server exposes livez and readyz, and healthz is deprecated. Adding verbose lists every individual check, which is how you find the one that is failing.
A worker node shows NotReady and its Pods are being replaced elsewhere. kubectl describe node reports the kubelet has stopped posting status. What do you check on the node itself?
The kubelet is a systemd service on the node. Check systemctl status kubelet and its logs with journalctl -u kubelet, which name the real failure.
A Pod cannot resolve any Service name. Other Pods in the cluster resolve names normally. Which check comes first?
With other Pods resolving fine, CoreDNS is healthy, so look at this Pod. Its /etc/resolv.conf and its dnsPolicy decide which resolver it uses at all.
On a node using containerd, you need to list containers and read one container's logs without going through the API server. Which tool is intended for this?
crictl is the CRI-compatible command line interface for inspecting containers on a node. It talks to the runtime directly, so it works when the control plane does not.
An application using the ServiceAccount ci in the build namespace gets a 403 listing Pods. Which command confirms the permission gap without deploying anything?
kubectl auth can-i with --as impersonates the identity and answers yes or no against the real authorization layer, so you can test a ServiceAccount's access from your own session.
A Pod's status is Init:0/1 and it has stayed that way for several minutes. Where is the problem, and how do you read its output?
An init container has not completed, and the app containers cannot start until it does. Read its logs by naming it with kubectl logs -c.