Incident Response

Your application suddenly can't connect to its RDS database. Walk through how you'd diagnose it.

Rule out the database's own health first, since that's a two-second check in the console, then work through the network path in order: security groups, subnet and public accessibility, DNS, and connection limits. A sudden failure that used to work points hardest at something that changed recently, not at a fundamentally broken setup.

A pod is stuck in Pending for 10 minutes in production. Walk me through how you diagnose it.

Pending means the scheduler hasn't placed the pod on a node. The cause is almost always resources, node selection constraints, or an unbound volume, and `kubectl describe pod` tells you which within seconds if you read the Events section rather than guessing.

The deploy says it shipped main, but production doesn't have the latest commit. How do you debug it?

Walk the chain, commit, CI checkout, build, artifact, deploy, running pod, and compare the SHA at each step. main is a moving pointer, so the usual cause is that something in the chain resolved it at a different moment, or shipped a cached artifact instead of a fresh one.

Your CI uses git clone --depth=1 and a deploy script that needs history suddenly fails. Why?

A shallow clone downloads the current tree and exactly one commit, no parents, usually no tags. Anything that reads history breaks: `git describe`, changelogs, `git diff HEAD~10`, commit counts, `merge-base`. Fetch the depth you actually need instead of defaulting to depth 1 everywhere.

A developer wants to git reset --hard and force-push a shared branch to undo a bad commit. Do you allow it?

On a shared branch, no. Use git revert, which undoes the change with a new commit and leaves history intact. Reset plus force-push rewrites history that other people, CI and deployment records already depend on.

A bad feature was merged into production. How would you undo it?

Roll back the running deployment first, that's faster than any Git fix, then fix Git properly. Reverting a merge needs git revert -m 1 <merge-commit>, and the catch nobody mentions upfront is that you have to revert the revert later or the feature will never merge back in.

A production bug was introduced somewhere in the last 50 commits. How do you find the exact commit?

`git bisect` does a binary search over the range, so 50 commits take about 6 tests instead of 50. The hard part isn't the commands, it's having a reliable test that says good or bad, which is what lets you automate the whole thing with `git bisect run`.

Your CI pipeline runs twice for every pull request. How do you investigate?

Almost always two triggers firing on one action, usually push and pull_request both matching the same branch. Read the event that started each run, then make the trigger config deliberate instead of deleting jobs until the noise stops.

A developer says their commit has disappeared. How do you investigate and get it back?

Commits are rarely deleted, they usually just lose their branch reference. git reflog records every move of HEAD locally, so a bad reset, rebase or checkout is almost always recoverable, and git fsck --lost-found catches most of the rest.

An AWS Lambda function is triggered by an Amazon SQS queue. A small number of malformed messages cause the function to fail repeatedly. These messages are retried continuously, which delays processing of valid messages and increases cost. What should a solutions architect do to resolve this?

A message that can never succeed will be retried until it expires, blocking the queue behind it. The fix is a dead-letter queue with a maxReceiveCount redrive policy, which moves the poison message aside after a set number of failures so everything else keeps flowing.

An apply is failing on one broken resource and a colleague suggests routinely using -target to apply the rest. What does HashiCorp say about that flag, and why?

-target is documented for exceptional recovery situations, not routine use. It applies part of the graph, so the resulting state can be inconsistent with the configuration as a whole.

Requests to a ClusterIP Service time out. kubectl get endpointslices shows no endpoints for it, though the Pods are Running. What are the TWO most likely causes?

A Service with no endpoints means nothing matched or nothing is ready. Check that the selector matches the Pod labels, and that the Pods pass their readiness probes.

An engineer is unsure whether a for expression over a map produces the shape they expect, and wants to try it against real state without running a plan. Which tool does that?

terraform console opens an interactive shell for evaluating expressions against the current configuration and state. It is read-only, so it changes nothing while you experiment.

kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137, and a restart count climbing. What does that tell you, and what does not follow from it?

The container was killed for exceeding its memory limit, or the node ran out of memory. It is a container-level kill, so the Pod stays and the container restarts in place.

You need a shell alongside a running Pod to test connectivity, but the image is distroless and kubectl exec fails because there is no shell in it. What is the intended approach?

kubectl debug adds an ephemeral container to the running Pod, sharing its network namespace. That gives you a shell and tooling without rebuilding the image or restarting the Pod.

An application takes about 90 seconds to warm up. Its liveness probe uses the defaults with no initialDelaySeconds, and the Pod restarts continuously. What is the best fix?

The liveness probe is failing during startup and killing the container before it is ready. A startupProbe handles slow starts properly, holding the liveness probe off until startup succeeds.

A namespace holds forty Pods with status Evicted and no containers running. Replacement Pods are healthy on other nodes. What do these objects represent?

Evicted Pods are terminated Pod objects the kubelet left behind as a record of node pressure. They consume no resources, and deleting them is safe once you have read why they were evicted.

During a rolling update, some requests fail with connection errors even though every Pod eventually becomes healthy. Which mechanism prevents this, and how?

A readiness probe keeps a Pod out of a Service's endpoints until it can serve. Without one, a Pod receives traffic as soon as its container starts.

You need to check whether the API server is healthy and which of its internal checks is failing. Which approach reflects current practice?

The API server exposes livez and readyz, and healthz is deprecated. Adding verbose lists every individual check, which is how you find the one that is failing.

A worker node shows NotReady and its Pods are being replaced elsewhere. kubectl describe node reports the kubelet has stopped posting status. What do you check on the node itself?

The kubelet is a systemd service on the node. Check systemctl status kubelet and its logs with journalctl -u kubelet, which name the real failure.

A Pod cannot resolve any Service name. Other Pods in the cluster resolve names normally. Which check comes first?

With other Pods resolving fine, CoreDNS is healthy, so look at this Pod. Its /etc/resolv.conf and its dnsPolicy decide which resolver it uses at all.

On a node using containerd, you need to list containers and read one container's logs without going through the API server. Which tool is intended for this?

crictl is the CRI-compatible command line interface for inspecting containers on a node. It talks to the runtime directly, so it works when the control plane does not.

An application using the ServiceAccount ci in the build namespace gets a 403 listing Pods. Which command confirms the permission gap without deploying anything?

kubectl auth can-i with --as impersonates the identity and answers yes or no against the real authorization layer, so you can test a ServiceAccount's access from your own session.

A Pod's status is Init:0/1 and it has stayed that way for several minutes. Where is the problem, and how do you read its output?

An init container has not completed, and the app containers cannot start until it does. Read its logs by naming it with kubectl logs -c.