Incident Response

Your application suddenly can't connect to its RDS database. Walk through how you'd diagnose it.

Rule out the database's own health first, since that's a two-second check in the console, then work through the network path in order: security groups, subnet and public accessibility, DNS, and connection limits. A sudden failure that used to work points hardest at something that changed recently, not at a fundamentally broken setup.

A pod is stuck in Pending for 10 minutes in production. Walk me through how you diagnose it.

Pending means the scheduler hasn't placed the pod on a node. The cause is almost always resources, node selection constraints, or an unbound volume, and `kubectl describe pod` tells you which within seconds if you read the Events section rather than guessing.

What dashboards do you open first during an incident?

Good responders have a rehearsed order: service health to find the blast radius, golden signals to work out what kind of failure it is, then traces to localise it. Opening application logs first is usually the tell that someone doesn't know their own observability stack.

How do you know whether an issue is infrastructure or application?

The shape of the metrics answers this before you have to think hard. Errors up with latency and resources flat is almost always application. Latency up with a resource saturated is infrastructure. And whether one pod is affected or all of them is usually the single most decisive fact available.

What metrics do you check before SSHing into an instance?

Metrics give you the fleet; SSH gives you one box. Check CPU, memory, disk, network and load across instances first, that tells you which instance is actually the problem, and often tells you the answer outright. Go to the shell only once you've narrowed it down and need something metrics can't show you.

Describe the last Sev-1 incident you handled.

This question isn't really about the outage, it's about how you behave during one. A good answer shows metric-driven diagnosis, mitigation running in parallel with investigation, real numbers, and a lesson that changed a system rather than blamed a person. Vague heroics score badly; a specific timeline scores well.

How do you investigate a latency spike that lasted only five minutes?

A spike that heals itself was self-limiting: a run of GC pauses, a cache stampede, a brief dependency blip, or a retry storm that burned out. It's over by the time you look, so the skill is knowing what to capture in the first thirty seconds and what to compare it against once the panic has passed.

How do you identify the exact deployment that introduced an issue?

Put the deployment log and the metrics timeline side by side. A regression that started fifteen minutes ago was almost certainly caused by something that shipped in that window. Confirm which commit is actually running from the image digest, then roll back and watch whether the metrics recover, that is your proof.

How do you safely rotate IAM credentials or database secrets without downtime?

The safe pattern is overlapping validity. Create the new secret while the old one still works, get code that accepts either one deployed everywhere, confirm the new one is actually being used, then revoke the old one. Rotating in a single step guarantees a window where something is still holding a credential that no longer works.

What happens if an Auto Scaling instance never becomes healthy?

Auto Scaling terminates and replaces an instance that never passes health checks, which is fine once and expensive in a loop. How long you burn depends on the grace period, the check thresholds, and which health check type the group is actually using. The fix is working out why it's unhealthy: usually startup config, a wrong health endpoint, or a grace period shorter than boot time.

p99 latency on a critical API jumped from 120ms to 3s an hour ago. p50 is unchanged. How do you find the cause?

An unchanged p50 with a blown p99 means most requests are fine and a specific subset isn't, which rules out broad causes like CPU saturation and points at something correlated: one dependency, one shard, one node, one customer, or garbage collection. Unlike a spike that heals on its own in a few minutes, this one is still happening, so the shape of the distribution is the first clue and the investigation runs in parallel with mitigation.

How would you debug an issue that only happens under production traffic and cannot be reproduced in staging?

Production-only issues are almost always data volume, traffic volume, latency, or environment drift from staging. The job is to make production observable without making it worse: sample or mirror real traffic, use feature flags to isolate the suspect, turn on verbose logging briefly. Hammering staging harder is usually wasted time if staging was never shaped like production.

You accidentally committed a .env file containing API keys. What do you do?

Rotate the credentials first. That's the only step that actually makes you safe, since the secret is already in every clone, fork and CI cache that pulled before you noticed. Cleaning up Git history is a second, separate job that comes after.

A developer says they pulled the latest code but their branch doesn't match the remote. How do you investigate?

It's almost always a tracking problem, not corruption. They pulled a different branch than they think, or their branch tracks something other than what they assume. `git branch -vv` and `git log HEAD..origin/main` usually answer it in two commands.

Code works on the developer's machine but fails in Jenkins. How could Git be involved?

Stop guessing about the code and compare commit SHAs first. CI often builds a different commit than the developer tested: a merge commit, a stale workspace, a shallow clone, missing submodules, or a file that's gitignored locally but needed at build time. Only once the SHAs match is it worth looking at the code.

A Git repository has become huge and cloning takes 20 minutes. What would you investigate?

Measure before you guess. Find the biggest objects in history, because size usually comes from binaries and build artifacts committed long ago. Deleting them today does nothing: the old blobs stay in history until you rewrite it or route developers around downloading them in the first place.

What is the difference between git fetch and git pull?

Fetch downloads remote commits and updates your remote-tracking branches, but never touches your working tree. Pull is fetch plus merge or rebase, so it changes your branch. Fetch is the safer move when you're debugging or scripting.

The deploy says it shipped main, but production doesn't have the latest commit. How do you debug it?

Walk the chain, commit, CI checkout, build, artifact, deploy, running pod, and compare the SHA at each step. main is a moving pointer, so the usual cause is that something in the chain resolved it at a different moment, or shipped a cached artifact instead of a fresh one.

Your CI uses git clone --depth=1 and a deploy script that needs history suddenly fails. Why?

A shallow clone downloads the current tree and exactly one commit, no parents, usually no tags. Anything that reads history breaks: `git describe`, changelogs, `git diff HEAD~10`, commit counts, `merge-base`. Fetch the depth you actually need instead of defaulting to depth 1 everywhere.

A developer wants to git reset --hard and force-push a shared branch to undo a bad commit. Do you allow it?

On a shared branch, no. Use git revert, which undoes the change with a new commit and leaves history intact. Reset plus force-push rewrites history that other people, CI and deployment records already depend on.

A bad feature was merged into production. How would you undo it?

Roll back the running deployment first, that's faster than any Git fix, then fix Git properly. Reverting a merge needs git revert -m 1 <merge-commit>, and the catch nobody mentions upfront is that you have to revert the revert later or the feature will never merge back in.

A production bug was introduced somewhere in the last 50 commits. How do you find the exact commit?

`git bisect` does a binary search over the range, so 50 commits take about 6 tests instead of 50. The hard part isn't the commands, it's having a reliable test that says good or bad, which is what lets you automate the whole thing with `git bisect run`.

Your CI pipeline runs twice for every pull request. How do you investigate?

Almost always two triggers firing on one action, usually push and pull_request both matching the same branch. Read the event that started each run, then make the trigger config deliberate instead of deleting jobs until the noise stops.

A developer says their commit has disappeared. How do you investigate and get it back?

Commits are rarely deleted, they usually just lose their branch reference. git reflog records every move of HEAD locally, so a bad reset, rebase or checkout is almost always recoverable, and git fsck --lost-found catches most of the rest.

A deploy of commit A is still running when commit B lands on main. What can go wrong?

If the pipeline resolves `main` at each step instead of pinning one commit early, later stages can pick up B while earlier ones tested A. You get mixed versions, out-of-order deploys, and a rollback target that no longer means anything. The fix is pinning the SHA once plus serializing production deploys.

A secret was committed six months ago and exists in hundreds of commits. What do you do?

Treat it as a security incident, not a Git cleanup task. Rotate first, then work out the blast radius, then decide honestly whether rewriting history is worth the cost. The rewrite is the most visible part of the response and the least important one.

Someone force-pushed a branch and deleted important commits. How do you recover them?

The commits almost certainly still exist, they just have nothing pointing at them anymore. Find the old SHA from any clone, CI workspace, PR page or provider event log, then create a branch on it. Act the same day, because garbage collection is the real deadline.

A critical production bug needs an emergency fix, but your PR process takes two hours. What do you do?

Mitigate first: rollback or a feature flag beats writing code under pressure. If code is genuinely needed, use a documented hotfix lane: branch from the production tag, minimal fix, fast tests, one reviewer, deploy, then merge back. Emergency means a faster controlled process, not no process.

You use GitOps and someone changes Kubernetes manually. Git still has the old config. What happens?

You get configuration drift: Git says one thing, the cluster says another. What happens next depends on whether the controller self-heals or just reports it. Either way the manual change is temporary, and the fix is to put the intended change into Git, not to argue with the controller.

Argo CD keeps reverting your emergency production change. Why, and what should you do?

Self-heal is doing exactly what it was configured to do: pulling the cluster back to what Git says. During an incident, mitigate through something the controller doesn't manage, then get the real change into Git fast rather than fighting reconciliation head on.

Deployment says SUCCESS but production is running an older commit. You have 10 minutes. What do you check?

Ask the running process what it actually is, then walk backwards through the chain until the SHA stops matching. A green pipeline only proves each step exited zero, not that anything actually changed in production, and "unchanged" is a success message that means nothing happened at all.

Git is the source of truth, but production differs from Git. How do you prove where the drift happened?

Compare state at each stage, Git, rendered manifests, what was applied, and the live cluster, and use Kubernetes' own metadata to name the culprit. managedFields records which controller last wrote each field, which usually answers the question outright without any guessing.

Someone force-pushed main at 2 AM. How do you investigate?

Preserve evidence first, then answer four questions: what was main before, what is it now, who did it, and was anything deployed from the rewritten history. Treat it as potentially malicious until the audit log says otherwise, because a force-push at 2 AM is an unusual enough event to earn that default.

An AWS Lambda function is triggered by an Amazon SQS queue. A small number of malformed messages cause the function to fail repeatedly. These messages are retried continuously, which delays processing of valid messages and increases cost. What should a solutions architect do to resolve this?

A message that can never succeed will be retried until it expires, blocking the queue behind it. The fix is a dead-letter queue with a maxReceiveCount redrive policy, which moves the poison message aside after a set number of failures so everything else keeps flowing.

An apply is failing on one broken resource and a colleague suggests routinely using -target to apply the rest. What does HashiCorp say about that flag, and why?

-target is documented for exceptional recovery situations, not routine use. It applies part of the graph, so the resulting state can be inconsistent with the configuration as a whole.

Requests to a ClusterIP Service time out. kubectl get endpointslices shows no endpoints for it, though the Pods are Running. What are the TWO most likely causes?

A Service with no endpoints means nothing matched or nothing is ready. Check that the selector matches the Pod labels, and that the Pods pass their readiness probes.

A PVC with no storageClassName field is applied to a cluster and stays Pending. Which check explains it fastest?

A PVC that omits storageClassName uses the default StorageClass, marked by the storageclass.kubernetes.io/is-default-class annotation. With no default and no matching PV, the claim waits.

A Pod has been Pending for ten minutes. Which single command gives you the reason, and what should you expect to read in it?

kubectl describe pod shows the scheduler's FailedScheduling event, which names the reason node by node: insufficient CPU or memory, an untolerated taint, or no node matching the selector.

kubectl get pods shows a Pod with STATUS Running and READY 1/2, and requests to its Service are failing. What does that pair of columns tell you?

READY counts ready containers out of total containers in the Pod. At 1/2 the Pod is not ready, so it is left out of Service endpoints even though STATUS says Running.

A Pod is in ImagePullBackOff. Which TWO causes are consistent with that status?

ImagePullBackOff means the kubelet cannot fetch the image. A wrong name or tag and a missing registry credential are the two usual reasons, and the Pod's events name which one.

A Pod will not start and you need both the scheduling events and the exact resource requests as submitted. Which two commands give you those, and what does each leave out?

describe gives a human summary plus recent Events, which the YAML never contains. get -o yaml gives the exact stored object including defaults, which describe abbreviates.

A container is in CrashLoopBackOff and kubectl logs returns nothing useful because the container has just restarted. Which command shows the output from the failed run?

kubectl logs --previous returns the logs of the previous instantiation of the container, which is where the reason for the crash actually is.

A Pod is scheduled to a node but its status reads CreateContainerConfigError. The image pulled fine. What class of problem is this, and where do you look?

The kubelet could not assemble the container's configuration, usually because a referenced ConfigMap, Secret or key does not exist. The Pod events name the missing object.

An engineer is unsure whether a for expression over a map produces the shape they expect, and wants to try it against real state without running a plan. Which tool does that?

terraform console opens an interactive shell for evaluating expressions against the current configuration and state. It is read-only, so it changes nothing while you experiment.

kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137, and a restart count climbing. What does that tell you, and what does not follow from it?

The container was killed for exceeding its memory limit, or the node ran out of memory. It is a container-level kill, so the Pod stays and the container restarts in place.

You need a shell alongside a running Pod to test connectivity, but the image is distroless and kubectl exec fails because there is no shell in it. What is the intended approach?

kubectl debug adds an ephemeral container to the running Pod, sharing its network namespace. That gives you a shell and tooling without rebuilding the image or restarting the Pod.

An application takes about 90 seconds to warm up. Its liveness probe uses the defaults with no initialDelaySeconds, and the Pod restarts continuously. What is the best fix?

The liveness probe is failing during startup and killing the container before it is ready. A startupProbe handles slow starts properly, holding the liveness probe off until startup succeeds.

A namespace holds forty Pods with status Evicted and no containers running. Replacement Pods are healthy on other nodes. What do these objects represent?

Evicted Pods are terminated Pod objects the kubelet left behind as a record of node pressure. They consume no resources, and deleting them is safe once you have read why they were evicted.

During a rolling update, some requests fail with connection errors even though every Pod eventually becomes healthy. Which mechanism prevents this, and how?

A readiness probe keeps a Pod out of a Service's endpoints until it can serve. Without one, a Pod receives traffic as soon as its container starts.

You need to check whether the API server is healthy and which of its internal checks is failing. Which approach reflects current practice?

The API server exposes livez and readyz, and healthz is deprecated. Adding verbose lists every individual check, which is how you find the one that is failing.

A worker node shows NotReady and its Pods are being replaced elsewhere. kubectl describe node reports the kubelet has stopped posting status. What do you check on the node itself?

The kubelet is a systemd service on the node. Check systemctl status kubelet and its logs with journalctl -u kubelet, which name the real failure.

After editing the kube-apiserver static Pod manifest, kubectl fails with a connection refused error. Which approach diagnoses this?

With the API server down, kubectl is useless. Use crictl on the control plane node to find the container and read its logs, and check the kubelet journal for manifest errors.

A Pod writes a large volume of logs. kubectl logs returns only recent output, and the earlier lines you need are missing. Why, and what does that imply?

The kubelet rotates container logs and kubectl logs only reads the latest file. Keeping history means shipping logs off the node to a cluster-level logging system.

You are asked to find out what happened in a namespace over the last few minutes. Which command gives the most useful ordered picture, and what limitation should you expect?

kubectl get events --sort-by=.lastTimestamp gives a namespace timeline. Events are namespaced and short-lived, retained for one hour by default, so older history is simply gone.

kubectl top nodes fails with an error saying the metrics API is not available, though every node is Ready and workloads are healthy. What does that indicate?

kubectl top reads the Metrics API, which is served by metrics-server rather than by the API server itself. Without it installed there is nothing to answer, and resource usage has to come from elsewhere.

Pods on one node show status Evicted and the node reports the DiskPressure condition. Which explanation is correct?

The kubelet evicts Pods when a node-level resource crosses an eviction threshold. DiskPressure comes from node or image filesystem thresholds, whose defaults are 10% and 15% available.

A Pod cannot resolve any Service name. Other Pods in the cluster resolve names normally. Which check comes first?

With other Pods resolving fine, CoreDNS is healthy, so look at this Pod. Its /etc/resolv.conf and its dnsPolicy decide which resolver it uses at all.

A Pod has been Terminating for fifteen minutes after a delete. What is the correct sequence of things to consider?

Check whether the process is ignoring SIGTERM within its grace period, whether the node is unreachable, and whether a finalizer is holding the object. Force deletion is a last resort.

On a node using containerd, you need to list containers and read one container's logs without going through the API server. Which tool is intended for this?

crictl is the CRI-compatible command line interface for inspecting containers on a node. It talks to the runtime directly, so it works when the control plane does not.

An application using the ServiceAccount ci in the build namespace gets a 403 listing Pods. Which command confirms the permission gap without deploying anything?

kubectl auth can-i with --as impersonates the identity and answers yes or no against the real authorization layer, so you can test a ServiceAccount's access from your own session.

A Pod that was Running is suddenly gone, and events show the scheduler removed it to make room for a higher-priority Pod. Which mechanism is this, and what protects against it?

This is preemption, driven by PriorityClass. The scheduler removes lower priority Pods so a pending higher priority Pod can be scheduled, and preemptionPolicy: Never opts a Pod out of preempting others.

On a fresh install, kubectl get nodes fails with a message about the connection to the server localhost:8080 being refused. What does that specific message mean?

localhost:8080 is kubectl's fallback when it finds no kubeconfig. The fix is to point it at a real one, usually by copying admin.conf into ~/.kube/config.

A Pod's status is Init:0/1 and it has stayed that way for several minutes. Where is the problem, and how do you read its output?

An init container has not completed, and the app containers cannot start until it does. Read its logs by naming it with kubectl logs -c.