Incident Response

What dashboards do you open first during an incident?

Good responders have a rehearsed order: service health to find the blast radius, golden signals to work out what kind of failure it is, then traces to localise it. Opening application logs first is usually the tell that someone doesn't know their own observability stack.

How do you know whether an issue is infrastructure or application?

The shape of the metrics answers this before you have to think hard. Errors up with latency and resources flat is almost always application. Latency up with a resource saturated is infrastructure. And whether one pod is affected or all of them is usually the single most decisive fact available.

What metrics do you check before SSHing into an instance?

Metrics give you the fleet; SSH gives you one box. Check CPU, memory, disk, network and load across instances first, that tells you which instance is actually the problem, and often tells you the answer outright. Go to the shell only once you've narrowed it down and need something metrics can't show you.

How do you investigate a latency spike that lasted only five minutes?

A spike that heals itself was self-limiting: a run of GC pauses, a cache stampede, a brief dependency blip, or a retry storm that burned out. It's over by the time you look, so the skill is knowing what to capture in the first thirty seconds and what to compare it against once the panic has passed.

How do you identify the exact deployment that introduced an issue?

Put the deployment log and the metrics timeline side by side. A regression that started fifteen minutes ago was almost certainly caused by something that shipped in that window. Confirm which commit is actually running from the image digest, then roll back and watch whether the metrics recover, that is your proof.

What happens if an Auto Scaling instance never becomes healthy?

Auto Scaling terminates and replaces an instance that never passes health checks, which is fine once and expensive in a loop. How long you burn depends on the grace period, the check thresholds, and which health check type the group is actually using. The fix is working out why it's unhealthy: usually startup config, a wrong health endpoint, or a grace period shorter than boot time.

A deploy of commit A is still running when commit B lands on main. What can go wrong?

If the pipeline resolves `main` at each step instead of pinning one commit early, later stages can pick up B while earlier ones tested A. You get mixed versions, out-of-order deploys, and a rollback target that no longer means anything. The fix is pinning the SHA once plus serializing production deploys.

A secret was committed six months ago and exists in hundreds of commits. What do you do?

Treat it as a security incident, not a Git cleanup task. Rotate first, then work out the blast radius, then decide honestly whether rewriting history is worth the cost. The rewrite is the most visible part of the response and the least important one.

Someone force-pushed a branch and deleted important commits. How do you recover them?

The commits almost certainly still exist, they just have nothing pointing at them anymore. Find the old SHA from any clone, CI workspace, PR page or provider event log, then create a branch on it. Act the same day, because garbage collection is the real deadline.

A critical production bug needs an emergency fix, but your PR process takes two hours. What do you do?

Mitigate first: rollback or a feature flag beats writing code under pressure. If code is genuinely needed, use a documented hotfix lane: branch from the production tag, minimal fix, fast tests, one reviewer, deploy, then merge back. Emergency means a faster controlled process, not no process.

You use GitOps and someone changes Kubernetes manually. Git still has the old config. What happens?

You get configuration drift: Git says one thing, the cluster says another. What happens next depends on whether the controller self-heals or just reports it. Either way the manual change is temporary, and the fix is to put the intended change into Git, not to argue with the controller.

Argo CD keeps reverting your emergency production change. Why, and what should you do?

Self-heal is doing exactly what it was configured to do: pulling the cluster back to what Git says. During an incident, mitigate through something the controller doesn't manage, then get the real change into Git fast rather than fighting reconciliation head on.

After editing the kube-apiserver static Pod manifest, kubectl fails with a connection refused error. Which approach diagnoses this?

With the API server down, kubectl is useless. Use crictl on the control plane node to find the container and read its logs, and check the kubelet journal for manifest errors.

A Pod writes a large volume of logs. kubectl logs returns only recent output, and the earlier lines you need are missing. Why, and what does that imply?

The kubelet rotates container logs and kubectl logs only reads the latest file. Keeping history means shipping logs off the node to a cluster-level logging system.

Pods on one node show status Evicted and the node reports the DiskPressure condition. Which explanation is correct?

The kubelet evicts Pods when a node-level resource crosses an eviction threshold. DiskPressure comes from node or image filesystem thresholds, whose defaults are 10% and 15% available.

A Pod has been Terminating for fifteen minutes after a delete. What is the correct sequence of things to consider?

Check whether the process is ignoring SIGTERM within its grace period, whether the node is unreachable, and whether a finalizer is holding the object. Force deletion is a last resort.

A Pod that was Running is suddenly gone, and events show the scheduler removed it to make room for a higher-priority Pod. Which mechanism is this, and what protects against it?

This is preemption, driven by PriorityClass. The scheduler removes lower priority Pods so a pending higher priority Pod can be scheduled, and preemptionPolicy: Never opts a Pod out of preempting others.