Debugging production, running incidents and writing them up.
7 items at expert level · all topics
Describe the last Sev-1 incident you handled.
This question isn't really about the outage, it's about how you behave during one. A good answer shows metric-driven diagnosis, mitigation running in parallel with investigation, real numbers, and a lesson that changed a system rather than blamed a person. Vague heroics score badly; a specific timeline scores well.
How do you safely rotate IAM credentials or database secrets without downtime?
The safe pattern is overlapping validity. Create the new secret while the old one still works, get code that accepts either one deployed everywhere, confirm the new one is actually being used, then revoke the old one. Rotating in a single step guarantees a window where something is still holding a credential that no longer works.
p99 latency on a critical API jumped from 120ms to 3s an hour ago. p50 is unchanged. How do you find the cause?
An unchanged p50 with a blown p99 means most requests are fine and a specific subset isn't, which rules out broad causes like CPU saturation and points at something correlated: one dependency, one shard, one node, one customer, or garbage collection. Unlike a spike that heals on its own in a few minutes, this one is still happening, so the shape of the distribution is the first clue and the investigation runs in parallel with mitigation.
How would you debug an issue that only happens under production traffic and cannot be reproduced in staging?
Production-only issues are almost always data volume, traffic volume, latency, or environment drift from staging. The job is to make production observable without making it worse: sample or mirror real traffic, use feature flags to isolate the suspect, turn on verbose logging briefly. Hammering staging harder is usually wasted time if staging was never shaped like production.
Deployment says SUCCESS but production is running an older commit. You have 10 minutes. What do you check?
Ask the running process what it actually is, then walk backwards through the chain until the SHA stops matching. A green pipeline only proves each step exited zero, not that anything actually changed in production, and "unchanged" is a success message that means nothing happened at all.
Git is the source of truth, but production differs from Git. How do you prove where the drift happened?
Compare state at each stage, Git, rendered manifests, what was applied, and the live cluster, and use Kubernetes' own metadata to name the culprit. managedFields records which controller last wrote each field, which usually answers the question outright without any guessing.
Someone force-pushed main at 2 AM. How do you investigate?
Preserve evidence first, then answer four questions: what was main before, what is it now, who did it, and was anything deployed from the rewritten history. Treat it as potentially malicious until the audit log says otherwise, because a force-push at 2 AM is an unusual enough event to earn that default.