Debugging production, running incidents and writing them up.
33 items · all topics
Your application suddenly can't connect to its RDS database. Walk through how you'd diagnose it.
Rule out the database's own health first, since that's a two-second check in the console, then work through the network path in order: security groups, subnet and public accessibility, DNS, and connection limits. A sudden failure that used to work points hardest at something that changed recently, not at a fundamentally broken setup.
A pod is stuck in Pending for 10 minutes in production. Walk me through how you diagnose it.
Pending means the scheduler hasn't placed the pod on a node. The cause is almost always resources, node selection constraints, or an unbound volume, and `kubectl describe pod` tells you which within seconds if you read the Events section rather than guessing.
What dashboards do you open first during an incident?
Good responders have a rehearsed order: service health to find the blast radius, golden signals to work out what kind of failure it is, then traces to localise it. Opening application logs first is usually the tell that someone doesn't know their own observability stack.
How do you know whether an issue is infrastructure or application?
The shape of the metrics answers this before you have to think hard. Errors up with latency and resources flat is almost always application. Latency up with a resource saturated is infrastructure. And whether one pod is affected or all of them is usually the single most decisive fact available.
What metrics do you check before SSHing into an instance?
Metrics give you the fleet; SSH gives you one box. Check CPU, memory, disk, network and load across instances first, that tells you which instance is actually the problem, and often tells you the answer outright. Go to the shell only once you've narrowed it down and need something metrics can't show you.
Describe the last Sev-1 incident you handled.
This question isn't really about the outage, it's about how you behave during one. A good answer shows metric-driven diagnosis, mitigation running in parallel with investigation, real numbers, and a lesson that changed a system rather than blamed a person. Vague heroics score badly; a specific timeline scores well.
How do you investigate a latency spike that lasted only five minutes?
A spike that heals itself was self-limiting: a run of GC pauses, a cache stampede, a brief dependency blip, or a retry storm that burned out. It's over by the time you look, so the skill is knowing what to capture in the first thirty seconds and what to compare it against once the panic has passed.
How do you identify the exact deployment that introduced an issue?
Put the deployment log and the metrics timeline side by side. A regression that started fifteen minutes ago was almost certainly caused by something that shipped in that window. Confirm which commit is actually running from the image digest, then roll back and watch whether the metrics recover, that is your proof.
How do you safely rotate IAM credentials or database secrets without downtime?
The safe pattern is overlapping validity. Create the new secret while the old one still works, get code that accepts either one deployed everywhere, confirm the new one is actually being used, then revoke the old one. Rotating in a single step guarantees a window where something is still holding a credential that no longer works.
What happens if an Auto Scaling instance never becomes healthy?
Auto Scaling terminates and replaces an instance that never passes health checks, which is fine once and expensive in a loop. How long you burn depends on the grace period, the check thresholds, and which health check type the group is actually using. The fix is working out why it's unhealthy: usually startup config, a wrong health endpoint, or a grace period shorter than boot time.
p99 latency on a critical API jumped from 120ms to 3s an hour ago. p50 is unchanged. How do you find the cause?
An unchanged p50 with a blown p99 means most requests are fine and a specific subset isn't, which rules out broad causes like CPU saturation and points at something correlated: one dependency, one shard, one node, one customer, or garbage collection. Unlike a spike that heals on its own in a few minutes, this one is still happening, so the shape of the distribution is the first clue and the investigation runs in parallel with mitigation.
How would you debug an issue that only happens under production traffic and cannot be reproduced in staging?
Production-only issues are almost always data volume, traffic volume, latency, or environment drift from staging. The job is to make production observable without making it worse: sample or mirror real traffic, use feature flags to isolate the suspect, turn on verbose logging briefly. Hammering staging harder is usually wasted time if staging was never shaped like production.
You accidentally committed a .env file containing API keys. What do you do?
Rotate the credentials first. That's the only step that actually makes you safe, since the secret is already in every clone, fork and CI cache that pulled before you noticed. Cleaning up Git history is a second, separate job that comes after.
A developer says they pulled the latest code but their branch doesn't match the remote. How do you investigate?
It's almost always a tracking problem, not corruption. They pulled a different branch than they think, or their branch tracks something other than what they assume. `git branch -vv` and `git log HEAD..origin/main` usually answer it in two commands.
Code works on the developer's machine but fails in Jenkins. How could Git be involved?
Stop guessing about the code and compare commit SHAs first. CI often builds a different commit than the developer tested: a merge commit, a stale workspace, a shallow clone, missing submodules, or a file that's gitignored locally but needed at build time. Only once the SHAs match is it worth looking at the code.
A Git repository has become huge and cloning takes 20 minutes. What would you investigate?
Measure before you guess. Find the biggest objects in history, because size usually comes from binaries and build artifacts committed long ago. Deleting them today does nothing: the old blobs stay in history until you rewrite it or route developers around downloading them in the first place.
What is the difference between git fetch and git pull?
Fetch downloads remote commits and updates your remote-tracking branches, but never touches your working tree. Pull is fetch plus merge or rebase, so it changes your branch. Fetch is the safer move when you're debugging or scripting.
The deploy says it shipped main, but production doesn't have the latest commit. How do you debug it?
Walk the chain, commit, CI checkout, build, artifact, deploy, running pod, and compare the SHA at each step. main is a moving pointer, so the usual cause is that something in the chain resolved it at a different moment, or shipped a cached artifact instead of a fresh one.
Your CI uses git clone --depth=1 and a deploy script that needs history suddenly fails. Why?
A shallow clone downloads the current tree and exactly one commit, no parents, usually no tags. Anything that reads history breaks: `git describe`, changelogs, `git diff HEAD~10`, commit counts, `merge-base`. Fetch the depth you actually need instead of defaulting to depth 1 everywhere.
A developer wants to git reset --hard and force-push a shared branch to undo a bad commit. Do you allow it?
On a shared branch, no. Use git revert, which undoes the change with a new commit and leaves history intact. Reset plus force-push rewrites history that other people, CI and deployment records already depend on.
A bad feature was merged into production. How would you undo it?
Roll back the running deployment first, that's faster than any Git fix, then fix Git properly. Reverting a merge needs git revert -m 1 <merge-commit>, and the catch nobody mentions upfront is that you have to revert the revert later or the feature will never merge back in.
A production bug was introduced somewhere in the last 50 commits. How do you find the exact commit?
`git bisect` does a binary search over the range, so 50 commits take about 6 tests instead of 50. The hard part isn't the commands, it's having a reliable test that says good or bad, which is what lets you automate the whole thing with `git bisect run`.
Your CI pipeline runs twice for every pull request. How do you investigate?
Almost always two triggers firing on one action, usually push and pull_request both matching the same branch. Read the event that started each run, then make the trigger config deliberate instead of deleting jobs until the noise stops.
A developer says their commit has disappeared. How do you investigate and get it back?
Commits are rarely deleted, they usually just lose their branch reference. git reflog records every move of HEAD locally, so a bad reset, rebase or checkout is almost always recoverable, and git fsck --lost-found catches most of the rest.
A deploy of commit A is still running when commit B lands on main. What can go wrong?
If the pipeline resolves `main` at each step instead of pinning one commit early, later stages can pick up B while earlier ones tested A. You get mixed versions, out-of-order deploys, and a rollback target that no longer means anything. The fix is pinning the SHA once plus serializing production deploys.
A secret was committed six months ago and exists in hundreds of commits. What do you do?
Treat it as a security incident, not a Git cleanup task. Rotate first, then work out the blast radius, then decide honestly whether rewriting history is worth the cost. The rewrite is the most visible part of the response and the least important one.
Someone force-pushed a branch and deleted important commits. How do you recover them?
The commits almost certainly still exist, they just have nothing pointing at them anymore. Find the old SHA from any clone, CI workspace, PR page or provider event log, then create a branch on it. Act the same day, because garbage collection is the real deadline.
A critical production bug needs an emergency fix, but your PR process takes two hours. What do you do?
Mitigate first: rollback or a feature flag beats writing code under pressure. If code is genuinely needed, use a documented hotfix lane: branch from the production tag, minimal fix, fast tests, one reviewer, deploy, then merge back. Emergency means a faster controlled process, not no process.
You use GitOps and someone changes Kubernetes manually. Git still has the old config. What happens?
You get configuration drift: Git says one thing, the cluster says another. What happens next depends on whether the controller self-heals or just reports it. Either way the manual change is temporary, and the fix is to put the intended change into Git, not to argue with the controller.
Argo CD keeps reverting your emergency production change. Why, and what should you do?
Self-heal is doing exactly what it was configured to do: pulling the cluster back to what Git says. During an incident, mitigate through something the controller doesn't manage, then get the real change into Git fast rather than fighting reconciliation head on.
Deployment says SUCCESS but production is running an older commit. You have 10 minutes. What do you check?
Ask the running process what it actually is, then walk backwards through the chain until the SHA stops matching. A green pipeline only proves each step exited zero, not that anything actually changed in production, and "unchanged" is a success message that means nothing happened at all.
Git is the source of truth, but production differs from Git. How do you prove where the drift happened?
Compare state at each stage, Git, rendered manifests, what was applied, and the live cluster, and use Kubernetes' own metadata to name the culprit. managedFields records which controller last wrote each field, which usually answers the question outright without any guessing.
Someone force-pushed main at 2 AM. How do you investigate?
Preserve evidence first, then answer four questions: what was main before, what is it now, who did it, and was anything deployed from the rewritten history. Treat it as potentially malicious until the audit log says otherwise, because a force-push at 2 AM is an unusual enough event to earn that default.