Debugging production, running incidents and writing them up.
9 items at intermediate level · all topics
Your application suddenly can't connect to its RDS database. Walk through how you'd diagnose it.
Rule out the database's own health first, since that's a two-second check in the console, then work through the network path in order: security groups, subnet and public accessibility, DNS, and connection limits. A sudden failure that used to work points hardest at something that changed recently, not at a fundamentally broken setup.
A pod is stuck in Pending for 10 minutes in production. Walk me through how you diagnose it.
Pending means the scheduler hasn't placed the pod on a node. The cause is almost always resources, node selection constraints, or an unbound volume, and `kubectl describe pod` tells you which within seconds if you read the Events section rather than guessing.
The deploy says it shipped main, but production doesn't have the latest commit. How do you debug it?
Walk the chain, commit, CI checkout, build, artifact, deploy, running pod, and compare the SHA at each step. main is a moving pointer, so the usual cause is that something in the chain resolved it at a different moment, or shipped a cached artifact instead of a fresh one.
Your CI uses git clone --depth=1 and a deploy script that needs history suddenly fails. Why?
A shallow clone downloads the current tree and exactly one commit, no parents, usually no tags. Anything that reads history breaks: `git describe`, changelogs, `git diff HEAD~10`, commit counts, `merge-base`. Fetch the depth you actually need instead of defaulting to depth 1 everywhere.
A developer wants to git reset --hard and force-push a shared branch to undo a bad commit. Do you allow it?
On a shared branch, no. Use git revert, which undoes the change with a new commit and leaves history intact. Reset plus force-push rewrites history that other people, CI and deployment records already depend on.
A bad feature was merged into production. How would you undo it?
Roll back the running deployment first, that's faster than any Git fix, then fix Git properly. Reverting a merge needs git revert -m 1 <merge-commit>, and the catch nobody mentions upfront is that you have to revert the revert later or the feature will never merge back in.
A production bug was introduced somewhere in the last 50 commits. How do you find the exact commit?
`git bisect` does a binary search over the range, so 50 commits take about 6 tests instead of 50. The hard part isn't the commands, it's having a reliable test that says good or bad, which is what lets you automate the whole thing with `git bisect run`.
Your CI pipeline runs twice for every pull request. How do you investigate?
Almost always two triggers firing on one action, usually push and pull_request both matching the same branch. Read the event that started each run, then make the trigger config deliberate instead of deleting jobs until the noise stops.
A developer says their commit has disappeared. How do you investigate and get it back?
Commits are rarely deleted, they usually just lose their branch reference. git reflog records every move of HEAD locally, so a bad reset, rebase or checkout is almost always recoverable, and git fsck --lost-found catches most of the rest.