Branching, history and collaboration workflow.
5 items at expert level · all topics
Deployment says SUCCESS but production is running an older commit. You have 10 minutes. What do you check?
Ask the running process what it actually is, then walk backwards through the chain until the SHA stops matching. A green pipeline only proves each step exited zero, not that anything actually changed in production, and "unchanged" is a success message that means nothing happened at all.
Git is the source of truth, but production differs from Git. How do you prove where the drift happened?
Compare state at each stage, Git, rendered manifests, what was applied, and the live cluster, and use Kubernetes' own metadata to name the culprit. managedFields records which controller last wrote each field, which usually answers the question outright without any guessing.
Someone force-pushed main at 2 AM. How do you investigate?
Preserve evidence first, then answer four questions: what was main before, what is it now, who did it, and was anything deployed from the rewritten history. Treat it as potentially malicious until the audit log says otherwise, because a force-push at 2 AM is an unusual enough event to earn that default.
An attacker steals a developer's Git credentials and pushes malicious code. How do you defend against this?
Assume one credential will eventually be stolen and design so that alone isn't enough to ship code. Layer identity, branch protection, review, signing, pipeline isolation and detection, and remember the attacker's real target is usually the CI workflow, not the application code itself.
Nobody knows which Git branch or tag corresponds to production. How would you fix the release process?
Establish one unbroken chain of identity from commit to running process, then make the pipeline the only way anything reaches production. Start by discovering what's actually deployed today, you can't design a release process around a system you can't describe.