Pipelines, deployment strategies and release safety.
10 items at advanced level · all topics
How do you identify the exact deployment that introduced an issue?
Put the deployment log and the metrics timeline side by side. A regression that started fifteen minutes ago was almost certainly caused by something that shipped in that window. Confirm which commit is actually running from the image digest, then roll back and watch whether the metrics recover, that is your proof.
A deploy of commit A is still running when commit B lands on main. What can go wrong?
If the pipeline resolves `main` at each step instead of pinning one commit early, later stages can pick up B while earlier ones tested A. You get mixed versions, out-of-order deploys, and a rollback target that no longer means anything. The fix is pinning the SHA once plus serializing production deploys.
Your 5 GB monorepo has code, binaries, Terraform and Helm charts, and clones are slow. What do you do?
Measure what the 5 GB actually is: big blobs, deep history, and a wide tree are three different problems with three different fixes. Work through them in order of cost, artifacts out of Git, binaries into LFS, partial clone and sparse checkout for developers, and treat splitting the repo as the last resort rather than the first idea.
Your organization wants signed commits for production code. How would you implement it?
git config user.email is free text, not identity, so signing is what turns authorship into a cryptographic claim. Roll out SSH signing, enforce it with branch protection and a trusted-key list, and plan for the parts that actually break rollouts: bots, squash merges, and key rotation.
Two CI pipelines try to create the same Git release tag at the same time. What happens?
The remote accepts one push and rejects the other, because ref updates are atomic. The danger isn't the collision, it's a pipeline that "fixes" the rejection with --force, which silently moves an existing release tag onto a different commit. Serialize releases and protect tags server-side.
A critical production bug needs an emergency fix, but your PR process takes two hours. What do you do?
Mitigate first: rollback or a feature flag beats writing code under pressure. If code is genuinely needed, use a documented hotfix lane: branch from the production tag, minimal fix, fast tests, one reviewer, deploy, then merge back. Emergency means a faster controlled process, not no process.
You use GitOps and someone changes Kubernetes manually. Git still has the old config. What happens?
You get configuration drift: Git says one thing, the cluster says another. What happens next depends on whether the controller self-heals or just reports it. Either way the manual change is temporary, and the fix is to put the intended change into Git, not to argue with the controller.
Someone moved a production Git tag to a different commit. Why is that dangerous?
A release tag is a promise that a version name means one exact, unchanging set of code. Moving it breaks that promise everywhere at once: rollbacks, audits, incident timelines, and any pipeline that deploys by tag now point somewhere different from what people believe, and some clones won't even notice the change.
Your repository has hundreds of branches, many already merged into main. What would you change?
Deleting merged branches is easy and worth automating, but it's the symptom. The real disease is branches living long enough to diverge, which shows up as painful merges and untested integration risk. Fix PR size and review speed first, and watch for squash-merged branches that `--merged` can't even see.
Your organization wants production deployments to be reproducible six months later. How does Git help?
Git pins the source exactly and that's all it pins. Real reproducibility also needs locked dependencies, versioned infrastructure, immutable artifacts, and a stored record linking a deployment to its digest, plus keeping the actual artifact, because rebuilding it later is rarely byte-identical.