Observability

How do you get visibility into what's happening on an EC2 fleet, and who changed what, without SSHing into boxes to find out?

These are two different questions wearing one sentence. CloudWatch answers "what is the fleet doing right now," metrics, logs, alarms. CloudTrail answers "who did what to it," an audit log of every API call. Confusing the two is the most common way this question goes wrong.

What dashboards do you open first during an incident?

Good responders have a rehearsed order: service health to find the blast radius, golden signals to work out what kind of failure it is, then traces to localise it. Opening application logs first is usually the tell that someone doesn't know their own observability stack.

How do you know whether an issue is infrastructure or application?

The shape of the metrics answers this before you have to think hard. Errors up with latency and resources flat is almost always application. Latency up with a resource saturated is infrastructure. And whether one pod is affected or all of them is usually the single most decisive fact available.

What metrics do you check before SSHing into an instance?

Metrics give you the fleet; SSH gives you one box. Check CPU, memory, disk, network and load across instances first, that tells you which instance is actually the problem, and often tells you the answer outright. Go to the shell only once you've narrowed it down and need something metrics can't show you.

When would you scale vertically instead of horizontally?

Go vertical when the bottleneck doesn't split: database buffer cache, cache memory, network bandwidth on one node, a single-threaded lock. Go horizontal when the work partitions cleanly. The real question isn't which is better, it's whether the thing that's saturated can be divided across machines at all.

How do you investigate a latency spike that lasted only five minutes?

A spike that heals itself was self-limiting: a run of GC pauses, a cache stampede, a brief dependency blip, or a retry storm that burned out. It's over by the time you look, so the skill is knowing what to capture in the first thirty seconds and what to compare it against once the panic has passed.

How do you identify the exact deployment that introduced an issue?

Put the deployment log and the metrics timeline side by side. A regression that started fifteen minutes ago was almost certainly caused by something that shipped in that window. Confirm which commit is actually running from the image digest, then roll back and watch whether the metrics recover, that is your proof.

p99 latency on a critical API jumped from 120ms to 3s an hour ago. p50 is unchanged. How do you find the cause?

An unchanged p50 with a blown p99 means most requests are fine and a specific subset isn't, which rules out broad causes like CPU saturation and points at something correlated: one dependency, one shard, one node, one customer, or garbage collection. Unlike a spike that heals on its own in a few minutes, this one is still happening, so the shape of the distribution is the first clue and the investigation runs in parallel with mitigation.

How would you debug an issue that only happens under production traffic and cannot be reproduced in staging?

Production-only issues are almost always data volume, traffic volume, latency, or environment drift from staging. The job is to make production observable without making it worse: sample or mirror real traffic, use feature flags to isolate the suspect, turn on verbose logging briefly. Hammering staging harder is usually wasted time if staging was never shaped like production.

Your deployment system needs to know exactly which Git commit is running in production. How do you design that?

Stamp the commit SHA into the artifact at build time and expose it at runtime. Build once per commit, tag and deploy by digest, and serve a /version endpoint, so the answer to "what's live?" comes from the running process itself, not from a pipeline log someone has to go dig up.

Git is the source of truth, but production differs from Git. How do you prove where the drift happened?

Compare state at each stage, Git, rendered manifests, what was applied, and the live cluster, and use Kubernetes' own metadata to name the culprit. managedFields records which controller last wrote each field, which usually answers the question outright without any guessing.

Your LLM feature costs $40k/month and is growing 30% monthly. Leadership wants it cut by 70% without hurting quality. What do you do?

Most LLM spend goes on tokens that never needed generating, requests that never needed a frontier model, and identical prefixes reprocessed on every call. Measure where the money actually goes first, then attack caching, routing and prompt size before anything as drastic as self-hosting.