Observability

What dashboards do you open first during an incident?

Good responders have a rehearsed order: service health to find the blast radius, golden signals to work out what kind of failure it is, then traces to localise it. Opening application logs first is usually the tell that someone doesn't know their own observability stack.

How do you know whether an issue is infrastructure or application?

The shape of the metrics answers this before you have to think hard. Errors up with latency and resources flat is almost always application. Latency up with a resource saturated is infrastructure. And whether one pod is affected or all of them is usually the single most decisive fact available.

What metrics do you check before SSHing into an instance?

Metrics give you the fleet; SSH gives you one box. Check CPU, memory, disk, network and load across instances first, that tells you which instance is actually the problem, and often tells you the answer outright. Go to the shell only once you've narrowed it down and need something metrics can't show you.

When would you scale vertically instead of horizontally?

Go vertical when the bottleneck doesn't split: database buffer cache, cache memory, network bandwidth on one node, a single-threaded lock. Go horizontal when the work partitions cleanly. The real question isn't which is better, it's whether the thing that's saturated can be divided across machines at all.

How do you investigate a latency spike that lasted only five minutes?

A spike that heals itself was self-limiting: a run of GC pauses, a cache stampede, a brief dependency blip, or a retry storm that burned out. It's over by the time you look, so the skill is knowing what to capture in the first thirty seconds and what to compare it against once the panic has passed.

How do you identify the exact deployment that introduced an issue?

Put the deployment log and the metrics timeline side by side. A regression that started fifteen minutes ago was almost certainly caused by something that shipped in that window. Confirm which commit is actually running from the image digest, then roll back and watch whether the metrics recover, that is your proof.

Your LLM feature costs $40k/month and is growing 30% monthly. Leadership wants it cut by 70% without hurting quality. What do you do?

Most LLM spend goes on tokens that never needed generating, requests that never needed a frontier model, and identical prefixes reprocessed on every call. Measure where the money actually goes first, then attack caching, routing and prompt size before anything as drastic as self-hosting.

A multiplayer game server runs on Amazon EC2 instances behind Network Load Balancers in three AWS Regions. Players connect over UDP. The company needs to route each player to the lowest-latency healthy Region, to fail over within seconds if a Region becomes unhealthy, and to publish a fixed set of IP addresses that players' firewalls can allow. Which solution meets these requirements?

UDP, static IP addresses and fast regional failover all point at AWS Global Accelerator rather than CloudFront. Global Accelerator gives you anycast static IPs, carries traffic over the AWS backbone, and reroutes without waiting for DNS to expire anywhere.

A Pod writes a large volume of logs. kubectl logs returns only recent output, and the earlier lines you need are missing. Why, and what does that imply?

The kubelet rotates container logs and kubectl logs only reads the latest file. Keeping history means shipping logs off the node to a cluster-level logging system.