Observability

How do you get visibility into what's happening on an EC2 fleet, and who changed what, without SSHing into boxes to find out?

These are two different questions wearing one sentence. CloudWatch answers "what is the fleet doing right now," metrics, logs, alarms. CloudTrail answers "who did what to it," an audit log of every API call. Confusing the two is the most common way this question goes wrong.

What dashboards do you open first during an incident?

Good responders have a rehearsed order: service health to find the blast radius, golden signals to work out what kind of failure it is, then traces to localise it. Opening application logs first is usually the tell that someone doesn't know their own observability stack.

How do you know whether an issue is infrastructure or application?

The shape of the metrics answers this before you have to think hard. Errors up with latency and resources flat is almost always application. Latency up with a resource saturated is infrastructure. And whether one pod is affected or all of them is usually the single most decisive fact available.

What metrics do you check before SSHing into an instance?

Metrics give you the fleet; SSH gives you one box. Check CPU, memory, disk, network and load across instances first, that tells you which instance is actually the problem, and often tells you the answer outright. Go to the shell only once you've narrowed it down and need something metrics can't show you.

When would you scale vertically instead of horizontally?

Go vertical when the bottleneck doesn't split: database buffer cache, cache memory, network bandwidth on one node, a single-threaded lock. Go horizontal when the work partitions cleanly. The real question isn't which is better, it's whether the thing that's saturated can be divided across machines at all.

How do you investigate a latency spike that lasted only five minutes?

A spike that heals itself was self-limiting: a run of GC pauses, a cache stampede, a brief dependency blip, or a retry storm that burned out. It's over by the time you look, so the skill is knowing what to capture in the first thirty seconds and what to compare it against once the panic has passed.

How do you identify the exact deployment that introduced an issue?

Put the deployment log and the metrics timeline side by side. A regression that started fifteen minutes ago was almost certainly caused by something that shipped in that window. Confirm which commit is actually running from the image digest, then roll back and watch whether the metrics recover, that is your proof.

p99 latency on a critical API jumped from 120ms to 3s an hour ago. p50 is unchanged. How do you find the cause?

An unchanged p50 with a blown p99 means most requests are fine and a specific subset isn't, which rules out broad causes like CPU saturation and points at something correlated: one dependency, one shard, one node, one customer, or garbage collection. Unlike a spike that heals on its own in a few minutes, this one is still happening, so the shape of the distribution is the first clue and the investigation runs in parallel with mitigation.

How would you debug an issue that only happens under production traffic and cannot be reproduced in staging?

Production-only issues are almost always data volume, traffic volume, latency, or environment drift from staging. The job is to make production observable without making it worse: sample or mirror real traffic, use feature flags to isolate the suspect, turn on verbose logging briefly. Hammering staging harder is usually wasted time if staging was never shaped like production.

Your deployment system needs to know exactly which Git commit is running in production. How do you design that?

Stamp the commit SHA into the artifact at build time and expose it at runtime. Build once per commit, tag and deploy by digest, and serve a /version endpoint, so the answer to "what's live?" comes from the running process itself, not from a pipeline log someone has to go dig up.

Git is the source of truth, but production differs from Git. How do you prove where the drift happened?

Compare state at each stage, Git, rendered manifests, what was applied, and the live cluster, and use Kubernetes' own metadata to name the culprit. managedFields records which controller last wrote each field, which usually answers the question outright without any guessing.

Your LLM feature costs $40k/month and is growing 30% monthly. Leadership wants it cut by 70% without hurting quality. What do you do?

Most LLM spend goes on tokens that never needed generating, requests that never needed a frontier model, and identical prefixes reprocessed on every call. Measure where the money actually goes first, then attack caching, routing and prompt size before anything as drastic as self-hosting.

A web application runs on Amazon EC2 instances in an Auto Scaling group behind an Application Load Balancer. Users report being logged out at random. Investigation shows that session data is held in memory on each instance, so a request routed to a different instance loses the session. Which solution fixes this while keeping the web tier horizontally scalable?

Session state held on an instance makes that instance special, which is what breaks scaling and termination. Moving sessions to a shared store such as ElastiCache or DynamoDB makes every instance interchangeable, which is the property the rest of the architecture assumes.

A product catalogue application backed by Amazon RDS shows rising read latency. Analysis shows that a small number of identical queries account for most of the load, and the underlying data changes only a few times per day. The team wants to reduce latency to single-digit milliseconds with minimal application rework. What should a solutions architect recommend?

Repeated identical reads over rarely changing data is the textbook caching case. Amazon ElastiCache serves those queries from memory in well under a millisecond, and because the data changes a few times a day, staleness is cheap to manage.

A multiplayer game server runs on Amazon EC2 instances behind Network Load Balancers in three AWS Regions. Players connect over UDP. The company needs to route each player to the lowest-latency healthy Region, to fail over within seconds if a Region becomes unhealthy, and to publish a fixed set of IP addresses that players' firewalls can allow. Which solution meets these requirements?

UDP, static IP addresses and fast regional failover all point at AWS Global Accelerator rather than CloudFront. Global Accelerator gives you anycast static IPs, carries traffic over the AWS backbone, and reroutes without waiting for DNS to expire anywhere.

An application stores orders in an Amazon DynamoDB table with a partition key of orderId. A new reporting feature must list all orders for a given customerId, sorted by order date. Running the report currently scans the entire table and is slow and expensive. What should a solutions architect recommend?

A scan means the access pattern has no index behind it. A global secondary index with customerId as its partition key and the order date as its sort key turns that scan into a query. Only a global secondary index can introduce a new partition key on an existing table.

A HorizontalPodAutoscaler targeting 70% average CPU never scales, and kubectl describe hpa shows unknown for the current metric. Which TWO conditions would cause this?

CPU-based autoscaling needs a metrics source serving the Metrics API, normally metrics-server, and it needs CPU requests on the Pods' containers, since utilisation is a percentage of the request.

A container is in CrashLoopBackOff and kubectl logs returns nothing useful because the container has just restarted. Which command shows the output from the failed run?

kubectl logs --previous returns the logs of the previous instantiation of the container, which is where the reason for the crash actually is.

A Pod writes a large volume of logs. kubectl logs returns only recent output, and the earlier lines you need are missing. Why, and what does that imply?

The kubelet rotates container logs and kubectl logs only reads the latest file. Keeping history means shipping logs off the node to a cluster-level logging system.

You are asked to find out what happened in a namespace over the last few minutes. Which command gives the most useful ordered picture, and what limitation should you expect?

kubectl get events --sort-by=.lastTimestamp gives a namespace timeline. Events are namespaced and short-lived, retained for one hour by default, so older history is simply gone.

kubectl top nodes fails with an error saying the metrics API is not available, though every node is Ready and workloads are healthy. What does that indicate?

kubectl top reads the Metrics API, which is served by metrics-server rather than by the API server itself. Without it installed there is nothing to answer, and resource usage has to come from elsewhere.