Capacity, availability and staying up under load.
28 items · all topics
How an autoscaler actually decides, and why it always lags
A Horizontal Pod Autoscaler runs one formula on a loop: desired = ceil(replicas × current ÷ target). Everything people find surprising about autoscaling (the lag, the overshoot, the slow scale-down) falls out of that formula and the timers around it.
Liveness vs readiness: one restarts, one just stops traffic
A readiness probe decides whether a pod gets traffic. A liveness probe decides whether it gets killed. Point them at the same slow dependency and readiness protects you while liveness takes the whole deployment down.
How do you architect a highly available, zero-downtime database migration in AWS RDS?
Zero downtime means you never ask for a maintenance window. Bring the new database up next to the old one, keep it in sync while the old one carries all the traffic, check the data constantly, then switch connections over in a few seconds, with a way back you set up before you need it.
Your traffic is spiky and unpredictable during business hours. Design an EC2 Auto Scaling policy that reacts fast without overspending overnight.
Target tracking should be the default scaling policy, since it only needs one metric and handles the math itself. Layer scheduled actions on top for the predictable part of the day, and treat step scaling as the exception for spikes that need a bigger, immediate jump than target tracking gives you.
When would you use EC2 Spot Instances in production, and how do you design a workload to survive interruptions?
Spot is unused EC2 capacity sold at a steep discount, usually 60 to 90 percent off On-Demand, with the trade that AWS can reclaim it with two minutes of warning. It's a good fit for anything stateless, retryable, or checkpointed, and a bad fit for anything that can't tolerate an interruption.
How do you prepare an EC2-based application to fail over to a second AWS region if the primary region has an outage?
Disaster recovery for EC2 is really three separate problems: getting a bootable image into the DR region ahead of time, having a way to launch capacity there fast, and having something outside both regions to redirect traffic. The RTO and RPO you're targeting decide how much of that you keep running warm versus building on demand.
Multi-AZ deployments, read replicas, and Aurora all get mentioned as RDS availability features. What's actually different between them, and when do you use each?
Multi-AZ is a synchronous standby that exists purely for failover, not for serving traffic. A read replica is asynchronous and exists to serve read traffic, with failover as a secondary, slower use. Aurora replaces both mechanisms with a shared, distributed storage layer, which is why an Aurora Replica can do double duty that a standard RDS replica can't.
Compare blue-green, canary, and rolling deployments. When would you choose each?
Rolling replaces instances gradually, blue-green switches all traffic between two full environments at once, and canary sends a small slice of real traffic to the new version while watching metrics. They differ mainly in rollback speed, cost, and how much confidence you gain before full exposure.
When would you scale vertically instead of horizontally?
Go vertical when the bottleneck doesn't split: database buffer cache, cache memory, network bandwidth on one node, a single-threaded lock. Go horizontal when the work partitions cleanly. The real question isn't which is better, it's whether the thing that's saturated can be divided across machines at all.
Two CI pipelines try to create the same Git release tag at the same time. What happens?
The remote accepts one push and rejects the other, because ref updates are atomic. The danger isn't the collision, it's a pipeline that "fixes" the rejection with --force, which silently moves an existing release tag onto a different commit. Serialize releases and protect tags server-side.
Someone moved a production Git tag to a different commit. Why is that dangerous?
A release tag is a promise that a version name means one exact, unchanging set of code. Moving it breaks that promise everywhere at once: rollbacks, audits, incident timelines, and any pipeline that deploys by tag now point somewhere different from what people believe, and some clones won't even notice the change.
Nobody knows which Git branch or tag corresponds to production. How would you fix the release process?
Establish one unbroken chain of identity from commit to running process, then make the pipeline the only way anything reaches production. Start by discovering what's actually deployed today, you can't design a release process around a system you can't describe.
A service depends on a database that goes down for 60 seconds. Which probe configuration behaves correctly?
Readiness controls whether a pod receives traffic; liveness controls whether it gets killed and restarted. Checking a downstream dependency in a liveness probe turns a brief dependency blip into a cluster-wide restart storm.
You have 50M embeddings, need p99 under 50ms, and 95% recall is acceptable. Which vector index do you choose?
Vector index choice is a three-way trade between recall, latency and memory. At tens of millions of vectors with a tight latency budget and tolerance for approximate results, HNSW with quantization is the standard answer: exact search cannot meet the latency, and unquantized HNSW cannot meet the memory budget affordably.
An ecommerce application sends order events to a processing service. During sales events the processing service is overwhelmed and drops orders. Orders for a given customer must be processed in the sequence they were submitted, and no order may be processed twice. Which solution meets these requirements?
Two requirements in the stem, ordering and no duplicates, point at exactly one service. An SQS FIFO queue guarantees first-in-first-out delivery within a message group and provides exactly-once processing through deduplication. A standard queue gives neither.
An Amazon RDS for MySQL database serves a reporting application. Read queries have grown until they slow down writes, and the business separately requires that the database survive the loss of a single Availability Zone with automatic failover. Which combination meets both requirements?
Multi-AZ and read replicas solve different problems and the exam tests whether you conflate them. A Multi-AZ standby is synchronous and cannot serve reads. A read replica is asynchronous and does serve reads. Needing both availability and read scaling means deploying both.
A company runs a business-critical application in one AWS Region. The disaster recovery plan requires a recovery time objective of 10 minutes and a recovery point objective of 1 minute in a second Region. The company wants to avoid paying for a full duplicate of the production fleet. Which disaster recovery strategy should a solutions architect choose?
RTO and RPO in minutes rule out backup and restore and pilot light, because both require standing infrastructure up before traffic can move. Warm standby keeps a scaled-down copy always running, which is what buys minutes instead of hours without paying for a second full fleet.
A web application runs on Amazon EC2 instances in an Auto Scaling group behind an Application Load Balancer. Users report being logged out at random. Investigation shows that session data is held in memory on each instance, so a request routed to a different instance loses the session. Which solution fixes this while keeping the web tier horizontally scalable?
Session state held on an instance makes that instance special, which is what breaks scaling and termination. Moving sessions to a shared store such as ElastiCache or DynamoDB makes every instance interchangeable, which is the property the rest of the architecture assumes.
You are asked to back up the cluster state of a cluster where etcd runs as a static Pod on the control plane node. Which approach produces a restorable backup?
Take an etcdctl snapshot save against the etcd endpoint, passing the CA certificate, client certificate and key that etcd is configured with. That one file is the cluster state.
A media company stores original video files in an Amazon S3 bucket in us-east-1. Regulators require that a copy of every new file exists in a second AWS Region within 15 minutes of upload, and that no file can be permanently deleted by mistake. Which TWO actions should a solutions architect take?
Cross-Region Replication copies new objects to a bucket in another Region automatically, and versioning is its prerequisite as well as the control that makes deletion recoverable. The pair answers both halves of the requirement with one dependency between them.
A cluster must be rolled back to an etcd snapshot taken this morning. The control plane is still running. What does the documented restore procedure require?
Stop every API server first, restore the snapshot into a fresh data directory, then start the API servers again. Restoring under a live API server is explicitly warned against.
A team is planning a highly available control plane and must choose between the stacked etcd topology and the external etcd topology. Which statement correctly describes the trade-off?
Stacked etcd runs an etcd member on each control plane node, so losing a node loses both an API server and an etcd member. External etcd separates the two at the cost of twice the hosts.
A nightly batch job renders video segments. The work is split into thousands of independent tasks, any task can be retried safely if it fails, and the job must finish before 06:00 but has no other timing constraint. The company wants to minimise compute cost. Which approach should a solutions architect recommend?
Independent, retryable tasks with a loose deadline is the definition of an interruption-tolerant workload, and that is what Spot Instances are for. Spreading the request across several instance types and Availability Zones is what keeps a Spot fleet from being reclaimed all at once.
A HorizontalPodAutoscaler targeting 70% average CPU never scales, and kubectl describe hpa shows unknown for the current metric. Which TWO conditions would cause this?
CPU-based autoscaling needs a metrics source serving the Metrics API, normally metrics-server, and it needs CPU requests on the Pods' containers, since utilisation is a percentage of the request.
kubectl drain is taking a node out of service and appears to hang, evicting nothing further. A PodDisruptionBudget on the affected app sets minAvailable: 3 and exactly 3 Pods are ready. What is happening?
drain uses the Eviction API, which respects PodDisruptionBudgets. With minAvailable already at the limit, no further eviction is allowed until a replacement Pod becomes ready somewhere else.