Scheduling, probes, workloads and the failure modes that come with them.
21 items at advanced level · all topics
What happens if an Auto Scaling instance never becomes healthy?
Auto Scaling terminates and replaces an instance that never passes health checks, which is fine once and expensive in a loop. How long you burn depends on the grace period, the check thresholds, and which health check type the group is actually using. The fix is working out why it's unhealthy: usually startup config, a wrong health endpoint, or a grace period shorter than boot time.
You use GitOps and someone changes Kubernetes manually. Git still has the old config. What happens?
You get configuration drift: Git says one thing, the cluster says another. What happens next depends on whether the controller self-heals or just reports it. Either way the manual change is temporary, and the fix is to put the intended change into Git, not to argue with the controller.
Argo CD keeps reverting your emergency production change. Why, and what should you do?
Self-heal is doing exactly what it was configured to do: pulling the cluster back to what Git says. During an incident, mitigate through something the controller doesn't manage, then get the real change into Git fast rather than fighting reconciliation head on.
A cluster must be rolled back to an etcd snapshot taken this morning. The control plane is still running. What does the documented restore procedure require?
Stop every API server first, restore the snapshot into a fresh data directory, then start the API servers again. Restoring under a live API server is explicitly warned against.
A team is planning a highly available control plane and must choose between the stacked etcd topology and the external etcd topology. Which statement correctly describes the trade-off?
Stacked etcd runs an etcd member on each control plane node, so losing a node loses both an API server and an etcd member. External etcd separates the two at the cost of twice the hosts.
An application Pod needs to call the Kubernetes API using a ServiceAccount named deployer. How does the Pod get a token in a current cluster?
Setting serviceAccountName on the Pod is enough. The kubelet requests a short-lived, automatically rotated token through the TokenRequest API and mounts it as a projected volume.
A team wants to add a new resource type, backups.example.com, that users manage with kubectl. They have no appetite for running another API server. Which extension mechanism fits, and what do they still need?
A CustomResourceDefinition adds the type and gets storage, validation and kubectl support for free. It does nothing on its own, so a controller is still needed to act on the objects.
kubectl drain is taking a node out of service and appears to hang, evicting nothing further. A PodDisruptionBudget on the affected app sets minAvailable: 3 and exactly 3 Pods are ready. What is happening?
drain uses the Eviction API, which respects PodDisruptionBudgets. With minAvailable already at the limit, no further eviction is allowed until a replacement Pod becomes ready somewhere else.
A namespace has a ResourceQuota with requests.cpu and limits.memory set. A developer's Pod is rejected with a 403 saying it must specify resource limits. What is the cleanest fix for every future Pod in that namespace?
When a quota covers a compute resource, every Pod must specify that request or limit. A LimitRange in the namespace supplies defaults, so Pods that omit them are still admitted.
A log-shipping container must start before the application container and keep running for the life of the Pod. How is that expressed in the Pod spec?
A sidecar is an entry in initContainers with its own restartPolicy set to Always. It starts in init order, then keeps running while the application containers start.
An application needs the real client IP address. The Service is type LoadBalancer and currently reports the node IP as the source. Setting externalTrafficPolicy: Local fixes it. What is the cost?
Local preserves the client source IP by refusing to forward between nodes. A node with no local ready endpoint stops serving that Service, so traffic can be dropped and load can spread unevenly.
You are asked what actually makes a ClusterIP reachable from a Pod, given that the cluster IP is not assigned to any network interface. What is the accurate explanation?
A cluster IP is a virtual address. kube-proxy programs packet forwarding rules on every node, in iptables, IPVS or nftables mode, which rewrite traffic for that IP to a chosen endpoint.
A NetworkPolicy ingress rule has one from entry containing both a namespaceSelector (user=alice) and a podSelector (role=client). Which traffic is allowed?
Selectors inside one from entry are combined with AND. Splitting them into two entries in the from list makes it OR, and that single dash is the whole difference.
A shared Gateway lives in the infra namespace. A team applies an HTTPRoute in their own namespace referencing that Gateway, and it is not accepted. What is required?
A Gateway only accepts routes from its own namespace by default. Attaching a route from elsewhere requires the Gateway's listener to permit it through allowedRoutes.
In a cluster spread over three availability zones, Pods regularly fail to start because their volume was created in a zone the Pod cannot be scheduled into. Which StorageClass setting fixes this?
volumeBindingMode: WaitForFirstConsumer delays provisioning until a Pod using the claim is scheduled, so the volume is created where the Pod actually landed.
A StatefulSet with volumeClaimTemplates is deleted. Its PVCs are still present afterwards. Is that a bug, and how is the behaviour controlled?
Keeping the PVCs is the default and deliberate, so data survives a recreated StatefulSet. persistentVolumeClaimRetentionPolicy changes what happens on delete and on scale-down.
After editing the kube-apiserver static Pod manifest, kubectl fails with a connection refused error. Which approach diagnoses this?
With the API server down, kubectl is useless. Use crictl on the control plane node to find the container and read its logs, and check the kubelet journal for manifest errors.
A Pod writes a large volume of logs. kubectl logs returns only recent output, and the earlier lines you need are missing. Why, and what does that imply?
The kubelet rotates container logs and kubectl logs only reads the latest file. Keeping history means shipping logs off the node to a cluster-level logging system.
Pods on one node show status Evicted and the node reports the DiskPressure condition. Which explanation is correct?
The kubelet evicts Pods when a node-level resource crosses an eviction threshold. DiskPressure comes from node or image filesystem thresholds, whose defaults are 10% and 15% available.
A Pod has been Terminating for fifteen minutes after a delete. What is the correct sequence of things to consider?
Check whether the process is ignoring SIGTERM within its grace period, whether the node is unreachable, and whether a finalizer is holding the object. Force deletion is a last resort.
A Pod that was Running is suddenly gone, and events show the scheduler removed it to make room for a higher-priority Pod. Which mechanism is this, and what protects against it?
This is preemption, driven by PriorityClass. The scheduler removes lower priority Pods so a pending higher priority Pod can be scheduled, and preemptionPolicy: Never opts a Pod out of preempting others.