Scheduling, probes, workloads and the failure modes that come with them.
44 items at intermediate level · all topics
Explain the architectural difference between AWS ECS Fargate and Amazon EKS (Elastic Kubernetes Service).
ECS and EKS are both orchestrators. Fargate isn't: it's a compute mode that either one can run on, so all four combinations exist. The real comparison is ECS vs EKS on who operates what, and Fargate vs EC2 on whether you want to own nodes at all.
A pod is stuck in Pending for 10 minutes in production. Walk me through how you diagnose it.
Pending means the scheduler hasn't placed the pod on a node. The cause is almost always resources, node selection constraints, or an unbound volume, and `kubectl describe pod` tells you which within seconds if you read the Events section rather than guessing.
You need to change a kube-apiserver flag on a control plane node where the API server runs as a static Pod. What actually makes the change take effect?
A static Pod is defined by a file on the node, not by an object in the API. The kubelet watches its staticPodPath, so saving the edited manifest is the whole procedure.
A Pod create request is rejected with a message from a LimitRanger. At which stage of the API server request path did that happen, and what had already succeeded?
Every request runs authentication, then authorization, then admission control, then validation and storage. An admission rejection means the caller was already known and already permitted.
A team wants to record a build URL and a git commit on every Deployment, and also to select those Deployments by team name. Which field takes which piece of data?
Labels are identifying data that selectors query, with a restricted value format. Annotations hold arbitrary non-identifying metadata that nothing selects on.
A manifest is already applied to the cluster. Someone edits one field in the file and reruns the command. Which behaviour separates kubectl apply from kubectl create here?
create is imperative and fails when the object exists. apply is declarative: it creates the object the first time and merges changes into it on every run afterwards.
You are asked to back up the cluster state of a cluster where etcd runs as a static Pod on the control plane node. Which approach produces a restorable backup?
Take an etcdctl snapshot save against the etcd endpoint, passing the CA certificate, client certificate and key that etcd is configured with. That one file is the cluster state.
A ClusterRole named pod-reader grants get, list and watch on pods. You create a RoleBinding in the dev namespace that binds this ClusterRole to user alice. What can alice do?
A RoleBinding that references a ClusterRole grants those permissions only inside the RoleBinding's own namespace. The same ClusterRole reused this way is how one definition serves many namespaces.
You must deploy one cluster component to three environments where only a few values differ, using tooling already available with kubectl and with no templating language to learn. Which approach fits best?
Kustomize is built into kubectl and patches plain YAML through overlays, so no templating is involved. Helm is the right tool when you want packaging, release history and rollbacks.
A worker node needs a kernel patch and reboot. You run kubectl drain node-3 and it fails with an error about DaemonSet-managed Pods. Which command completes the drain as documented?
kubectl drain refuses to proceed while DaemonSet Pods are present, so pass --ignore-daemonsets. It cannot evict them usefully anyway, since the DaemonSet controller recreates them at once.
A bad image was rolled out an hour ago and several deployments have happened since. kubectl rollout history shows revisions but the CHANGE-CAUSE column is empty. Which statements are true? Choose TWO.
Rollback works because old ReplicaSets are kept, ten by default. CHANGE-CAUSE comes from the kubernetes.io/change-cause annotation, which nothing sets for you any more.
A ConfigMap is consumed by a Deployment in two ways: one key as an environment variable, another mounted as a file. You edit the ConfigMap. What happens inside the running Pods?
A mounted ConfigMap key is eventually updated by the kubelet. An environment variable is fixed when the container starts and needs a restart to change.
A CronJob runs every minute, but the job sometimes takes three minutes. Runs are piling up on top of each other. Which field stops that, and what are the choices?
concurrencyPolicy decides what happens when a run is due while the previous one is still going. Allow is the default, Forbid skips the new run, and Replace kills the old one and starts fresh.
A Pod has one container with memory request 256Mi, memory limit 512Mi, and a CPU request but no CPU limit. Which QoS class does it get, and what does that mean under node memory pressure?
Requests that do not equal limits make the Pod Burstable. Under node pressure the kubelet evicts BestEffort Pods first, then Burstable, and Guaranteed last.
A Job manifest sets restartPolicy: Always in its Pod template and is rejected. What are the valid values, and why is that one refused here?
restartPolicy is Always, OnFailure or Never, and it applies to containers inside a Pod. A Job allows only OnFailure or Never, because Always would mean the Pod could never complete.
A HorizontalPodAutoscaler targeting 70% average CPU never scales, and kubectl describe hpa shows unknown for the current metric. Which TWO conditions would cause this?
CPU-based autoscaling needs a metrics source serving the Metrics API, normally metrics-server, and it needs CPU requests on the Pods' containers, since utilisation is a percentage of the request.
Six replicas keep landing unevenly across three zones, and losing one zone takes most of them out. Which mechanism enforces an even spread, and what does maxSkew mean?
topologySpreadConstraints spread Pods across a topology key such as zone. maxSkew is the largest allowed difference between domains, and whenUnsatisfiable decides whether that is a hard rule.
GPU nodes carry the taint gpu=true:NoSchedule and the label gpu=true. You add a matching toleration to a training Pod. Where can that Pod now be scheduled?
A toleration only removes an objection. It does not attract a Pod to the tainted nodes, so a nodeSelector or node affinity is still needed to keep the Pod on the GPU nodes.
An input variable for an environment name must only ever be dev, staging or prod, and a wrong value should fail immediately with a clear message. Which feature does this?
A validation block inside the variable declaration checks the value with a condition and reports error_message when it fails. It runs before planning, so a bad value never reaches a provider.
A StatefulSet's Pods must each be addressable individually so peers can form a cluster. Which Service configuration provides that, and what does DNS return?
A headless Service, clusterIP: None, allocates no virtual IP. DNS returns the Pod IPs directly, and each StatefulSet Pod also gets its own name under the Service.
Requests to a ClusterIP Service time out. kubectl get endpointslices shows no endpoints for it, though the Pods are Running. What are the TWO most likely causes?
A Service with no endpoints means nothing matched or nothing is ready. Check that the selector matches the Pod labels, and that the Pods pass their readiness probes.
An Ingress rule is written with path /api and no pathType field. The manifest is rejected. What does pathType control, and which value matches /api and /api/v1 alike?
pathType is required on every path. Prefix matches element by element, so /api also matches /api/v1. Exact matches the whole path only, and ImplementationSpecific leaves it to the controller.
After applying a default-deny egress NetworkPolicy, Pods in the namespace cannot reach anything by name, though they can still reach IP addresses you allowed. Why?
A default-deny egress policy also blocks the UDP and TCP port 53 traffic that Pods send to CoreDNS, so every name lookup fails. DNS has to be allowed back explicitly.
A Service must send cluster traffic to a database running on a VM outside the cluster, at a fixed IP. Which approach keeps the in-cluster name and works with a raw IP address?
A Service with no selector gets no endpoints automatically, so you create an EndpointSlice yourself pointing at the external address. ExternalName cannot be used with a bare IP.
A child module declares variable "instance_type" with no default. The caller's module block omits it. What happens, and how should the caller supply the value?
A variable with no default is required, so Terraform errors out naming the missing argument. The caller sets it as an argument inside the module block, alongside source.
You apply a NetworkPolicy that should block all ingress to a namespace, but every Pod stays reachable. kubectl get networkpolicy shows the object exists. What is the most likely explanation?
NetworkPolicies are enforced by the network plugin. If the cluster's CNI plugin does not implement them, the objects are accepted by the API server and have no effect.
Write the smallest NetworkPolicy that denies all ingress traffic to every Pod in a namespace. Which spec achieves it?
An empty podSelector selects every Pod in the namespace, and policyTypes: Ingress with no ingress rules allows nothing. That combination is the documented default-deny policy.
A platform team owns the cluster's shared entry point and application teams own their own routes. Which Gateway API objects match that split?
GatewayClass describes an implementation, a Gateway is the entry point the platform team runs, and HTTPRoute is the routing an application team attaches to it. The kinds are modelled on those roles.
A PVC created from the default StorageClass is deleted. The team expected the data to survive and it is gone. What explains this, and how is it prevented next time?
A dynamically provisioned PV inherits its StorageClass's reclaim policy, which is Delete unless set otherwise. Retain keeps the PV and the backing storage after the claim goes away.
A ConfigMap key is mounted with subPath so it lands beside files the container already has. The ConfigMap is edited, and unlike other mounts this file never changes. Why?
A container using a ConfigMap or Secret as a subPath mount does not receive updates. Only a whole-volume mount is refreshed by the kubelet, so a subPath mount needs a Pod restart.
A database PVC is nearly full and must grow from 20Gi to 100Gi with no data loss. What does Kubernetes require for that to work?
Edit the PVC's requested size. It only works if the claim's StorageClass has allowVolumeExpansion: true, and PVCs can be grown but never shrunk.
You delete a PVC and it sits in Terminating. A Pod is still using it. What is holding the deletion, and what happens when the Pod goes away?
Storage object in use protection adds a kubernetes.io/pvc-protection finalizer. The PVC stays in Terminating while a Pod uses it, and deletion completes once no Pod references it.
An admin creates a 50Gi PV with accessModes ReadWriteOnce and no storage class. A PVC asking for 10Gi ReadWriteOnce stays Pending. Which mismatch most likely explains it?
Binding matches on capacity, access mode and storage class. A claim that omits storageClassName asks for the default class, which a PV with no class cannot satisfy.
kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137, and a restart count climbing. What does that tell you, and what does not follow from it?
The container was killed for exceeding its memory limit, or the node ran out of memory. It is a container-level kill, so the Pod stays and the container restarts in place.
You need a shell alongside a running Pod to test connectivity, but the image is distroless and kubectl exec fails because there is no shell in it. What is the intended approach?
kubectl debug adds an ephemeral container to the running Pod, sharing its network namespace. That gives you a shell and tooling without rebuilding the image or restarting the Pod.
An application takes about 90 seconds to warm up. Its liveness probe uses the defaults with no initialDelaySeconds, and the Pod restarts continuously. What is the best fix?
The liveness probe is failing during startup and killing the container before it is ready. A startupProbe handles slow starts properly, holding the liveness probe off until startup succeeds.
A namespace holds forty Pods with status Evicted and no containers running. Replacement Pods are healthy on other nodes. What do these objects represent?
Evicted Pods are terminated Pod objects the kubelet left behind as a record of node pressure. They consume no resources, and deleting them is safe once you have read why they were evicted.
During a rolling update, some requests fail with connection errors even though every Pod eventually becomes healthy. Which mechanism prevents this, and how?
A readiness probe keeps a Pod out of a Service's endpoints until it can serve. Without one, a Pod receives traffic as soon as its container starts.
You need to check whether the API server is healthy and which of its internal checks is failing. Which approach reflects current practice?
The API server exposes livez and readyz, and healthz is deprecated. Adding verbose lists every individual check, which is how you find the one that is failing.
A worker node shows NotReady and its Pods are being replaced elsewhere. kubectl describe node reports the kubelet has stopped posting status. What do you check on the node itself?
The kubelet is a systemd service on the node. Check systemctl status kubelet and its logs with journalctl -u kubelet, which name the real failure.
A Pod cannot resolve any Service name. Other Pods in the cluster resolve names normally. Which check comes first?
With other Pods resolving fine, CoreDNS is healthy, so look at this Pod. Its /etc/resolv.conf and its dnsPolicy decide which resolver it uses at all.
On a node using containerd, you need to list containers and read one container's logs without going through the API server. Which tool is intended for this?
crictl is the CRI-compatible command line interface for inspecting containers on a node. It talks to the runtime directly, so it works when the control plane does not.
An application using the ServiceAccount ci in the build namespace gets a 403 listing Pods. Which command confirms the permission gap without deploying anything?
kubectl auth can-i with --as impersonates the identity and answers yes or no against the real authorization layer, so you can test a ServiceAccount's access from your own session.
A Pod's status is Init:0/1 and it has stayed that way for several minutes. Where is the problem, and how do you read its output?
An init container has not completed, and the app containers cannot start until it does. Read its logs by naming it with kubectl logs -c.