# Kubernetes: etcd-diagnosis turns vague control-plane pain into an actual incident workflow - Link: https://thestackobserver.com/kubernetes-etcd-diagnosis-turns-vague-control-plane-pain-into-an-actual-incident-workflow/ - Published: 2026-03-13T10:00:55-04:00 - Author: Stackxx The most useful etcd news is rarely glamorous. This week’s CNCF write-up on `etcd- diagnosis` and `etcd-recovery` matters because it tackles the miserable middle of a production incident: the part where the API server is timing out, teams are staring at ambiguous messages, and nobody is yet sure whether they have a storage problem, a network problem, a resource-pressure problem, or a genuine quorum disaster. That middle is where real time gets burned. Not on typing commands, but on deciding which commands are worth typing and what evidence to collect before someone panics and reaches for recovery. The etcd-diagnosis work is interesting for exactly that reason. It tries to turn tribal etcd lore into a report you can hand to another operator or upstream maintainer without three more rounds of “can you also gather…” ## What actually changed The CNCF post does not announce a shiny new Kubernetes feature. It outlines an operating model. The core idea is that `etcd-diagnosis report` should gather the signals teams repeatedly need during serious etcd incidents: * cluster health and membership state * disk I/O latency, especially WAL fsync behavior * network round-trip time between members * resource pressure signals such as memory and disk usage * high-value etcd metrics that otherwise require manual scraping That sounds obvious. It is not. A lot of Kubernetes shops still treat etcd debugging as “check a few logs, maybe run `etcdctl endpoint health`, then improvise.” Improvisation is exactly what makes bad control-plane days worse. ## Why operators should care etcd failures are notorious because the symptoms are generic while the blast radius is immediate. “Apply request took too long” does not tell you whether your disk is stalling. “mvcc: database space exceeded” tells you you are in pain, not why you got there. The practical win here is shortening the distance between symptom and useful classification. My read is simple: the tooling matters less as software than as discipline. Teams that already have an etcd incident runbook will get faster. Teams that do not will at least have a better template for one. Either way, the operational lesson is the same: you should separate _quick safety checks_ from _deep evidence capture_, and you should treat full-cluster recovery as a controlled exception, not a reflex. ## Goal Standardize an etcd incident flow that moves from fast health validation to evidence- rich diagnosis, and only escalates to recovery when the cluster is truly unrecoverable through normal reconciliation. ## Prereqs * Access to control-plane nodes or the etcd containers that run on them * Working `etcdctl` credentials and endpoint knowledge for the cluster * A place to store incident artifacts, not just terminal scrollback * Clarity on who owns the final call for recovery actions ## Steps **1) Start with quorum and member sanity checks.** Before you collect anything fancy, answer the most boring questions first: are members reachable, is quorum intact, and are Raft indexes still progressing? ```wp-block-code etcdctl endpoint health --cluster etcdctl endpoint status --cluster etcdctl member list ``` **2) If the quick checks look weird or incomplete, generate a full diagnostic report.** This is the real operational shift. Do not wait until the call is already chaotic. ```wp-block-code etcd-diagnosis report --endpoints=https://10.0.0.10:2379,https://10.0.0.11:2379,https://10.0.0.12:2379 --output ./incident-artifacts/etcd-report-$(date +%F-%H%M) ``` **3) Classify the incident before choosing the fix.** The CNCF write-up keeps returning to three families of trouble: * **Disk:** slow WAL fsync, saturated volumes, or noisy storage neighbors * **Network:** latency between members, packet loss, or unhealthy control-plane placement * **Resource pressure:** CPU starvation, memory pressure, or disk exhaustion The practical point is that “etcd is slow” is not a diagnosis. It is a category error until you know which subsystem is dragging it down. **4) Treat database space alerts as data-shape problems first.** The post’s best reminder is that `mvcc: database space exceeded` should lead you to ask what is consuming the space, not just whether to compact and defrag. Compaction might be necessary. It is not the whole story if a workload or controller is generating pathological key growth. **5) Escalate with artifacts, not vibes.** If you need vendor or upstream help, hand over the quick-check output, the diagnostic bundle, and a short timeline of symptoms. This is how you compress the back-and-forth. **6) Reserve recovery for actual quorum-loss or unrecoverable states.** The linked` etcd-recovery` tooling exists for serious cases, but the post is explicit that recovery is a last resort. If a single member failed while quorum is intact, the more correct action is often to let your cluster-management layer replace the node after you fix the underlying issue. ## Common pitfalls * **Skipping the quick checks.** Teams sometimes jump straight into “advanced debugging” before confirming whether the cluster is fundamentally healthy. * **Collecting evidence too late.** By the time the incident is fully on fire, useful transient signals may already be gone. * **Assuming every etcd symptom is an etcd bug.** Disk and network pathologies are often the real culprit. * **Using recovery as emotional relief.** Recovery can feel decisive while still being the wrong choice. * **Leaving no paper trail.** If the only record is someone’s shell history, the next incident starts from zero again. ## Verify * Run the three `etcdctl` quick checks on a healthy cluster and save example output in your runbook. * Test whether your operators can execute `etcd-diagnosis report` with the credentials and host access they actually have. * Document where diagnostic bundles are stored during a real incident. * Write down who is authorized to approve recovery so that decision is not made by whoever is most stressed. The best thing about this week’s etcd tooling push is not that it promises perfect recovery. It promises calmer judgment earlier in the incident. For Kubernetes operators, that is the part worth stealing. --- ## Sources * [CNCF Blog: Making etcd incidents easier to debug in production Kubernetes](https://www.cncf.io/blog/2026/03/12/making-etcd-incidents-easier-to-debug-in-production-kubernetes/) * [GitHub: vmware/etcd-diagnosis](https://github.com/vmware/etcd-diagnosis) * [GitHub: vmware/etcd-recovery](https://github.com/vmware/etcd-recovery) * [Kubernetes docs: Operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)