The Stack Observer
Cloud Native

Why Forensic Container Checkpointing on EKS Is a Production Security Breakthrough

•Stackxx•Cloud Native, Kubernetes

When a container in your Amazon EKS cluster starts behaving suspiciously, the evidence is volatile. Kubernetes routinely reschedules and replaces workloads, and the moment a pod is terminated or evicted, your runtime state disappears: in-memory credentials, active network connections, injected processes, and ephemeral filesystem changes are all gone. Security teams have historically faced a brutal tradeoff—terminate the container to contain the breach and destroy the evidence, or leave it running to preserve the evidence and extend the exposure window.

Amazon EKS 1.34 changes that calculus. With the Kubelet Checkpoint API now functional on EKS, operators can capture a running container’s full runtime state—memory pages, file descriptors, network sockets, and process metadata—typically in under ten seconds, without stopping the workload. The resulting checkpoint is packaged as an OCI image and pushed to Amazon ECR for offline forensic analysis. This is not a theoretical capability. It is a practical, unprivileged workflow that security and platform teams can deploy today.

What the Checkpoint API Actually Does

The Kubelet Checkpoint API, introduced in Kubernetes 1.25 and promoted to beta in 1.30, exposes a simple HTTP endpoint on every node:

POST https://<node>:10250/checkpoint/<namespace>/<pod>/<container>

When called, the kubelet delegates to the container runtime through the standard CRI CheckpointContainer RPC. On EKS 1.34, which ships with containerd 2.x, that RPC invokes CRIU (Checkpoint/Restore In Userspace) to freeze the container process and write its full state to a tar archive on the node. The container then resumes running. In AWS testing with typical microservice containers (200–400 MB resident memory) on m5.xlarge nodes, the CRIU capture completes in under 10 seconds and adds less than 5% CPU overhead during the capture window. The full end-to-end flow, including packaging the checkpoint as an OCI image and pushing it to Amazon ECR, typically finishes in under 30 seconds.

Larger containers (1–2 GB resident memory) may take 30–60 seconds to capture, with time scaling roughly linearly with memory size. The key point is that the workload never stops serving traffic, so there is no downtime penalty for capturing evidence.

The Architecture: DaemonSet with Node-Aware Routing

AWS recommends deploying the checkpoint agent as a DaemonSet with one unprivileged pod per node, rather than as a sidecar injected into every application. This design has three operational advantages:

  • No application changes required — applications run unmodified with no sidecar injection.
  • Node-wide coverage — a single agent can checkpoint any pod on its node.
  • Simpler operations — one DaemonSet is easier to manage than sidecars in every deployment.

The agent exposes two endpoints: POST /checkpoint triggers a capture and pushes to ECR, and GET /healthz provides health checks. When an Application Load Balancer routes a request to an agent on a different node than the target pod, the receiving agent transparently proxies the request to the correct node by querying the Kubernetes API for the target pod’s spec.nodeName and forwarding to the checkpoint agent running there. This means operators can use a single ALB endpoint without managing pod-to-node affinity manually.

Prerequisites and Gotchas

Before deploying, there are several hard requirements and sharp edges to understand:

  • EKS 1.34 or later is required. containerd 1.7.x, shipped with EKS 1.30–1.33, does not implement the CheckpointContainer RPC.
  • CRIU must be installed on worker nodes. Amazon EKS AL2023 AMIs do not include CRIU by default. AWS provides a CRIU installer DaemonSet that runs as a privileged init container once per node to install the package.
  • IAM Roles for Service Accounts (IRSA) provide the agent’s credentials to push checkpoint images to Amazon ECR. Standard IAM policies apply.

One important limitation: while checkpoints captured on a given host can be analyzed or restored on that same host, migrating checkpoints to different hosts requires care. TCP connections typically do not survive migration, and hardware-specific states may not transfer. The current AWS walkthrough focuses on forensic capture and analysis, not on live migration or restore.

What This Means for Production Security Operations

The significance of this capability is not just technical—it is operational. Prior approaches to live container forensics often required mounting the container runtime socket directly into a privileged container, granting full host access. The Kubelet Checkpoint API removes that requirement. The checkpoint agent is unprivileged; it uses standard service account tokens for kubelet authentication and standard CRI interfaces for the capture. This is a meaningful reduction in attack surface for a security tool.

For compliance frameworks like PCI DSS and SOC 2, which require forensic evidence retention, the ability to capture and store container runtime state as versioned OCI artifacts in ECR provides an auditable trail that survives pod termination. The SANS 2024 Incident Response Survey noted that organizations unable to preserve volatile evidence during container security events face investigation delays of hours to days. Reducing that to under 30 seconds changes the incident response timeline materially.

Adoption Path for Platform Teams

For teams running EKS at scale, the recommended adoption path is incremental:

Phase 1: Deploy the CRIU installer and checkpoint agent DaemonSets on a non-production cluster. Validate capture latency and CPU overhead against your workload profiles. Memory-heavy workloads will take longer to capture, so baseline those numbers before any production deployment.

Phase 2: Integrate checkpoint triggering into your SIEM or incident response playbook. The agent’s /checkpoint endpoint can be called by automated workflows when anomaly detection rules fire, turning forensic capture into a first-line response step rather than a manual post-incident scramble.

Phase 3: Establish retention policies for checkpoint OCI images in ECR. Checkpoints contain full memory dumps and may include sensitive data. Treat them as confidential artifacts, with lifecycle policies that expire them after investigation is complete.

The Ecosystem Context

EKS is not the only place the Checkpoint API is becoming available. The capability depends on containerd 2.x implementing the CRI CheckpointContainer RPC, which means any Kubernetes distribution running that runtime version can support it. The community is actively working on restore APIs, and containerd 2.1+ already includes experimental restore support. The pattern AWS is demonstrating—unprivileged capture agents, OCI packaging, and registry-based storage—is likely to become a cross-platform standard for container forensics.

Sources