The Kubelet and Container Runtime Interface (CRI): How Nodes Execute Pods
Understanding node sync loops, gRPC CRI Services, and Pod Sandbox pause containers.
Part 4 in Series — Catch up on the previous article: etcd Internals: Raft Consensus, MVCC Key-Value Storage, and Watch Streams (Part 3) before diving into this post.
An engineer deploys a new application to a Kubernetes cluster.
Running kubectl get pods shows:
NAME READY STATUS RESTARTS AGE
payment-6f8b9c4d2-x9k42 0/1 ContainerCreating 0 4m
The Pod hangs in ContainerCreating for four minutes.
System metrics show the worker node’s CPU and RAM are completely idle. kubectl get nodes reports the host node as Ready.
Inspecting kubelet logs on the worker node (journalctl -u kubelet) exposes the root cause:
E0908 18:40:00.124501 kubelet.go:2210] "Failed to create pod sandbox" err="rpc error: code = Unavailable desc = connection error: desc = transport: Error while dialing socket: dial unix /run/containerd/containerd.sock: connect: connection refused"
The kubelet node agent was attempting to issue gRPC instructions to containerd over the Container Runtime Interface (CRI) socket, but the runtime socket was unresponsive.
How does the kubelet translate high-level Kubernetes Pod declarations into running Linux container processes on a host node?
To understand node execution, we must explore the Kubelet Sync Loop, the Container Runtime Interface (CRI), and Pod Sandbox Creation.
1. The Role of the Kubelet
The Kubelet is the primary agent running on every worker node host machine as a system daemon (systemd).
Unlike controllers running in the control plane, the Kubelet is responsible for host-level execution:
[ Control Plane API Server ] <--- HTTPS / REST (Port 6443) ---> [ Kubelet Daemon ]
|
gRPC CRI Socket
(/run/containerd/containerd.sock)
v
[ Container Runtime ]
(containerd / CRI-O)
Primary Duties of the Kubelet:
- Pod State Synchronization: Watches the API Server for
PodSpecassignments matching its node name. - CRI Invocation: Instructs the container runtime (e.g.,
containerdorCRI-O) to create or destroy container processes. - Health Probing: Executes
liveness,readiness, andstartupprobes against running containers. - Status Reporting: Reports node resource usage and container status back to the API Server.
2. The Kubelet Sync Loop Architecture
Inside the Kubelet binary, execution is driven by a continuous Pod Sync Loop (syncLoop):
[ Kubelet Sync Loop ]
|
+---------------------------+---------------------------+
| | |
v v v
[ Pod Config Channel ] [ PLEG Channel ] [ Probe Channel ]
(Watches API Server (Pod Lifecycle Event (Receives Health Check
for PodSpec changes) Generator: Container state) Probe results)
| | |
+---------------------------+---------------------------+
|
v
[ syncPod() Invocation ]
|
+-------------------+-------------------+
| Desired != Actual | Actual == Desired |
v v
[ Issue CRI gRPC ] [ No Action Needed ]
Key Event Inputs to the Sync Loop:
- Pod Config Channel: Emits events when new PodSpecs are assigned or deleted by the control plane scheduler.
- PLEG (Pod Lifecycle Event Generator): Periodically queries the container runtime to detect container state changes (e.g., container exited or crashed).
- Probe Manager Channel: Delivers results of execution health probes.
When an event arrives, the Kubelet compares the Desired PodSpec against the Actual Running Containers. If a discrepancy exists, it invokes syncPod() to make gRPC CRI calls to the container runtime.
3. The Container Runtime Interface (CRI)
Historically, early Kubernetes versions contained hardcoded Docker integration code directly inside the Kubelet binary.
To decouple Kubernetes from specific runtime implementations, Kubernetes introduced the Container Runtime Interface (CRI)—a standardized gRPC protocol.
Any container engine (like containerd or CRI-O) can serve as a Kubernetes runtime simply by implementing the CRI gRPC interface.
+-------------------------------------------------------------------+
| KUBELET DAEMON |
+-------------------------------------------------------------------+
|
gRPC API over /run/containerd/containerd.sock
|
+---------------------+---------------------+
| |
v v
+-----------------------+ +-----------------------+
| 1. RuntimeService | | 2. ImageService |
| - RunPodSandbox | | - PullImage |
| - CreateContainer | | - ListImages |
| - StartContainer | | - RemoveImage |
| - StopContainer | +-----------------------+
+-----------------------+
4. Step-by-Step Trace: Executing a Pod
When Kubelet receives a new PodSpec containing two containers (web-app and sidecar-logging), it executes a 5-step CRI sequence:
Kubelet Daemon CRI Runtime (containerd)
| |
|--- 1. ImageService.PullImage("nginx:latest") ----------->|
|<-- Image Pulled Successfully ----------------------------|
| |
|--- 2. RuntimeService.RunPodSandbox(PodSandboxConfig) --->|
| (Creates Pause Container & Shared NetNS/IPC) |
|<-- Returns PodSandboxID ("sb-9f8e7d") -------------------|
| |
|--- 3. RuntimeService.CreateContainer(sb-9f8e7d, web) --->|
|<-- Returns ContainerID ("c-1a2b3c") ---------------------|
| |
|--- 4. RuntimeService.StartContainer("c-1a2b3c") ------->|
|<-- Container Started ------------------------------------|
The Pause Container (RunPodSandbox)
Why does CRI create a Pod Sandbox before creating application containers?
In Kubernetes, all containers in a Pod must share the exact same Network Namespace (IP address, port space) and IPC namespace.
To achieve this:
RunPodSandboxcreates an infrastructure container called the Pause Container (registry.k8s.io/pause).- The Pause Container acquires the Pod’s IP address and keeps the shared Network Namespace open.
- When application containers (
web-appandsidecar) are subsequently created viaCreateContainer, they are configured to join the Pause Container’s existing Network Namespace (--net=container:pause_id).
Even if application containers crash and restart, the underlying network namespace remains open and stable because the Pause Container never exits!
Summary & Next Steps
The Kubelet translates high-level Kubernetes declarations into node-level runtime execution:
- The Kubelet runs as a host system daemon monitoring
PodSpecassignments. - The Sync Loop processes events from the API Server, PLEG, and Probe Manager to reconcile node state.
- The Container Runtime Interface (CRI) defines standard gRPC services (
RuntimeService,ImageService) separating Kubelet from runtime engines. - Pod Sandboxes launch an infrastructure Pause Container to hold open shared Network and IPC namespaces for all containers inside the Pod.
In the next article, we transition to Module 2 and explore Declarative Desired State vs Imperative Commands: The Kubernetes Operating Philosophy.
References & Further Reading
- Kubernetes SIG Architecture. Kubernetes API Conventions & Declarative Principles. CNCF GitHub.
- Hausenblas, M., & Schimanski, S. (2019). Programming Kubernetes (Chapter 3: Custom Resources and Controllers). O’Reilly Media.
- CNCF. Declarative Management Principles in Kubernetes. CNCF Docs.
Part 5: Declarative Desired State vs Imperative Commands: The Kubernetes Operating Philosophy
Continue to Part 5 →