Informers, Listers, and the HTTP/2 Watch API: Efficient State Synchronization

A software team builds a custom Kubernetes operator to monitor application pods across a 500-node cluster.

To implement the “Observe” step of their control loop, the developers write a simple polling loop:

// Naive Polling Loop (DO NOT DO THIS IN PRODUCTION!)
while (true) {
    List<Pod> pods = kubeApiClient.getPods(); // Issues HTTP GET /api/v1/pods
    evaluatePods(pods);
    Thread.sleep(500); // Polls every 500 milliseconds
}

When deployed to production against 10,000 active pods, system metrics deteriorate instantly:

• API Server CPU utilization hits 100%.
• etcd network bandwidth exhausts as 10,000 pod JSON manifests are serialized and transmitted across the network twice per second.
• kube-apiserver begins rejecting requests with HTTP 429 Too Many Requests.

Why did naive HTTP polling crash the Kubernetes control plane?

To query cluster state efficiently without overwhelming the API Server, Kubernetes controllers rely on SharedInformers, In-Memory Indexers (Listers), and the HTTP/2 Watch API.

---

1. The Client-Go Informer Architecture

To eliminate expensive HTTP polling, the Kubernetes Go client library (client-go) defines a multi-stage event processing pipeline:

+-------------------------------------------------------------------+
|                        KUBE-APISERVER / ETCD                      |
+-------------------------------------------------------------------+
                                  |
            HTTP/2 ListWatch Stream (gRPC / Chunked JSON)
                                  v
+-------------------------------------------------------------------+
| 1. Reflector                                                      |
|    Issues initial LIST, then establishes persistent WATCH stream  |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
| 2. DeltaFIFO Queue                                                |
|    Buffers state change deltas (Added, Updated, Deleted)          |
+-------------------------------------------------------------------+
                                  |
            +---------------------+---------------------+
            |                                           |
            v                                           v
+-----------------------+                   +-----------------------+
| 3. Indexer (Lister)   |                   | 4. Resource Event     |
|    In-Memory RAM Cache|                   |    Handlers           |
|    (O(1) Local Reads!)|                   |    (Add/Update/Delete) |
+-----------------------+                   +-----------------------+
                                                        |
                                                        v
                                            +-----------------------+
                                            | 5. WorkQueue          |
                                            |    Worker Goroutines  |
                                            +-----------------------+

---

2. Component Pipeline Breakdown

Component 1: The Reflector (ListWatch)

The Reflector manages state synchronization over the network:

1. Initial LIST: When initialized, it issues a single HTTP GET request to fetch all objects of a given resource type (e.g., LIST /api/v1/pods) at resource revision $R_0$.
2. Persistent WATCH: It opens a persistent HTTP/2 chunked stream starting from revision $R_0$.
3. Populates DeltaFIFO: Whenever kube-apiserver pushes a change event down the stream, the Reflector wraps the event in a Delta payload and pushes it into the DeltaFIFO Queue.

---

Component 2: The In-Memory Indexer (Lister)

Why don’t controllers query the API Server when evaluating spec vs status?

The Informer pipeline consumes events from the DeltaFIFO queue and writes the deserialized object state into an in-memory cache called the Indexer (or Lister).

• When a controller executes lister.Pods(namespace).Get(name), zero network requests are issued to the API Server.
• The query reads directly from local process RAM in $\mathcal{O}(1)$ microsecond time.

---

Component 3: Resource Event Handlers & WorkQueue

The Informer dispatches notifications to registered event callback hooks:

• AddFunc(obj interface{}): Invoked when a new resource is created.
• UpdateFunc(oldObj, newObj interface{}): Invoked when a resource spec or status changes.
• DeleteFunc(obj interface{}): Invoked when a resource is deleted.

Instead of executing complex business logic inside event callbacks directly (which would block the Informer pipeline), handlers extract the resource key (e.g., "production/payment-api") and push it onto a rate-limiting WorkQueue.

Worker goroutines pull keys off the WorkQueue and execute the 3-step reconciliation loop.

---

3. SharedInformers: RAM Deduplication

If a single node runs 10 separate controllers (e.g., DeploymentController, ReplicaSetController, PodGCController), would each controller maintain its own separate cache of 10,000 Pods, wasting gigabytes of node RAM?

Kubernetes uses SharedInformers.

A SharedInformerFactory creates a single, shared in-memory cache per resource type. All 10 controllers register their event handlers against the exact same shared Informer cache, reducing memory footprint and network connections to a single stream!

---

Direct API Polling vs SharedInformer Architecture

Feature / Metric	Direct HTTP API Polling	SharedInformer Architecture
Network Overhead	High ($N$ full object HTTP transfers per second)	Ultra-Low (Single initial LIST + minimal delta stream)
API Server Load	High (Heavy CPU serialization & etcd reads)	Minimal (Single HTTP/2 watch connection per factory)
Read Query Latency	High (10ms – 500ms network roundtrip)	Microsecond ($\mathcal{O}(1)$ Local RAM Lookup)
Event Loss Resilience	Vulnerable to missing transient states	Guaranteed recovery via Resource Version Sync
Memory Footprint	Low initial memory, high network churn	Dedicated local RAM cache shared by controllers

---

Summary & Next Steps

SharedInformers provide scalable, event-driven state synchronization for Kubernetes controllers:

• Direct HTTP Polling exhausts API Server CPU and network bandwidth.
• Reflectors use initial LIST calls followed by persistent HTTP/2 WATCH streams to receive state deltas.
• Indexers (Listers) cache deserialized resource objects in process RAM for $\mathcal{O}(1)$ local reads.
• SharedInformers deduplicate network streams and RAM caches across multiple controllers.
• WorkQueues buffer keys for worker goroutines executing the reconciliation loop.

In the next article, we open Module 3 with The Atomic Unit of Scheduling: Why Kubernetes Uses Pods Instead of Containers.

References & Further Reading

1. Cloud Native Computing Foundation. StatefulSet Controller & Ordered Pod Management. CNCF Docs.
2. CNCF etcd Project. Running etcd on Kubernetes Best Practices. etcd Docs.
3. Burns, B., et al. (2022). Kubernetes: Up and Running (Chapter 10: StatefulSets). O’Reilly Media.

---