The Reconciliation Loop Engine: Observe, Diff, and Act Mechanics

A sysadmin logs directly into a physical worker node and manually terminates a container:

docker kill c4a91b2c3d4e

The container process dies instantly.

Yet, when the sysadmin checks kubectl get pods 3 seconds later:

NAME                     READY   STATUS    RESTARTS   AGE
payment-api-7d4f9-x9k42   1/1     Running   1          3s

The container is back online.

The sysadmin did not execute a kubectl command. No CI/CD pipeline triggered a rebuild.

A background system daemon detected that the running container count dropped from 1 to 0, compared it to the desired state of 1, and created a replacement container automatically.

This self-healing behavior is driven by the core engine of Kubernetes: The Reconciliation Loop.

---

1. The 3-Step Reconciliation Loop Architecture

Every controller inside kube-controller-manager (as well as custom Operators) operates as an infinite control loop executing three sequential steps:

                  +-----------------------------------+
                  |   1. OBSERVE                      |
                  |   Query actual state from etcd    |
                  |   & local informers               |
                  +-----------------------------------+
                                    |
                                    v
                  +-----------------------------------+
                  |   2. DIFF                         |
                  |   Calculate state difference:     |
                  |   Delta = spec - status           |
                  +-----------------------------------+
                                    |
                                    v
                  +-----------------------------------+
                  |   3. ACT                          |
                  |   Issue REST API calls to         |
                  |   reconcile discrepancy           |
                  +-----------------------------------+
                                    |
                                    +--- (Repeat Loop Forever)

---

Step 1: OBSERVE

The controller reads the desired state (spec) and the actual state (status) for resources it manages.

• Example: ReplicaSetController observes a manifest specifying replicas: 3 and queries cluster informers to find 2 matching active Pods.

Step 2: DIFF

The controller calculates the exact delta between desired state and actual state:

$\Delta = \text{Desired State } (\text{spec}) - \text{Actual State } (\text{status})$

• If $\Delta = 0$, the system is in equilibrium. The controller takes no action.
• If $\Delta > 0$, the system has fewer resources than declared (e.g., $3 - 2 = +1$). The controller must create 1 new Pod.
• If $\Delta < 0$, the system has more resources than declared (e.g., $3 - 4 = -1$). The controller must delete 1 extra Pod.

Step 3: ACT

The controller issues HTTP REST requests to kube-apiserver to execute corrective state transitions (creating, updating, or deleting resources).

---

2. Level-Triggered vs Edge-Triggered Architecture

Why is Kubernetes self-healing resilient even when network connections drop or control plane components restart?

The secret lies in its Level-Triggered architectural design.

EDGE-TRIGGERED SYSTEM (Event-Driven)            LEVEL-TRIGGERED SYSTEM (State-Driven)
"Recreate container when Event #42 occurs!"     "Ensure active container count EQUALS 3!"

Event #42 Fails / Missed in Network Stream?     Network Stream Restarts / Event Missed?
  --> System STAYS BROKEN forever!                --> Inspects current state level (2),
                                                      sees 2 != 3, and FIXES IT instantly!

Edge-Triggered Systems (Interrupt-Driven)

• React to state transitions (edges), such as PodDeletedEvent.
• Vulnerability: If a network partition causes a controller to miss an event notification, the system never recovers from the missed state change.

Level-Triggered Systems (State-Driven)

• React to the current state level (spec vs status), regardless of how many events occurred in the past.
• Resilience: Even if a controller misses 50 intermediate network events while offline, upon restarting it reads the current state level, detects that actual != desired, and reconciles the system back to equilibrium immediately.

---

3. Controller Pseudocode Implementation

To understand how controllers work under the hood, consider the simplified logic of a Java/Go ReplicaSetController:

public class ReplicaSetController implements Runnable {
    private final KubernetesApiClient api;

    @Override
    public void run() {
        while (true) {
            try {
                // 1. OBSERVE
                List<ReplicaSet> desiredReplicaSets = api.getReplicaSets();

                for (ReplicaSet rs : desiredReplicaSets) {
                    int desiredCount = rs.getSpec().getReplicas();
                    List<Pod> actualPods = api.getPodsForSelector(rs.getSpec().getSelector());
                    int actualCount = actualPods.size();

                    // 2. DIFF
                    int delta = desiredCount - actualCount;

                    // 3. ACT
                    if (delta > 0) {
                        for (int i = 0; i < delta; i++) {
                            System.out.println("[Controller] Creating Pod replica...");
                            api.createPod(rs.getSpec().getPodTemplate());
                        }
                    } else if (delta < 0) {
                        for (int i = 0; i < Math.abs(delta); i++) {
                            System.out.println("[Controller] Terminating extra Pod...");
                            api.deletePod(actualPods.get(i).getId());
                        }
                    }
                }
                Thread.sleep(1000); // Poll / Watch stream interval
            } catch (Exception e) {
                System.err.println("[Controller] Error in reconciliation loop: " + e.getMessage());
            }
        }
    }
}

---

Summary & Next Steps

The reconciliation loop engine provides the self-healing foundation of Kubernetes:

• The Control Loop executes three steps continuously: Observe current state, calculate Diff, and Act to resolve state discrepancies.
• Level-Triggered Design ensures state convergence based on current state levels rather than transient edge events, guaranteeing self-healing after network failures.
• Controllers drive actual state ($\text{status}$) until it converges with user intent ($\text{spec}$).

In the next article, we examine Informers, Listers, and the HTTP/2 Watch API: Efficient State Synchronization.

References & Further Reading

1. Cloud Native Computing Foundation. Deployments & ReplicaSet Controller Implementation. CNCF Docs.
2. Kubernetes SIG Apps. MaxSurge and MaxUnavailable Rolling Update Strategies. CNCF Docs.
3. Burns, B., et al. (2022). Kubernetes: Up and Running (Chapter 9: Deployments). O’Reilly Media.

---