The Fallacies of Distributed Computing: PACELC, CAP Theorem, and Network Partitions
Deconstructing network unreliability, latency trade-offs, and consistency guarantees in distributed architectures
Part 1 in Series — Catch up on the previous article: Mastering System Design & Distributed Systems: Series Introduction & Learning Roadmap (Part 0) before diving into this post.
Why You Need This in Real Life
A major payment microservice team once refactored their monolithic architecture into 20 distributed microservices across AWS regions. During their architectural review, the team lead stated: “Network calls inside the VPC take under 1 millisecond. We can treat RPC method calls between services as if they were local in-memory method calls.”
Three weeks after launch, a fiber optic cable cut between availability zones triggered a network partition. Instead of failing gracefully, microservice worker threads hung indefinitely waiting for TCP socket responses. Within 4 minutes, connection pools across all 20 microservices locked up in socketRead0(), freezing the entire platform.
The team made the most dangerous mistake in system architecture: assuming the network is reliable, instantaneous, and infinite.
To build distributed systems that survive real-world cloud environments, you must master Deutsch’s 8 Fallacies of Distributed Computing, Eric Brewer’s CAP Theorem, and Abadi’s PACELC Theorem.
Part 1: Deutsch’s 8 Fallacies of Distributed Computing
In 1994, L. Peter Deutsch and fellow Sun Microsystems architects articulated eight false assumptions that developers make when transitioning from single-node applications to distributed networks:
+-----------------------------------------------------------------------------+
| The 8 Fallacies of Distributed Computing |
| |
| 1. The network is reliable. |
| 2. Latency is zero. |
| 3. Bandwidth is infinite. |
| 4. The network is secure. |
| 5. Topology doesn't change. |
| 6. There is one administrator. |
| 7. Transport cost is zero. |
| 8. The network is homogeneous. |
+-----------------------------------------------------------------------------+
The Unreliable Network Reality
Unlike in-memory method calls where execution either completes or throws a stack exception on the local CPU, an RPC network call has three potential outcomes:
- Success: Request reached server, server processed it, response returned.
- Failure: Network packet dropped or remote server crashed before processing.
- Ambiguity (The Black Hole): Request reached remote server and executed, but the ACK response packet dropped on the network return path.
When an RPC call times out, the client cannot distinguish between outcome 2 and outcome 3. Retrying a non-idempotent operation (like chargeCreditCard()) can result in duplicate billing.
Part 2: The CAP Theorem Deconstructed
Formulated by Eric Brewer in 2000 and proven mathematically by Seth Gilbert and Nancy Lynch in 2002, the CAP Theorem governs shared-data systems:
Consistency (C)
/ \
/ \
/ \
/ \
/ CP \ CA (Impossible in Distributed Networks)
/ \
/ \
Availability (A) -------------- Partition Tolerance (P)
AP
The 3 CAP Properties
- Consistency (C): Every read receives the most recent write or an error (Linearizability / Strong Consistency).
- Availability (A): Every non-failing node returns a non-error response for every request (without guaranteeing it contains the most recent write).
- Partition Tolerance (P): The system continues operating despite network packet drops or network partitions between nodes.
Why “Pick Any Two” is a Misleading Myth
Many textbooks claim CAP means you can pick CP, AP, or CA. This is false.
In distributed networks, network partitions (P) are physical hardware facts of life (cable cuts, switch crashes, firewall misconfigurations). Therefore, Partition Tolerance (P) is mandatory.
You cannot choose CA. Your real architectural choice during a network partition is between CP and AP:
- CP (Consistency + Partition Tolerance): When a network partition occurs, isolated nodes reject reads and writes to prevent serving stale or split-brain data. (e.g., etcd, ZooKeeper, HBase).
- AP (Availability + Partition Tolerance): Nodes continue accepting reads and writes during a partition, accepting that nodes on opposite sides of the partition will temporarily diverge. (e.g., Cassandra, DynamoDB with eventual consistency).
Part 3: Beyond CAP: The PACELC Theorem
In 2012, Daniel Abadi recognized that the CAP Theorem only describes system behavior during a network partition. But network partitions are rare; systems spend 99.9% of their time operating normally.
The PACELC Theorem extends CAP to describe trade-offs during normal execution:
IF Partition (P):
Choose between Availability (A) OR Consistency (C)
ELSE (E):
Choose between Latency (L) OR Consistency (C)
+-----------------------------------------------------------------------------+
| PACELC Trade-Off Matrix |
| |
| System | Partition Mode (P/A or P/C) | Normal Mode (E/L or E/C) |
| ----------------+-----------------------------+-------------------------|
| DynamoDB / | PA | EL |
| Cassandra | (Favors Availability) | (Favors Low Latency) |
| ----------------+-----------------------------+-------------------------|
| MongoDB | PC | EC |
| ----------------+-----------------------------+-------------------------|
| Spanner / | PC | EC |
| CockroachDB | (Favors Consistency) | (Favors Consistency) |
+-----------------------------------------------------------------------------+
The Latency vs Consistency Trade-Off (EL vs EC)
During normal operations (no network partition):
- If you choose Consistency (EC), writes must wait for cross-node replication acknowledgments (e.g., quorum consensus), increasing write latency.
- If you choose Latency (EL), writes return immediately after hitting a single local node, sacrificing immediate read consistency for speed.
Part 4: Handling Network Partitions in Java Code
When designing microservice clients, never allow raw network sockets to block indefinitely. Always wrap network invocations with explicit connection and read timeouts.
package com.example.network;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
public class ResilientRpcClient {
private final HttpClient httpClient;
public ResilientRpcClient() {
// Enforce strict connection and execution timeouts to prevent thread starvation
this.httpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofMillis(500)) // Max 500ms for TCP handshake
.build();
}
public String executeRequest(String serviceUrl) {
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(serviceUrl))
.timeout(Duration.ofMillis(2000)) // Max 2000ms read timeout
.GET()
.build();
try {
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
if (response.statusCode() == 200) {
return response.body();
}
throw new RuntimeException("Downstream HTTP error: " + response.statusCode());
} catch (Exception ex) {
// Log ambiguous failure and delegate to fallback strategy
System.err.println("[RPC FAILURE] Network call failed: " + ex.getMessage());
return fetchFallbackCachedData();
}
}
private String fetchFallbackCachedData() {
return "{\"status\": \"FALLBACK_DATA\", \"stale\": true}";
}
}
Next Steps
Now that we understand the fallacies of distributed computing, CAP, and PACELC, we will explore time in distributed systems in Part 2: examining physical clock drift, NTP skew, and Lamport Timestamps.
References & Further Reading
- Deutsch, L. P. (1994). The Eight Fallacies of Distributed Computing. Sun Microsystems Technical Report.
- Gilbert, S., & Lynch, N. (2002). Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services. ACM SIGACT News, 33(2), 51–59.
- Abadi, D. (2012). Consistency Tradeoffs in Modern Distributed Database System Design (PACELC). IEEE Computer, 45(2), 37–42.
Part 2: Time in Distributed Systems: Physical Clock Skew, NTP Drift, and Lamport Timestamps
Continue to Part 2 →