Kafka Performance Secrets: Why Sequential Disk I/O Beats Random RAM Access

Ask any software engineer why traditional messaging systems are slow, and they will likely give you the standard answer: “Because they write messages to disk, and disk I/O is slow.”

This belief led developers to design in-memory message brokers that store all active messages inside RAM.

However, when Jay Kreps and the engineering team at LinkedIn designed Kafka, they made a radical architectural choice: Kafka stores every single message directly on disk from the second it arrives.

Despite writing every record to physical disk, Kafka streams millions of messages per second per node.

How can a disk-backed storage engine outperform in-memory systems?

Because disk I/O speed depends entirely on how you access the disk hardware.

The Hardware Reality: Sequential vs Random I/O

A physical hard disk drive (HDD) consists of spinning magnetic platters and read/write heads mounted on a mechanical actuator arm. Solid-State Drives (SSDs) use flash memory cells organized in blocks and pages.

When an application performs random I/O (seeking across random file locations), physical hardware pays a massive penalty:

On an HDD, the physical actuator arm must seek across magnetic tracks. An HDD handles roughly 100 to 200 random I/O operations per second (IOPS), topping out at ~1MB/s throughput.
On an SSD, random writes force expensive block erase-and-rewrite cycles, triggering flash translation layer wear and write amplification.

However, when an application performs sequential I/O (appending data continuously to the end of a contiguous file), mechanical disk heads do not seek, and SSD controllers stream data across parallel flash channels.

HARDWARE THROUGHPUT SPECTRUM

Random Disk Writes (HDD):   | 1 MB/s
Random RAM Access:          | 10-20 MB/s (Cache misses)
Random SSD Writes:          | 50-100 MB/s
Sequential RAM Access:      | 10,000 MB/s
Sequential Disk Writes:     | 600 - 2,000 MB/s (SATA / NVMe SSD)

Sequential disk throughput often approaches or exceeds the throughput of random RAM access.

Kafka structures its disk storage as an append-only sequential log. It never updates old messages in place and never seeks backward during writes.

Leveraging the OS Page Cache Instead of JVM Heap

Traditional Java applications allocate large in-memory byte buffers inside the JVM heap to cache data.

This approach creates two severe problems:

Java Object Overhead: Storing raw data as Java objects doubles or triples the memory footprint due to object headers and reference padding.
Garbage Collection Pauses: Maintaining a 50GB in-memory cache inside the JVM heap forces the Java Garbage Collector to perform long Stop-The-World (STW) pauses during major GC cycles.

Kafka bypasses the JVM heap for caching. It delegates memory management directly to the Linux Operating System Page Cache.

JVM HEAP MEMORY                                 LINUX OS PAGE CACHE (System RAM)
+------------------------------------+          +------------------------------------+
| Kafka JVM Process (~4GB allocation)|          | OS Kernel Free Memory (~60GB RAM)  |
|                                    |          |                                    |
| Minimal heap memory footprint.     |          | Caches raw disk file blocks        |
| Zero GC pause overhead!            |          | automatically at native CPU speed. |
+------------------------------------+          +------------------------------------+

When Kafka writes a message to disk via standard OS file APIs, the operating system writes the data into its kernel Page Cache memory first.

If a consumer reads that message shortly after it was published, the OS serves the read directly from Page Cache RAM without hitting physical disk tracks.

If the Kafka JVM process crashes or restarts, the OS Page Cache remains intact in kernel memory. No cache warm-up period is required!

Page Cache Flushing: Write-Back Mechanics

When Kafka appends a record to a log segment file, it relies on OS background kernel threads (pdflush / flush) to write dirty page cache memory to physical disk blocks asynchronously.

Instead of calling fsync() after every single incoming message (which forces physical disk head flushes), Kafka lets the operating system handle disk flushing in large, contiguous background batches.

Durability is achieved through cluster replication across multiple independent broker nodes rather than forcing blocking per-message disk syncs on a single machine.

Real-World Benchmark: Sequential Log Write Performance

Consider this simple benchmark comparing sequential log appends against random record updates:

import java.io.File;
import java.io.FileOutputStream;
import java.io.RandomAccessFile;

public class DiskBenchmark {
    public static void main(String[] args) throws Exception {
        int recordCount = 1_000_000;
        byte[] payload = new byte[100]; // 100-byte message payload

        // Benchmark 1: Sequential Append (Kafka Style)
        File seqFile = new File("sequential.log");
        long startSeq = System.currentTimeMillis();
        try (FileOutputStream out = new FileOutputStream(seqFile)) {
            for (int i = 0; i < recordCount; i++) {
                out.write(payload);
            }
        }
        long seqTime = System.currentTimeMillis() - startSeq;
        System.out.println("Sequential Append Time: " + seqTime + " ms");

        // Benchmark 2: Random Seek Writes (Traditional DB Style)
        File randFile = new File("random.db");
        long startRand = System.currentTimeMillis();
        try (RandomAccessFile raf = new RandomAccessFile(randFile, "rw")) {
            for (int i = 0; i < 10_000; i++) { // Only 10,000 iterations due to slowness!
                long randomOffset = (long) (Math.random() * 1000) * 100;
                raf.seek(randomOffset);
                raf.write(payload);
            }
        }
        long randTime = System.currentTimeMillis() - startRand;
        System.out.println("10,000 Random Writes Time: " + randTime + " ms");
    }
}

On a standard NVMe SSD, appending 1,000,000 sequential records takes ~150 milliseconds (~660 MB/s). Executing 10,000 random seeks takes several seconds.

Quick Summary

Disk I/O is slow for random access, but fast for sequential appends.
Kafka structures storage as append-only logs, achieving disk write speeds that rival RAM access.
Kafka delegates caching to the Linux OS Page Cache instead of JVM heap memory, avoiding Garbage Collection pauses and Java object overhead.
Per-message fsync() is replaced by asynchronous OS background flushing combined with multi-node replication.

References & Further Reading

Linux Kernel Organization. Page Cache & Page Writeback Architecture. Linux Kernel Docs.
Apache Software Foundation. Apache Kafka Documentation: Efficiency / Pagecache and Zero-Copy. Apache Kafka Docs.
Gregg, B. (2020). Systems Performance: Enterprise and the Cloud (2nd Edition) — Chapter 8: File Systems. Addison-Wesley.