Kafka Storage Internals: Log Segments, Sparse Indexes & Log Compaction

Suppose a single partition in your Kafka cluster has been running for six months, accumulating 500 gigabytes of log data.

A consumer service recovers from a network outage and asks to read messages starting at offset 42,500,102.

If Kafka stored all partition data inside one monolithic 500GB file, locating offset 42,500,102 would require scanning through hundreds of gigabytes of raw data from offset 0. Disk read performance would collapse.

Furthermore, physical disk space is finite. A system that appends data forever without purging historical files eventually crashes the host OS.

Kafka solves log navigation and storage cleanup using three techniques: Log Segments, Sparse Memory-Mapped Indexes, and Log Compaction.

---

Log Segments: Splitting the Append-Only Log

Kafka does not store a partition inside a single continuous file. It splits a partition log into smaller file chunks called Log Segments.

PARTITION DIRECTORY: /var/lib/kafka/data/orders-0/

├── 00000000000000000000.log  (Base Offset: 0)
├── 00000000000000000000.index
├── 00000000000000000000.timeindex
├── 00000000000000004500.log  (Base Offset: 4,500)
├── 00000000000000004500.index
├── 00000000000000004500.timeindex
└── 00000000000000009000.log  (Active Segment: Base Offset 9,000)

Segment File Naming Rule

Each segment file is named after its base offset (the offset of the first record stored inside that segment), padded to 20 digits with leading zeros.

Active vs Inactive Segments

Only the single latest segment file (the active segment) receives incoming writes. When the active segment reaches a size threshold (segment.bytes = 1GB) or time threshold (segment.ms = 7 days), Kafka closes it and rolls a new active segment file.

---

Sparse Memory-Mapped Indexes (.index and .timeindex)

To find offset 4,620 without scanning through gigabytes of payload data, Kafka maintains two index files for every log segment:

1. .index: Maps logical relative offsets to physical byte positions in the .log file.
2. .timeindex: Maps timestamps to logical relative offsets for time-based queries.

Sparse Indexing Mechanics

Instead of recording an index entry for every single message (which would inflate index size), Kafka uses sparse indexing.

By default (index.interval.bytes = 4096), Kafka writes an entry to the .index file only after every 4KB of log data.

SPARSE OFFSET INDEX (.index)             LOG DATA FILE (.log)
Relative Offset | Physical Position      Physical Byte Offset | Record
----------------|------------------      ---------------------|----------------------
Offset 0        | Position 0        ---> Position 0           | Msg at Offset 4500
Offset 4        | Position 4120     ---> Position 4120        | Msg at Offset 4504
Offset 9        | Position 9200     ---> Position 9200        | Msg at Offset 4509

Microsecond Binary Search Workflow

When a consumer requests offset 4,506:

1. Find Segment File: Kafka runs binary search on segment base offsets to identify the target segment file (00000000000000004500.log).
2. Find Index Range: Kafka runs binary search on the sparse .index file. Relative offset 6 lies between index entry 4 (position 4120) and index entry 9 (position 9200).
3. Sequential Scan: Kafka jumps straight to physical byte position 4120 in the .log file and scans sequentially for a few hundred bytes until it reaches offset 4,506.

Because index files are memory-mapped (mmap) into kernel RAM, message lookup executes in microseconds.

---

Storage Cleanup Strategies: Deletion vs Compaction

Kafka provides two retention policies for purging old data (cleanup.policy):

1. Delete Policy (cleanup.policy = delete)

Old inactive segment files are purged based on age or total partition size:

• Time-Based Deletion (log.retention.hours = 168): Deletes segment files older than 7 days.
• Size-Based Deletion (log.retention.bytes = 107374182400): Deletes the oldest segment files when partition size exceeds 100GB.

Deletion happens at the segment file level. Kafka simply deletes old .log and .index segment files from disk.

---

2. Log Compaction (cleanup.policy = compact)

For key-value streams (such as user account profiles, stock prices, or state store restore topics), consumers only care about the latest value for a given key.

Log Compaction ensures that Kafka retains at least the last known value for every message key within a partition.

BEFORE COMPACTION:
Offset:   0        1        2        3        4        5
Key:    "K1"     "K2"     "K1"     "K3"     "K2"     "K1"
Val:   "v1.0"   "v1.0"   "v1.1"   "v1.0"   "v2.0"   "v1.2"

AFTER COMPACTION:
Offset:   3        4        5
Key:    "K3"     "K2"     "K1"
Val:   "v1.0"   "v2.0"   "v1.2"  (Obsolete offsets 0, 1, 2 purged!)

Tombstone Deletion

If a producer publishes a key with a null payload (a Tombstone record), Log Compaction retains the tombstone for a period (delete.retention.ms), allowing consumers to detect key deletions before purging the key completely.

---

Quick Summary

• Partitions split into smaller file chunks called Log Segments named after their base offset.
• Sparse memory-mapped .index files map relative offsets to physical byte positions every 4KB, enabling microsecond binary search lookups.
• The delete retention policy purges entire inactive segment files based on age or size.
• Log Compaction retains the latest value for every key, freeing disk space without losing current domain state.

References & Further Reading

1. Apache Kafka Wiki. KIP-98: Exactly Once Delivery and Transactional Messaging. Kafka Improvement Proposals.
2. Apache Software Foundation. Apache Kafka Documentation: Idempotent Producer Architecture. Apache Kafka Docs.
3. Vogels, W. (2009). Eventually Consistent. Communications of the ACM, 52(1), 40–44.

---