Apache Kafka & Event Streaming
Technical essays and low-level architectural breakdowns exploring mechanical execution, system trade-offs, and engine internals.
Mastering Apache Kafka from First Principles: Series Introduction & Learning Roadmap
Discover what you will learn in this 19-part series on Apache Kafka internals. Master distributed append-only logs, zero-copy I/O, and KRaft.
Why Apache Kafka Exists: Solving Microservice N² Integration Spaghetti
Discover why Apache Kafka was created at LinkedIn. Learn how central append-only event logs solve microservice point-to-point integration spaghetti.
Kafka Performance Secrets: Why Sequential Disk I/O Beats Random RAM Access
Learn why Kafka stores all events on disk. Understand how sequential disk writes and Linux OS Page Cache bypass JVM Garbage Collection pauses.
The Append-Only Log Abstraction: Why Immutability Rules Event Streaming
Explore the append-only log data structure behind Apache Kafka. Learn how immutability enables lock-free concurrency and multi-team data replay.
Kafka Architecture Deep Dive: Topics, Partitions, and Offset Ordering Rules
Deconstruct Kafka storage anatomy: Topics, Partitions, and Offsets. Learn how partition sharding scales write throughput and consumer parallelism.
Kafka Zero-Copy Optimization: How sendfile() Streams Millions of Events/Sec
Discover how Kafka uses Java NIO transferTo() and Linux sendfile() Zero-Copy optimization to stream gigabits of data per second with minimal CPU load.
Kafka Storage Internals: Log Segments, Sparse Indexes & Log Compaction
Learn how Kafka locates any message in microseconds using sparse memory-mapped index files (.index) and prunes stale keys using Log Compaction.
Kafka Producer Internals: Tuning RecordAccumulator, batch.size & linger.ms
Deep dive into KafkaProducer mechanics. Learn how RecordAccumulator, batch.size, linger.ms, and ZSTD batch compression maximize streaming throughput.