Articles tagged with #Kafka
A curated list of engineering series, deep dives, and notes related to #Kafka.
Building a Real-Time Event-Driven Order System with Apache Kafka
Build an end-to-end event-driven microservices architecture in Java using Apache Kafka, Schema Registry, Avro, Dead Letter Topics, and Kafka Streams.
Operating Kafka in Production: Critical JMX Metrics, Kernel Tuning & Runbooks
SRE runbook for operating Kafka in production. Monitor critical JMX metrics (UnderReplicatedPartitions, ConsumerLag), tune Linux sysctl, and troubleshoot outages.
Kafka Connect vs Kafka Streams: Zero-Code ELT Pipelines vs Real-Time Analytics
Compare Kafka Connect vs Kafka Streams. Learn when to use zero-code ELT connectors versus in-process Java stream processing with embedded RocksDB.
Kafka Schema Registry & Avro: Guarding Against Breaking Payload Changes
Prevent production microservice crashes with Confluent Schema Registry and Apache Avro. Learn 5-byte wire format headers and compatibility modes.
Kafka Exactly-Once Semantics (EOS): Idempotent Producers & 2PC Transactions
Learn how Kafka achieves Exactly-Once Semantics (EOS) across read-process-write stream loops using Idempotent Producers, Transactional Coordinators, and 2PC.
Distributed Message Queues: Log-Based (Kafka) vs AMQP Broker-Based (RabbitMQ)
Master distributed message queues. Compare log-based event streaming (Kafka) with AMQP smart broker routing (RabbitMQ).
Kafka KRaft Consensus Mode: Replacing ZooKeeper for Million-Partition Scale
Understand KIP-500 and KRaft mode in Kafka. Learn how self-managed Raft metadata consensus eliminates ZooKeeper and unlocks million-partition scale.
Kafka Partition Replication: High Watermark, LEO & In-Sync Replicas (ISR)
Learn how Kafka achieves high availability without data corruption using leader/follower replication, High Watermarks, ISR tracking, and Leader Epochs.
Kafka Consumer Rebalancing: Eager Storms vs Cooperative Sticky Assignors
Eliminate stop-the-world consumer processing outages in Kafka. Compare legacy Eager Rebalancing with modern Cooperative Sticky Assignors.
Kafka Offset Management: Manual Commits, __consumer_offsets & Message Replay
Master Kafka offset management and delivery semantics. Implement manual commits to achieve At-Least-Once processing and seek historical offsets.
Kafka Consumer Groups & Pull Model: Scale-Out Processing Without Lock Contention
Learn how Kafka Consumer Groups scale out event processing. Understand why Kafka uses a Pull Model for native backpressure protection.
Kafka Producer Reliability: Balancing acks=all, min.insync.replicas & Data Loss
Understand Kafka producer durability trade-offs. Configure acks=all and min.insync.replicas=2 to guarantee zero data loss on critical event streams.
Kafka Partition Routing: MurmurHash2 Keys, Sticky Partitioning & Idempotence
Master Kafka message ordering. Learn how MurmurHash2 routes keyed records, how Sticky Partitioning optimizes null keys, and how idempotence fixes out-of-order retries.
Kafka Producer Internals: Tuning RecordAccumulator, batch.size & linger.ms
Deep dive into KafkaProducer mechanics. Learn how RecordAccumulator, batch.size, linger.ms, and ZSTD batch compression maximize streaming throughput.
Kafka Storage Internals: Log Segments, Sparse Indexes & Log Compaction
Learn how Kafka locates any message in microseconds using sparse memory-mapped index files (.index) and prunes stale keys using Log Compaction.
Kafka Zero-Copy Optimization: How sendfile() Streams Millions of Events/Sec
Discover how Kafka uses Java NIO transferTo() and Linux sendfile() Zero-Copy optimization to stream gigabits of data per second with minimal CPU load.
Kafka Architecture Deep Dive: Topics, Partitions, and Offset Ordering Rules
Deconstruct Kafka storage anatomy: Topics, Partitions, and Offsets. Learn how partition sharding scales write throughput and consumer parallelism.
The Append-Only Log Abstraction: Why Immutability Rules Event Streaming
Explore the append-only log data structure behind Apache Kafka. Learn how immutability enables lock-free concurrency and multi-team data replay.
Kafka Performance Secrets: Why Sequential Disk I/O Beats Random RAM Access
Learn why Kafka stores all events on disk. Understand how sequential disk writes and Linux OS Page Cache bypass JVM Garbage Collection pauses.
Why Apache Kafka Exists: Solving Microservice N² Integration Spaghetti
Discover why Apache Kafka was created at LinkedIn. Learn how central append-only event logs solve microservice point-to-point integration spaghetti.