Kafka Partition Routing: MurmurHash2 Keys, Sticky Partitioning & Idempotence

Suppose you are building a digital wallet application.

User #402 deposits $100 (Event A) at 10:00:00 AM. One second later, User #402 transfers $80 (Event B) to a merchant.

If Event A and Event B land in different partitions, two separate consumer threads process them concurrently.

If Event B is processed first, your system evaluates User #402’s current balance ($0), rejects the $80 transfer due to insufficient funds, and sends an erroneous overdraft alert to the customer.

To prevent out-of-order data processing bugs, Kafka requires all events belonging to the same entity (e.g. User #402) to be routed to the exact same partition.

Why You Need This in Real Life

Partition routing determines how records distribute across cluster hardware:

Entity Order Guarantees: Passing an entity ID as the message key guarantees that all records for that entity land in a single partition, preserving strict chronological processing order.
Load Balancing: Null-key messages must distribute evenly across partitions to prevent hotspotting a single broker node.
Out-of-Order Retry Bugs: Retrying failed network batches can flip event ordering on the broker unless idempotence controls are enabled.

The Hashing Formula: How Keys Map to Partitions

When a record contains a non-null key, Kafka’s DefaultPartitioner calculates the target partition using the MurmurHash2 algorithm:

$\text{Partition} = \left(\text{Utils.murmur2}(\text{keyBytes}) \ \& \ \text{0x7FFFFFFF}\right) \pmod{\text{numPartitions}}$

Record Key: "USER-402"
       |
       v
MurmurHash2("USER-402") ----> 1,489,203,114
       |
       v
Modulo Partitions Count (3) ----> Index 1 (Partition 1)

Because MurmurHash2 is deterministic, any record with key "USER-402" will always route to Partition 1, as long as total partition count remains unchanged.

Warning: If you alter a topic’s partition count (e.g. expanding from 3 to 6 partitions), the modulo calculation changes. Future records for "USER-402" will route to a different partition, breaking historical key-to-partition ordering!

Null-Key Messages: Sticky Partitioning

What happens when a record key is null?

In older Kafka versions (pre-2.4), null-key messages were assigned to partitions using round-robin distribution (Msg 1 -> P0, Msg 2 -> P1, Msg 3 -> P2).

Round-robin created a severe batching problem: small 100-byte batches were queued across all partitions, preventing RecordAccumulator from filling batches to batch.size.

Modern Kafka uses Sticky Partitioning:

STICKY PARTITIONING MECHANISM (null keys)

Batch 1 (Accumulating for Partition 0): [ Record 1 ][ Record 2 ][ Record 3 ] (Fills batch!)
         | (Once Batch 1 fills or lingers...)
Batch 2 (Switches to Partition 1):     [ Record 4 ][ Record 5 ]

The partitioner sticks to a single partition until its batch fills or lingers, then switches to the next partition. This maximizes batch compression and reduces network request overhead while maintaining even distribution over time.

Writing a Custom Partitioner

If your domain logic requires custom sharding (e.g. routing premium enterprise customer events to dedicated partitions), implement Kafka’s Partitioner interface:

import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.common.Cluster;

import java.util.Map;

public class VIPPartitioner implements Partitioner {
    @Override
    public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
        int numPartitions = cluster.partitionsForTopic(topic).size();
        
        if (key != null && key.toString().startsWith("VIP-")) {
            return 0; // Route all VIP accounts exclusively to Partition 0
        }
        
        // Route standard accounts across remaining partitions
        int hash = Math.abs(key != null ? key.hashCode() : value.hashCode());
        return 1 + (hash % (numPartitions - 1));
    }

    @Override
    public void close() {}

    @Override
    public void configure(Map<String, ?> configs) {}
}

Specify your class in producer properties:

props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, VIPPartitioner.class.getName());

The Out-of-Order Retry Bug & How to Fix It

Suppose a producer sends Batch 1 (Offset 0..9) and Batch 2 (Offset 10..19) to Partition 0 concurrently (max.in.flight.requests.per.connection = 5).

1. Producer transmits Batch 1 and Batch 2 over the wire.
2. Network glitch causes Batch 1 to FAIL.
3. Batch 2 succeeds on broker (written to offsets 0..9).
4. Producer retries Batch 1 -> Batch 1 succeeds on broker (written to offsets 10..19).

RESULT: Batch 2 is stored BEFORE Batch 1! Message ordering is broken!

The Solution: Enable Producer Idempotence

To prevent out-of-order retries without capping flight requests to 1, enable producer idempotence:

props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");

When idempotence is enabled (enable.idempotence = true), the broker assigns a Sequence Number to every record batch. If Batch 2 arrives before Batch 1, the broker rejects Batch 2 until Batch 1 arrives, preserving exact chronological log sequence.

Quick Summary

Passing a non-null key routes records deterministically via MurmurHash2(key) % numPartitions.
Null-key messages use Sticky Partitioning to fill batch buffers before switching partition targets.
Expanding topic partition counts changes modulo hashing, altering key-to-partition assignments.
Enabling enable.idempotence = true prevents out-of-order log writes caused by network retries.

References & Further Reading

Apache Kafka Wiki. KIP-429: Kafka Consumer Static Membership. Kafka Improvement Proposals.
Apache Kafka Wiki. KIP-345: Introduce Static Membership to Consumer Groups. Kafka Improvement Proposals.
Confluent Inc. Cooperative Sticky Assignor: Incremental Rebalancing in Kafka. Confluent Blog.