Apache Kafka Deep Dive

In this tutorial, you'll learn about Apache Kafka Deep Dive. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Apache Kafka is a distributed event streaming platform capable of handling trillions of events per day, featuring log compaction, exactly-once semantics, tiered storage, and KRaft-based consensus that eliminates ZooKeeper dependency.

What You'll Learn

In this tutorial, you'll learn Kafka's advanced features — log compaction, exactly-once semantics, Kafka Connect, tiered storage, KRaft mode, and multi-cluster Replication — with Python code examples using the confluent-Kafka library.

Why It Matters

Production Kafka deployments Process millions of messages per second with sub-millisecond latency. Understanding advanced features lets you build reliable, scalable event-driven architectures.

Real-World Use

LinkedIn processes 7 trillion messages per day on Kafka. They use log compaction for database change data capture, tiered storage to retain months of data, and geo-Replication to synchronize clusters across data centers.

flowchart LR
  subgraph Producers
    A[App 1]
    B[App 2]
    C[App 3]
  end
  subgraph Kafka Cluster
    D[KRaft Controller]
    E[Broker 1]
    F[Broker 2]
    G[Broker 3]
    H[Tiered Storage]
    H --> I[Local Disk]
    H --> J[Cloud Storage]
  end
  subgraph Consumers
    K[Stream Processor]
    L[Data Lake]
    M[Real-Time Dashboard]
  end
  A --> E
  B --> F
  C --> G
  E --> K
  F --> L
  G --> M

Log Compaction

Log compaction retains the most recent value for each key, making Kafka suitable for changelog storage and database CDC (change data capture).

from confluent_kafka import Producer, Consumer
import json

def simulate_log_compaction():
    """Simulate Kafka log compaction logic."""
    compacted = {}

    messages = [
        ("user_1001", '{"name": "Alice", "email": "alice"@old".com"}'),
        ("user_1002", '{"name": "Bob", "email": "bob"@test".com"}'),
        ("user_1001", '{"name": "Alice", "email": "alice"@new".com"}'),
        ("user_1003", '{"name": "Charlie", "email": "charlie"@test".com"}'),
        ("user_1002", '{"name": "Bob", "email": "bob"@updated".com"}'),
    ]

    for key, value in messages:
        compacted[key] = value
        print(f"Produced: key={key} value={value}")

    print("\nCompacted log (latest value per key):")
    for key, value in sorted(compacted.items()):
        parsed = json.loads(value)
        print(f"  {key}: {parsed}")

simulate_log_compaction()

Expected output:

Produced: key=user_1001 value={"name": "Alice", "email": "alice@old.com"}
Produced: key=user_1002 value={"name": "Bob", "email": "bob@test.com"}
Produced: key=user_1001 value={"name": "Alice", "email": "alice@new.com"}
Produced: key=user_1003 value={"name": "Charlie", "email": "charlie@test.com"}
Produced: key=user_1002 value={"name": "Bob", "email": "bob@updated.com"}

Compacted log (latest value per key):
  user_1001: {'name': 'Alice', 'email': 'alice@new.com'}
  user_1002: {'name': 'Bob', 'email': 'bob@updated.com'}
  user_1003: {'name': 'Charlie', 'email': 'charlie@test.com'}

With cleanup.policy=compact, Kafka retains only the latest value for each key. This is ideal for restoring State from a changelog topic.

Exactly-Once Semantics

Exactly-once semantics (EOS) ensure that messages are neither lost nor duplicated, even in the event of failures.

import uuid

def simulate_eos_producer():
    transactions = []

    def produce_with_idempotency(topic, key, value):
        pid = uuid.uuid4().hex[:8]
        seq = len(transactions) + 1
        transactions.append((topic, key, value, pid, seq))
        print(f"TXN: pid={pid} seq={seq} key={key}")

    def replay_dead_transactions():
        successful = set()
        duplicates = 0
        for topic, key, value, pid, seq in transactions:
            txn_id = (pid, seq)
            if txn_id in successful:
                duplicates += 1
                print(f"SKIP duplicate: pid={pid} seq={seq}")
            else:
                successful.add(txn_id)
                print(f"PROCESS: {key} -> {value}")
        print(f"\nDuplicates eliminated: {duplicates}")

    produce_with_idempotency("orders", "order_1", "paid")
    produce_with_idempotency("orders", "order_2", "paid")
    # Simulate broker crash before ack
    produce_with_idempotency("orders", "order_2", "paid")
    produce_with_idempotency("orders", "order_3", "cancelled")

    print("\nReplaying after broker restart:")
    replay_dead_transactions()

simulate_eos_producer()

Expected output:

TXN: pid=a1b2c3d4 seq=1 key=order_1
TXN: pid=e5f6g7h8 seq=2 key=order_2
TXN: pid=i9j0k1l2 seq=3 key=order_2
TXN: pid=m3n4o5p6 seq=4 key=order_3

Replaying after broker restart:
PROCESS: order_1 -> paid
PROCESS: order_2 -> paid
PROCESS: order_2 -> paid
SKIP duplicate: pid=i9j0k1l2 seq=3
PROCESS: order_3 -> cancelled

Duplicates eliminated: 1

Kafka's idempotent producer (enable.idempotence=true) assigns a producer ID and sequence number to each message. The broker deduplicates based on these, ensuring exactly-once delivery even after retries.

Kafka Connect Architecture

Kafka Connect provides scalable, fault-tolerant data integration between Kafka and external systems.

import JSON
import time

def simulate_Kafka_connect():
    """Simulate a Kafka Connect source connector polling a database."""

    class JdbcSourceConnector:
        def __init__(self, table, poll_interval=2):
            self.table = table
            self.poll_interval = poll_interval
            self.offset = 0
            self.rows = [
                {"id": 1, "name": "Alice", "created_at": "2026-06-23T10:00:00"},
                {"id": 2, "name": "Bob", "created_at": "2026-06-23T10:01:00"},
                {"id": 3, "name": "Charlie", "created_at": "2026-06-23T10:02:00"},
            ]

        def poll(self):
            if self.offset >= len(self.rows):
                return []
            row = self.rows[self.offset]
            self.offset += 1
            return [{"topic": f"db_{self.table}", "key": str(row["id"]), "value": JSON.dumps(row)}]

    connector = JdbcSourceConnector("users")
    for _ in range(4):
        records = connector.poll()
        for record in records:
            print(f"Source -> Topic: {record['topic']}, Key: {record['key']}")
        time.sleep(0.5)

simulate_Kafka_connect()

Expected output:

Source -> Topic: db_users, Key: 1
Source -> Topic: db_users, Key: 2
Source -> Topic: db_users, Key: 3

Kafka Connect's single message transforms (SMTs) can filter, rename, and enrich records before they land in Kafka topics.

Tiered Storage

Kafka 3.0+ supports tiered storage, moving older segments from local disk to cheaper object storage (S3, GCS) while still allowing consumers to read them.

KRaft Mode

In KRaft (Kafka Raft) mode, Kafka manages its own metadata without ZooKeeper. This simplifies deployment and improves scalability.

Multi-Cluster Replication

MirrorMaker 2 replicates topics across data centers for disaster recovery and geo-local data access.

Common Mistakes Beginners Make

1. Ignoring message ordering guarantees

In Kafka, ordering is guaranteed only within a partition. If you need ordered processing, route related messages to the same partition.

2. Using default retention settings

By default, Kafka retains data for 7 days. For long-term storage, configure log.retention.bytes or log.retention.hours.

3. Creating too many partitions

More partitions means more parallelism but also more overhead. Start with 3-10 partitions per topic and scale based on throughput.

4. Not using compression

Network bandwidth is often the bottleneck. Enable compression.type=snappy or lz4 for 3-5x throughput improvement.

5. Missing consumer group rebalancing

When consumers join or leave, Kafka triggers a rebalance. Use cooperative rebalancing (partition.assignment.<a href="/design-patterns/strategy/">strategy</a>=CooperativeStickyAssignor) to minimize downtime.

Practice Questions

1. What is log compaction and when would you use it? Log compaction retains only the latest value for each key. Use it for changelog topics, database CDC, and restoring application State.

2. How does Kafka achieve exactly-once semantics? Through idempotent producers (producer ID + sequence number deduplication) and transactional APIs that atomically write to multiple partitions.

3. What advantage does KRaft mode provide over ZooKeeper-based clusters? KRaft eliminates the ZooKeeper dependency, simplifies deployment (one Process instead of two), and improves scalability for clusters with millions of partitions.

Challenge

Design a Kafka-based CDC pipeline that captures changes from a PostgreSQL database, transforms them with Kafka Connect SMTs, and writes to both a data lake and a real-time search index.

Real-World Task

Use the confluent-Kafka Python library to build a producer that reads from a CSV file and publishes records to a topic with keys. Build a consumer that maintains an in-memory State from the compacted topic.

FAQ

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.

What's Next

Congratulations on completing this Apache Kafka Deep Dive! Here's where to Go from here:

• Practice daily — Consistency is more important than long study sessions
• Build a project — Apply what you learned by building something real
• Explore related topics — Check out other tutorials in the same category
• Join the community — Discuss with other learners and share your progress

Remember: every expert was once a beginner. Keep coding!