Kafka Brokers, Topics, and Partitions

Core Components

Brokers: Store data and serve reads/writes. Topics: Logical categories for messages. Partitions: Parallel units within a topic. Replicas: Copies for fault tolerance.

Message Flow

Producer -> Broker (leader partition) -> Replicated to followers -> Consumer Group reads from assigned partitions -> Offset committed after processing.

Ordering Guarantees

Order is guaranteed WITHIN a partition, NOT across partitions. Design partition keys to ensure related messages land in the same partition.

Code Example

Producer configuration

producer = KafkaProducer(
bootstrap_servers=['broker1:9092', 'broker2:9092'],
value_serializer=lambda v: json.dumps(v).encode('utf-8'),
acks='all', # Wait for all replicas
retries=3
)

Send with partition key

producer.send('orders', value=order_data, key=order_id.encode())

Consumer configuration

consumer = KafkaConsumer(
'orders',
group_id='etl-pipeline',
auto_offset_reset='earliest',
enable_auto_commit=False # Manual commit for exactly-once
)

Best Practices

• Choose partition count based on throughput
• Use consumer groups for parallelism
• Monitor consumer lag
• Use schema registry for data contracts

Interview Tips

• Explain partitioning and ordering
• Discuss consumer group rebalancing
• How to achieve exactly-once semantics?