What is Apache Kafka
Main Concepts
Event
- A record that something actually happened, i.e., a message When you read/write to Kafka, you do this in the form of events
Producers
Use the producer api to produce and send events
Topics
- Events are organized and durably organized into topics
- Topics are categorized into which the events are published that trigger other actions
Consumers
Those who subscribe (read and process) the events sent by producers
Real time processing(Streams)
Partitions for Scalability and Performance
Consumer Groups for scalable Message Consumption
Kafka Brokers
- A server that stores data(messages), topics, and manages message distribution to consumers
- Fault tolerance since topic partitions are distributed across multiple brokers
- Each partition has a leader broker and multiple replicas Apache Kafka - stores messages on disk for a certain configurable retention period Benefits of data retention
- Real-time processing of data
- consumers can read multiple times they want
- replay of messages, debugging of historical data Apache Kafka focuses on event streaming and long-term data retention Optimized for high throughput
Zookeper
Cluster Management
maintains a registry of all brokers in the cluster
Leader Election
Each Kafka partition has a leader broker
Metadata Configuration
And traditionally, Kfaka used an external tool Zookeeper for managing metadata and coordination of tasks in a distributed system
External dependency for Kafka
The newer versions of Kafka from v3 now use KRaft
Kafka version 3.0 and above
- removed the dependency for zookeeper and introduced KRaft
- And metadata is now being managed natively within Kafka brokers
- Raft consensus Algorithm for leader election
