- Pub sub messaging system.
- In pub sub you have publishers (producers) and subscribers (consumers) of messages.
- Producers and consumers send messages to a specific location called Topic. (Grouping or collection of topics)
- Topics can be defined up front or on demand.
- Producers know the topic name and have permission to send to it.
- Consumers have permissions to read topics that it’s interested in.
- Messages and topics are stored in Broker.
- The broker is the daemon process that runs on a machine or a container.
- Broker has access to file system that it uses to store topics and messages.
- Kafka can scale horizontally by increasing the number of brokers.
- Cluster - Grouping of Kafka brokers.
- Controller, worker, tasks.
- Controller => Leader => Peers
- Replication factor - Ability to protect against loss.
- Work - Brokers receive messages, categorize them into topics and reliably persisting them for eventual retrieval.
- The effort to handle messages from the producer is substantially less than what is required by the consumer.
Distributed systems - Communication and Consensus
- Worker node membership and naming
- Configuration management
- Leader election
- Health status
- Configuration information
- Health status
- Group membership
- Distributed system consisting of multiple nodes in “ensemble” aka cluster.
- Apache Zookeeper provides Kafka with the metadata it needs.
Creating your first topic
kafka-topics --create --topic foo --partitions 1 --replication-factor 1 --if-not-exists --zookeeper zookeeper:2181
kafka-topics --describe --topic foo --zookeeper zookeeper:2181
comments powered by Disqus