Kafka Cheetsheet/notes

By   02-15-2018 · 2 minute read · 235 words


Architecture

  • Pub sub messaging system.
  • In pub sub you have publishers (producers) and subscribers (consumers) of messages.
  • Producers and consumers send messages to a specific location called Topic. (Grouping or collection of topics)
  • Topics can be defined up front or on demand.
  • Producers know the topic name and have permission to send to it.
  • Consumers have permissions to read topics that it’s interested in.
  • Messages and topics are stored in Broker.
  • The broker is the daemon process that runs on a machine or a container.
  • Broker has access to file system that it uses to store topics and messages.

Scaling

  • Kafka can scale horizontally by increasing the number of brokers.
  • Cluster - Grouping of Kafka brokers.

Distributed systems

  • Controller, worker, tasks.
  • Controller => Leader => Peers
  • Replication factor - Ability to protect against loss.
  • Work - Brokers receive messages, categorize them into topics and reliably persisting them for eventual retrieval.
  • The effort to handle messages from the producer is substantially less than what is required by the consumer.

Distributed systems - Communication and Consensus

  • Worker node membership and naming
  • Configuration management
  • Leader election
  • Health status

Apache zookeeper

  • Configuration information
  • Health status
  • Group membership
  • Distributed system consisting of multiple nodes in “ensemble” aka cluster.
  • Apache Zookeeper provides Kafka with the metadata it needs.

Creating your first topic

kafka-topics --create --topic foo --partitions 1 --replication-factor 1 --if-not-exists --zookeeper zookeeper:2181

Describing topic

kafka-topics --describe --topic foo --zookeeper zookeeper:2181

kafka  

comments powered by Disqus