How is kafka used in big-data

Question

A Big-Data cluster is set up using Apache Kafka and its storage unit which is Zookeeper. But how does one understand the use of them and in which scenario?

Answer 1

I can brief you the answer here but in case if you wish to learn more about Kafka, You can read the article linked.

Data accessed from any source such as twitter etc is called as a topic. in a more simpler language, consider a specific TV channel in a collection of channels available. say HBO. then HBO is your topic to stream.

Producer: He/She can be anyone who generates the topic is the producer.

Consumer: Consumer is anyone who requests access for data and also can subscribe to the data.

Broker: The published Topics are then stored in a set of servers called the brokers or Kafka CLuster.

Zookeeper: Zookeeper is the storage system of Apache Kafka. It is a distributed and Hierarchal storage.