Explain to me the functionality of Kafka in a Big-Data Cluster

0 votes

I have finished setting up a Big Data cluster. I have used Apache Kafka and zookeeper. I am unable to understand its functionality in a cluster. When exactly are both required and for what purpose?

Apr 30 in Apache Kafka by nitinrawat895
• 10,690 points
82 views

1 answer to this question.

0 votes

Let me explain to you about Apache Kafka in brief.

Apache Kafka is a distributed publish-subscribe messaging system which was originally developed at LinkedIn and later on became a part of the Apache project. Kafka is fast, agile, scalable and distributed by design which has the following components.

  • Zookeeper
  • Producer 
  • Consumer
  • Broker
Apache ZooKeeper is a software project of the Apache Software Foundation. It is essentially a centralized service for distributed systems to a hierarchical key-value store, which is used to provide a distributed configuration service, synchronization service, and naming registry for large distributed systems.
Producer:
A producer can be any application that can publish messages to a topic.
Consumer:
A consumer can be any application that subscribes to topics and consumes the messages.
Broker:
Kafka cluster is a set of servers, each of which is called a broker.

What’s the role of ZooKeeper?

Each Kafka broker coordinates with other Kafka brokers using ZooKeeper. Producers and Consumers are notified by the ZooKeeper service about the presence of new brokers or failure of the broker in the Kafka system.

single-node-single-broker

Single Node Single Broker

single-node-multiple-brokers

Single Node Multiple Brokers

multiple-node-multiple-broker Multiple Nodes Multiple Brokers

answered Apr 30 by ravikiran
• 4,560 points

Related Questions In Apache Kafka

0 votes
1 answer

What is the optimum number of producers to create in Kafka?

In general, a single producer for all ...READ MORE

answered Jul 10, 2018 in Apache Kafka by nitinrawat895
• 10,690 points
1,499 views
0 votes
1 answer

Is there any change in consumer offsets if a new partition(s) is added to a Kafka topic?

Yes, it stays the same. An offset is ...READ MORE

answered Jul 9, 2018 in Apache Kafka by nitinrawat895
• 10,690 points
318 views
0 votes
10 answers

Writing the Kafka consumer output to a file

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, ...READ MORE

answered Dec 7, 2018 in Apache Kafka by Harsh
7,047 views
0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,300 points
2,480 views
0 votes
1 answer

How to commit message offsets in Kafka for reliable data pipeline?

You can use auto.commit.enable to allow Kafka ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,300 points
348 views
0 votes
1 answer

How to delete a topic in Kafka 0.8.1.1?

Deleting topic isn't always working in 0.8.1.1 Deletion ...READ MORE

answered Sep 4, 2018 in Apache Kafka by nitinrawat895
• 10,690 points
162 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,690 points
3,085 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
15,153 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
1,127 views
0 votes
1 answer