Explain to me the functionality of Kafka in a Big-Data Cluster

0 votes

I have finished setting up a Big Data cluster. I have used Apache Kafka and zookeeper. I am unable to understand its functionality in a cluster. When exactly are both required and for what purpose?

Apr 30, 2019 in Apache Kafka by nitinrawat895
• 11,380 points
1,133 views

1 answer to this question.

0 votes

Let me explain to you about Apache Kafka in brief.

Apache Kafka is a distributed publish-subscribe messaging system which was originally developed at LinkedIn and later on became a part of the Apache project. Kafka is fast, agile, scalable and distributed by design which has the following components.

  • Zookeeper
  • Producer 
  • Consumer
  • Broker
Apache ZooKeeper is a software project of the Apache Software Foundation. It is essentially a centralized service for distributed systems to a hierarchical key-value store, which is used to provide a distributed configuration service, synchronization service, and naming registry for large distributed systems.
Producer:
A producer can be any application that can publish messages to a topic.
Consumer:
A consumer can be any application that subscribes to topics and consumes the messages.
Broker:
Kafka cluster is a set of servers, each of which is called a broker.

What’s the role of ZooKeeper?

Each Kafka broker coordinates with other Kafka brokers using ZooKeeper. Producers and Consumers are notified by the ZooKeeper service about the presence of new brokers or failure of the broker in the Kafka system.

single-node-single-broker

Single Node Single Broker

single-node-multiple-brokers

Single Node Multiple Brokers

multiple-node-multiple-broker Multiple Nodes Multiple Brokers

answered Apr 30, 2019 by ravikiran
• 4,620 points

Related Questions In Apache Kafka

0 votes
1 answer

What is the optimum number of producers to create in Kafka?

In general, a single producer for all ...READ MORE

answered Jul 10, 2018 in Apache Kafka by nitinrawat895
• 11,380 points
8,474 views
0 votes
1 answer

How to start a consumer in Kafka cluster?

Hi@akhtar, In Kafka, each topic is divided into a ...READ MORE

answered Dec 7, 2020 in Apache Kafka by MD
• 95,460 points
789 views
0 votes
1 answer

Is there any change in consumer offsets if a new partition(s) is added to a Kafka topic?

Yes, it stays the same. An offset is ...READ MORE

answered Jul 9, 2018 in Apache Kafka by nitinrawat895
• 11,380 points
2,412 views
+2 votes
9 answers

Writing the Kafka consumer output to a file

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, ...READ MORE

answered Dec 7, 2018 in Apache Kafka by Harsh
38,647 views
0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 10, 2018 in Apache Kafka by Shubham
• 13,490 points
14,034 views
0 votes
1 answer

How to commit message offsets in Kafka for reliable data pipeline?

You can use auto.commit.enable to allow Kafka ...READ MORE

answered Jul 10, 2018 in Apache Kafka by Shubham
• 13,490 points
3,588 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
11,059 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
108,991 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,350 points
4,629 views
0 votes
1 answer
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP