What is the optimum number of producers to create in Kafka?

0 votes

In a high volume real time java web app I'm sending messages to apache kafka. As of now, I've created single topic and I'm sending to that single topic, but there might be a case when I need to send messages to multiple topics.

In this case I'm not sure weather to create a producer per topic or should I use a single producer to all my topics?

Here is my code:

props = new Properties();
props.put("zk.connect", <zk-ip1>:<2181>,<zk-ip3>:<2181>,<zk-ip3>:<2181>);
props.put("zk.connectiontimeout.ms", "100000");
props.put("producer.type", "async");

Producer<String, Message> producer = new kafka.javaapi.producer.Producer<String, Message>(new ProducerConfig(props));

ProducerData<String, Message> producerData1 = new ProducerData<String, Message>("DemoTopic1", messageTosend);
ProducerData<String, Message> producerData2 = new ProducerData<String, Message>("DemoTopic2", messageTosend);

producer.send(producerData1);
producer.send(producerData2);


If my app sends to multiple topics (each topic gets different data) should I use a single producer or should I create multiple producers? When should I use more than a single producer?

Can someone please help,

Jul 10, 2018 in Apache Kafka by Meci Matt
• 9,400 points
869 views
Hello. And Bye.
Hello! But why Bye?

We wish to walk a long way with you, please come back!

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

In general, a single producer for all topics will be more network efficient.

If the Kafka client sees more than one topic+partition on the same Kafka Node, it can send messages for both topic+partitions in a single message. Kafka optimizes for message batches so this is efficient.

In addition, your web servers only need to maintain at-most one tcp connection to each Kafka node, instead of one connection per producer, per node.

For more info on Kafka's design: https://kafka.apache.org/documentation.html#design

Hope this will help you to some extent.

answered Jul 10, 2018 by nitinrawat895
• 9,030 points

Related Questions In Apache Kafka

0 votes
1 answer

Explain to me the functionality of Kafka in a Big-Data Cluster

Let me explain to you about Apache ...READ MORE

answered Apr 30 in Apache Kafka by ravikiran
• 1,460 points
16 views
0 votes
1 answer

How to commit message offsets in Kafka for reliable data pipeline?

You can use auto.commit.enable to allow Kafka ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 12,230 points
165 views
0 votes
0 answers

How to check pending messages in KAFKA topic?

Let say we have first_topic which has kafka_server_brokertopicmetrics_messagesin_total{instance="localhost:1120",job="kafka",topic="first_topic"}  ...READ MORE

May 2 in Apache Kafka by anonymous
37 views
0 votes
10 answers

Writing the Kafka consumer output to a file

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, ...READ MORE

answered Dec 7, 2018 in Apache Kafka by Harsh
3,319 views
0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 12,230 points
1,208 views
0 votes
11 answers

How can I delete all the messages from a Kafka topic?

To purge the Kafka topic, you need ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 12,230 points
15,897 views
0 votes
1 answer

Is Kafka and Zookeeper are required in a Big Data Cluster?

Apache Kafka is one of the components ...READ MORE

answered Mar 22, 2018 in Big Data Hadoop by nitinrawat895
• 9,030 points
253 views
0 votes
1 answer

Is there any change in consumer offsets if a new partition(s) is added to a Kafka topic?

Yes, it stays the same. An offset is ...READ MORE

answered Jul 9, 2018 in Apache Kafka by nitinrawat895
• 9,030 points
140 views
0 votes
1 answer

How to delete a topic in Kafka 0.8.1.1?

Deleting topic isn't always working in 0.8.1.1 Deletion ...READ MORE

answered Sep 4, 2018 in Apache Kafka by nitinrawat895
• 9,030 points
69 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.