What is the optimum number of producers to create in Kafka

0 votes

In a high volume real time java web app I'm sending messages to apache kafka. As of now, I've created single topic and I'm sending to that single topic, but there might be a case when I need to send messages to multiple topics.

In this case I'm not sure weather to create a producer per topic or should I use a single producer to all my topics?

Here is my code:

props = new Properties();
props.put("zk.connect", <zk-ip1>:<2181>,<zk-ip3>:<2181>,<zk-ip3>:<2181>);
props.put("zk.connectiontimeout.ms", "100000");
props.put("producer.type", "async");

Producer<String, Message> producer = new kafka.javaapi.producer.Producer<String, Message>(new ProducerConfig(props));

ProducerData<String, Message> producerData1 = new ProducerData<String, Message>("DemoTopic1", messageTosend);
ProducerData<String, Message> producerData2 = new ProducerData<String, Message>("DemoTopic2", messageTosend);

producer.send(producerData1);
producer.send(producerData2);


If my app sends to multiple topics (each topic gets different data) should I use a single producer or should I create multiple producers? When should I use more than a single producer?

Can someone please help,

Jul 10, 2018 in Apache Kafka by Meci Matt
• 9,460 points
7,934 views
Hello. And Bye.
Hello! But why Bye?

We wish to walk a long way with you, please come back!

1 answer to this question.

0 votes

In general, a single producer for all topics will be more network efficient.

If the Kafka client sees more than one topic+partition on the same Kafka Node, it can send messages for both topic+partitions in a single message. Kafka optimizes for message batches so this is efficient.

In addition, your web servers only need to maintain at-most one tcp connection to each Kafka node, instead of one connection per producer, per node.

For more info on Kafka's design: https://kafka.apache.org/documentation.html#design

Hope this will help you to some extent.

To know more about Kafka, get your Apache Kafka certification course today.

Thank You

answered Jul 10, 2018 by nitinrawat895
• 11,380 points

Related Questions In Apache Kafka

+1 vote
3 answers

What is the need of zookeeper in kafka? Can I use kafka without zookeeper?

I just would like to complete the ...READ MORE

answered Dec 4, 2020 in Apache Kafka by Tamás
• 140 points
8,201 views
0 votes
1 answer

What is the need of serializer in Kafka Producer API?

Hi@akhtar, If we try to use String key ...READ MORE

answered Feb 11, 2020 in Apache Kafka by MD
• 95,440 points
852 views
0 votes
1 answer

What is the use of acts configuration in Kafka Producer API?

Hi@akhtar, The acts configuration is used to configure ...READ MORE

answered Feb 11, 2020 in Apache Kafka by MD
• 95,440 points
1,256 views
0 votes
1 answer

What is the meaning of retries in Kafka Producer API?

Hi@akhtar, It defines how many time the producer ...READ MORE

answered Feb 11, 2020 in Apache Kafka by MD
• 95,440 points
4,073 views
+2 votes
10 answers

Writing the Kafka consumer output to a file

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, ...READ MORE

answered Dec 7, 2018 in Apache Kafka by Harsh
36,656 views
0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 10, 2018 in Apache Kafka by Shubham
• 13,490 points
13,676 views
0 votes
14 answers

How can I delete all the messages from a Kafka topic?

To delete all the messages from a Kafka topic.  There are many approach ...READ MORE

answered Jun 19, 2020 in Apache Kafka by Pawan
• 380 points
226,271 views
+1 vote
0 answers

Want to delete consumed topic programatically

directly delete kafka log folder and zeekeeper ...READ MORE

Jun 4, 2019 in Apache Kafka by sarvesh
1,390 views
0 votes
1 answer

Is there any change in consumer offsets if a new partition(s) is added to a Kafka topic?

Yes, it stays the same. An offset is ...READ MORE

answered Jul 9, 2018 in Apache Kafka by nitinrawat895
• 11,380 points
2,087 views
0 votes
1 answer

How to delete a topic in Kafka 0.8.1.1?

Deleting topic isn't always working in 0.8.1.1 Deletion ...READ MORE

answered Sep 4, 2018 in Apache Kafka by nitinrawat895
• 11,380 points
1,524 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP