What is the optimum number of producers to create in Kafka?

0 votes

In a high volume real time java web app I'm sending messages to apache kafka. As of now, I've created single topic and I'm sending to that single topic, but there might be a case when I need to send messages to multiple topics.

In this case I'm not sure weather to create a producer per topic or should I use a single producer to all my topics?

Here is my code:

props = new Properties();
props.put("zk.connect", <zk-ip1>:<2181>,<zk-ip3>:<2181>,<zk-ip3>:<2181>);
props.put("zk.connectiontimeout.ms", "100000");
props.put("producer.type", "async");

Producer<String, Message> producer = new kafka.javaapi.producer.Producer<String, Message>(new ProducerConfig(props));

ProducerData<String, Message> producerData1 = new ProducerData<String, Message>("DemoTopic1", messageTosend);
ProducerData<String, Message> producerData2 = new ProducerData<String, Message>("DemoTopic2", messageTosend);

producer.send(producerData1);
producer.send(producerData2);


If my app sends to multiple topics (each topic gets different data) should I use a single producer or should I create multiple producers? When should I use more than a single producer?

Can someone please help,

Jul 10, 2018 in Apache Kafka by Meci Matt
• 9,460 points
3,920 views
Hello. And Bye.
Hello! But why Bye?

We wish to walk a long way with you, please come back!

1 answer to this question.

0 votes

In general, a single producer for all topics will be more network efficient.

If the Kafka client sees more than one topic+partition on the same Kafka Node, it can send messages for both topic+partitions in a single message. Kafka optimizes for message batches so this is efficient.

In addition, your web servers only need to maintain at-most one tcp connection to each Kafka node, instead of one connection per producer, per node.

For more info on Kafka's design: https://kafka.apache.org/documentation.html#design

Hope this will help you to some extent.

answered Jul 10, 2018 by nitinrawat895
• 10,950 points

Related Questions In Apache Kafka

+1 vote
2 answers

What is the need of zookeeper in kafka? Can I use kafka without zookeeper?

At present, There is no any alternative ...READ MORE

answered Jun 17 in Apache Kafka by Pawan
• 380 points
3,380 views
0 votes
1 answer

What is the need of serializer in Kafka Producer API?

Hi@akhtar, If we try to use String key ...READ MORE

answered Feb 11 in Apache Kafka by MD
• 79,930 points
177 views
0 votes
1 answer

What is the use of acts configuration in Kafka Producer API?

Hi@akhtar, The acts configuration is used to configure ...READ MORE

answered Feb 11 in Apache Kafka by MD
• 79,930 points
182 views
0 votes
1 answer

What is the meaning of retries in Kafka Producer API?

Hi@akhtar, It defines how many time the producer ...READ MORE

answered Feb 11 in Apache Kafka by MD
• 79,930 points
1,177 views
+1 vote
10 answers

Writing the Kafka consumer output to a file

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, ...READ MORE

answered Dec 7, 2018 in Apache Kafka by Harsh
17,548 views
0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,450 points
9,141 views
+1 vote
13 answers

How can I delete all the messages from a Kafka topic?

To delete all the messages from a Kafka topic.  There are many approach ...READ MORE

answered Jun 19 in Apache Kafka by Pawan
• 380 points
128,517 views
+1 vote
0 answers

Want to delete consumed topic programatically

directly delete kafka log folder and zeekeeper ...READ MORE

Jun 4, 2019 in Apache Kafka by sarvesh
351 views
0 votes
1 answer

Is there any change in consumer offsets if a new partition(s) is added to a Kafka topic?

Yes, it stays the same. An offset is ...READ MORE

answered Jul 9, 2018 in Apache Kafka by nitinrawat895
• 10,950 points
789 views
0 votes
1 answer

How to delete a topic in Kafka 0.8.1.1?

Deleting topic isn't always working in 0.8.1.1 Deletion ...READ MORE

answered Sep 4, 2018 in Apache Kafka by nitinrawat895
• 10,950 points
646 views