What is the optimum number of producers to create in Kafka?

0 votes

In a high volume real time java web app I'm sending messages to apache kafka. As of now, I've created single topic and I'm sending to that single topic, but there might be a case when I need to send messages to multiple topics.

In this case I'm not sure weather to create a producer per topic or should I use a single producer to all my topics?

Here is my code:

props = new Properties();
props.put("zk.connect", <zk-ip1>:<2181>,<zk-ip3>:<2181>,<zk-ip3>:<2181>);
props.put("zk.connectiontimeout.ms", "100000");
props.put("producer.type", "async");

Producer<String, Message> producer = new kafka.javaapi.producer.Producer<String, Message>(new ProducerConfig(props));

ProducerData<String, Message> producerData1 = new ProducerData<String, Message>("DemoTopic1", messageTosend);
ProducerData<String, Message> producerData2 = new ProducerData<String, Message>("DemoTopic2", messageTosend);

producer.send(producerData1);
producer.send(producerData2);


If my app sends to multiple topics (each topic gets different data) should I use a single producer or should I create multiple producers? When should I use more than a single producer?

Can someone please help,

Jul 10, 2018 in Apache Kafka by Meci Matt
• 9,420 points
1,369 views
Hello. And Bye.
Hello! But why Bye?

We wish to walk a long way with you, please come back!

1 answer to this question.

0 votes

In general, a single producer for all topics will be more network efficient.

If the Kafka client sees more than one topic+partition on the same Kafka Node, it can send messages for both topic+partitions in a single message. Kafka optimizes for message batches so this is efficient.

In addition, your web servers only need to maintain at-most one tcp connection to each Kafka node, instead of one connection per producer, per node.

For more info on Kafka's design: https://kafka.apache.org/documentation.html#design

Hope this will help you to some extent.

answered Jul 10, 2018 by nitinrawat895
• 10,670 points

Related Questions In Apache Kafka

0 votes
1 answer

Explain to me the functionality of Kafka in a Big-Data Cluster

Let me explain to you about Apache ...READ MORE

answered Apr 30 in Apache Kafka by ravikiran
• 4,560 points
66 views
0 votes
1 answer

How to commit message offsets in Kafka for reliable data pipeline?

You can use auto.commit.enable to allow Kafka ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,290 points
313 views
0 votes
0 answers

How to check pending messages in KAFKA topic?

Let say we have first_topic which has kafka_server_brokertopicmetrics_messagesin_total{instance="localhost:1120",job="kafka",topic="first_topic"}  ...READ MORE

May 2 in Apache Kafka by anonymous
449 views
0 votes
10 answers

Writing the Kafka consumer output to a file

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, ...READ MORE

answered Dec 7, 2018 in Apache Kafka by Harsh
6,001 views
0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,290 points
2,200 views
0 votes
12 answers

How can I delete all the messages from a Kafka topic?

Add one line to server.properties file under ...READ MORE

answered Nov 27, 2018 in Apache Kafka by Lavanya
31,818 views
0 votes
0 answers

Want to delete consumed topic programatically

directly delete kafka log folder and zeekeeper ...READ MORE

Jun 4 in Apache Kafka by sarvesh
80 views
0 votes
1 answer

Is there any change in consumer offsets if a new partition(s) is added to a Kafka topic?

Yes, it stays the same. An offset is ...READ MORE

answered Jul 9, 2018 in Apache Kafka by nitinrawat895
• 10,670 points
269 views
0 votes
1 answer

How to delete a topic in Kafka 0.8.1.1?

Deleting topic isn't always working in 0.8.1.1 Deletion ...READ MORE

answered Sep 4, 2018 in Apache Kafka by nitinrawat895
• 10,670 points
129 views