Re-balancing error while reading messages from Kafka.

0 votes

When I am trying to read messages from a Kafka topic, but the process gets killed with following error:

[2017-08-18 08:11:21,215] ERROR Error processing message, stopping consumer:  (kafka.consumer.ConsoleConsumer$)
kafka.common.ConsumerRebalanceFailedException: topic1 can't rebalance after 4 retries
    at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:428)
    at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:718)
    at kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:752)
    at kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:142)
    at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:196)
    at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
Consumed 0 messages

Can anyone help me with this rebalancing issue?

Jul 9, 2018 in Apache Kafka by coldcode
• 2,020 points
539 views

1 answer to this question.

0 votes

rebalance.backoff.ms defines the time for which Kafka will wait before rebalancing & zookeeper.session.timeout.ms parameter specifies the time for which Kafka will wait to connect to zookeeper. So, try adjusting these two parameters.

Hope this helps!

answered Jul 9, 2018 by Shubham
• 13,300 points

Related Questions In Apache Kafka

0 votes
12 answers

How can I delete all the messages from a Kafka topic?

Add one line to server.properties file under ...READ MORE

answered Nov 27, 2018 in Apache Kafka by Lavanya
41,882 views
0 votes
1 answer

error message while running

Ideally, installing the urllib3 module should solve this error. ...READ MORE

answered Feb 25 in Apache Kafka by Omkar
• 67,660 points
45 views
0 votes
0 answers

How to check pending messages in KAFKA topic?

Let say we have first_topic which has kafka_server_brokertopicmetrics_messagesin_total{instance="localhost:1120",job="kafka",topic="first_topic"}  ...READ MORE

May 2 in Apache Kafka by anonymous
662 views
0 votes
1 answer

Is there any change in consumer offsets if a new partition(s) is added to a Kafka topic?

Yes, it stays the same. An offset is ...READ MORE

answered Jul 9, 2018 in Apache Kafka by nitinrawat895
• 10,710 points
333 views
0 votes
10 answers

Writing the Kafka consumer output to a file

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, ...READ MORE

answered Dec 7, 2018 in Apache Kafka by Harsh
7,594 views
0 votes
1 answer

Is Kafka and Zookeeper are required in a Big Data Cluster?

Apache Kafka is one of the components ...READ MORE

answered Mar 22, 2018 in Big Data Hadoop by nitinrawat895
• 10,710 points
426 views
0 votes
1 answer

which one to choose for log analysis?

It is not about that you can ...READ MORE

answered Apr 6, 2018 in Big Data Hadoop by nitinrawat895
• 10,710 points
38 views
0 votes
1 answer

Getting error while connecting zookeeper in Kafka - Spark Streaming integration

I guess you need provide this kafka.bootstrap.servers ...READ MORE

answered May 24, 2018 in Apache Spark by Shubham
• 13,300 points
680 views
0 votes
1 answer

Not able to fetch messages from Kafka cluster

You need to add the hostname in ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,300 points
590 views
0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,300 points
2,664 views