Kafka consumer not able to start reading messages from the last uncommitted offsets

+1 vote
I have a 3-node Kafka cluster setup. I am using Apache spark (consumer) to read messages from Kafka broker. Each topic has 6 partitions. everything was working fine. when logs are coming from Apache Nifi to Kafka queue, spark consumer can read the messages in offsets smoothly, but in case of consumer crash, the spark consumer will not be able to read the remaining messages from Kafka.

I am reading latest offsets from Kafka brokers to my consumer.

I am using Kafka 2.11
Feb 1, 2020 in Apache Kafka by Prashant
• 130 points
7,514 views

1 answer to this question.

0 votes

Hi@prasant,

I am not sure it will work or not. But you can increase your receive buffer size in the server.properties file.

Hope this works for you!

Get your Kafka certification today to become certified professional

Thank You

answered Feb 18, 2020 by MD
• 95,360 points

Related Questions In Apache Kafka

0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 10, 2018 in Apache Kafka by Shubham
• 13,490 points
12,589 views
0 votes
2 answers

Not able to start Kafka service

If your service, in this case kafka, ...READ MORE

answered May 10, 2019 in Apache Kafka by Novice
12,639 views
0 votes
1 answer

Is there any change in consumer offsets if a new partition(s) is added to a Kafka topic?

Yes, it stays the same. An offset is ...READ MORE

answered Jul 9, 2018 in Apache Kafka by nitinrawat895
• 11,380 points
1,289 views
0 votes
1 answer

Not able to download Kafka using curl

The output to download this is in ...READ MORE

answered Feb 21, 2019 in Apache Kafka by Omkar
• 69,210 points
1,657 views
0 votes
1 answer

Not able to fetch messages from Kafka cluster

You need to add the hostname in ...READ MORE

answered Jul 10, 2018 in Apache Kafka by Shubham
• 13,490 points
2,090 views
0 votes
1 answer

How to deploy kafka cluster on cloud?

You can run kafka in both ways ...READ MORE

answered Jun 14, 2018 in DevOps Tools by DareDev
• 6,890 points
279 views
0 votes
1 answer

Re-balancing error while reading messages from Kafka.

rebalance.backoff.ms defines the time for which Kafka ...READ MORE

answered Jul 10, 2018 in Apache Kafka by Shubham
• 13,490 points
2,024 views
+2 votes
10 answers

Writing the Kafka consumer output to a file

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, ...READ MORE

answered Dec 7, 2018 in Apache Kafka by Harsh
28,422 views
0 votes
14 answers

How can I delete all the messages from a Kafka topic?

To delete all the messages from a Kafka topic.  There are many approach ...READ MORE

answered Jun 19, 2020 in Apache Kafka by Pawan
• 380 points
211,869 views
0 votes
1 answer

[Mac] Writing the Kafka consumer output to a CSV file

Hi@ I don't know it will work or ...READ MORE

answered Feb 18, 2020 in Apache Kafka by MD
• 95,360 points
2,694 views
webinar REGISTER FOR FREE WEBINAR X
Send OTP
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP