Kafka consumer not able to start reading messages from the last uncommitted offsets

+1 vote
I have a 3-node Kafka cluster setup. I am using Apache spark (consumer) to read messages from Kafka broker. Each topic has 6 partitions. everything was working fine. when logs are coming from Apache Nifi to Kafka queue, spark consumer can read the messages in offsets smoothly, but in case of consumer crash, the spark consumer will not be able to read the remaining messages from Kafka.

I am reading latest offsets from Kafka brokers to my consumer.

I am using Kafka 2.11
Feb 1, 2020 in Apache Kafka by Prashant
• 130 points
4,613 views

1 answer to this question.

0 votes

Hi@prasant,

I am not sure it will work or not. But you can increase your receive buffer size in the server.properties file.

Thank You

answered Feb 18, 2020 by MD
• 95,060 points

Related Questions In Apache Kafka

0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,480 points
10,213 views
0 votes
2 answers

Not able to start Kafka service

If your service, in this case kafka, ...READ MORE

answered May 9, 2019 in Apache Kafka by Novice
7,225 views
0 votes
1 answer

Is there any change in consumer offsets if a new partition(s) is added to a Kafka topic?

Yes, it stays the same. An offset is ...READ MORE

answered Jul 9, 2018 in Apache Kafka by nitinrawat895
• 11,380 points
858 views
0 votes
1 answer

Not able to download Kafka using curl

The output to download this is in ...READ MORE

answered Feb 21, 2019 in Apache Kafka by Omkar
• 69,090 points
1,074 views
0 votes
1 answer

Not able to fetch messages from Kafka cluster

You need to add the hostname in ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,480 points
1,413 views
0 votes
1 answer

How to deploy kafka cluster on cloud?

You can run kafka in both ways ...READ MORE

answered Jun 14, 2018 in DevOps Tools by DareDev
• 6,890 points
172 views
0 votes
1 answer

Re-balancing error while reading messages from Kafka.

rebalance.backoff.ms defines the time for which Kafka ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,480 points
1,455 views
+1 vote
10 answers

Writing the Kafka consumer output to a file

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, ...READ MORE

answered Dec 7, 2018 in Apache Kafka by Harsh
19,492 views
+1 vote
14 answers

How can I delete all the messages from a Kafka topic?

To delete all the messages from a Kafka topic.  There are many approach ...READ MORE

answered Jun 19, 2020 in Apache Kafka by Pawan
• 380 points
144,216 views
0 votes
1 answer

[Mac] Writing the Kafka consumer output to a CSV file

Hi@ I don't know it will work or ...READ MORE

answered Feb 18, 2020 in Apache Kafka by MD
• 95,060 points
1,559 views