Kafka consumer not able to start reading messages from the last uncommitted offsets

+1 vote
I have a 3-node Kafka cluster setup. I am using Apache spark (consumer) to read messages from Kafka broker. Each topic has 6 partitions. everything was working fine. when logs are coming from Apache Nifi to Kafka queue, spark consumer can read the messages in offsets smoothly, but in case of consumer crash, the spark consumer will not be able to read the remaining messages from Kafka.

I am reading latest offsets from Kafka brokers to my consumer.

I am using Kafka 2.11
Feb 1 in Apache Kafka by Prashant
• 130 points
879 views

1 answer to this question.

0 votes

Hi@prasant,

I am not sure it will work or not. But you can increase your receive buffer size in the server.properties file.

Thank You

answered Feb 18 by MD
• 22,520 points

Related Questions In Apache Kafka

0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,380 points
5,807 views
0 votes
2 answers

Not able to start Kafka service

If your service, in this case kafka, ...READ MORE

answered May 9, 2019 in Apache Kafka by Novice
3,785 views
0 votes
1 answer

Is there any change in consumer offsets if a new partition(s) is added to a Kafka topic?

Yes, it stays the same. An offset is ...READ MORE

answered Jul 9, 2018 in Apache Kafka by nitinrawat895
• 10,920 points
574 views
+1 vote
12 answers

How can I delete all the messages from a Kafka topic?

As of kafka 2.3.0 version, there is ...READ MORE

answered Sep 8, 2019 in Apache Kafka by Brajkishore
• 240 points
87,043 views
0 votes
1 answer

Not able to fetch messages from Kafka cluster

You need to add the hostname in ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,380 points
958 views
0 votes
1 answer

How to deploy kafka cluster on cloud?

You can run kafka in both ways ...READ MORE

answered Jun 14, 2018 in DevOps Tools by DareDev
• 6,850 points
95 views
0 votes
1 answer

Re-balancing error while reading messages from Kafka.

rebalance.backoff.ms defines the time for which Kafka ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,380 points
879 views
+1 vote
10 answers

Writing the Kafka consumer output to a file

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, ...READ MORE

answered Dec 7, 2018 in Apache Kafka by Harsh
12,840 views
0 votes
1 answer

[Mac] Writing the Kafka consumer output to a CSV file

Hi@ I don't know it will work or ...READ MORE

answered Feb 18 in Apache Kafka by MD
• 22,520 points
431 views
0 votes
1 answer

How to create Kafka consumer API?

Hi@akhtar, If you know the kafka producer API ...READ MORE

answered Feb 11 in Apache Kafka by MD
• 22,520 points
124 views