Kafka consumer not able to start reading messages from the last uncommitted offsets

+1 vote
I have a 3-node Kafka cluster setup. I am using Apache spark (consumer) to read messages from Kafka broker. Each topic has 6 partitions. everything was working fine. when logs are coming from Apache Nifi to Kafka queue, spark consumer can read the messages in offsets smoothly, but in case of consumer crash, the spark consumer will not be able to read the remaining messages from Kafka.

I am reading latest offsets from Kafka brokers to my consumer.

I am using Kafka 2.11
Feb 1 in Apache Kafka by Prashant
• 130 points
50 views

No answer to this question. Be the first to respond.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.

Related Questions In Apache Kafka

0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,370 points
3,770 views
0 votes
2 answers

Not able to start Kafka service

If your service, in this case kafka, ...READ MORE

answered May 9, 2019 in Apache Kafka by Novice
2,136 views
0 votes
1 answer

Is there any change in consumer offsets if a new partition(s) is added to a Kafka topic?

Yes, it stays the same. An offset is ...READ MORE

answered Jul 9, 2018 in Apache Kafka by nitinrawat895
• 10,840 points
460 views
+1 vote
12 answers

How can I delete all the messages from a Kafka topic?

As of kafka 2.3.0 version, there is ...READ MORE

answered Sep 8, 2019 in Apache Kafka by Brajkishore
• 240 points
61,824 views
0 votes
1 answer

Not able to download Kafka using curl

The output to download this is in ...READ MORE

answered Feb 21, 2019 in Apache Kafka by Omkar
• 68,940 points
478 views
0 votes
0 answers

[Mac] Writing the Kafka consumer output to a CSV file

I'm very new to kafka, and so ...READ MORE

5 days ago in Apache Kafka by anonymous
• 120 points
20 views
0 votes
1 answer

Not able to fetch messages from Kafka cluster

You need to add the hostname in ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,370 points
759 views
0 votes
1 answer

How to deploy kafka cluster on cloud?

You can run kafka in both ways ...READ MORE

answered Jun 14, 2018 in DevOps Tools by DareDev
• 6,850 points
77 views
0 votes
1 answer

Re-balancing error while reading messages from Kafka.

rebalance.backoff.ms defines the time for which Kafka ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,370 points
689 views
+1 vote
10 answers

Writing the Kafka consumer output to a file

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, ...READ MORE

answered Dec 7, 2018 in Apache Kafka by Harsh
10,018 views