I have a 3-node Kafka cluster setup. I am using Apache spark (consumer) to read messages from Kafka broker. Each topic has 6 partitions. everything was working fine. when logs are coming from Apache Nifi to Kafka queue, spark consumer can read the messages in offsets smoothly, but in case of consumer crash, the spark consumer will not be able to read the remaining messages from Kafka.
I am reading latest offsets from Kafka brokers to my consumer.
I am using Kafka 2.11