How can I delete all the messages from a Kafka topic?

+1 vote

Announcement! Career Guide 2019 is out now. Explore careers to become a Big Data Developer or Architect!

I created a Kafka topic and pushed large messages to that kafka topic. Now I am getting error saying:

kafka.common.InvalidMessageSizeException: invalid message size

How I want to purge the topic so, that I can set the fetch.size for the topic ?

Jul 9, 2018 in Apache Kafka by coldcode
• 2,050 points
121,463 views

13 answers to this question.

+1 vote

To purge the Kafka topic, you need to change the retention time of that topic. The default retention time is 168 hours, i.e. 7 days. So, you have to change the retention time to 1 second, after which the messages from the topic will be deleted. Then, you can go ahead and change the retention time of the topic back to 168 hours.

To change the retention of the topic:

kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic1 --config retention.ms=1000
answered Jul 9, 2018 by Shubham
• 13,450 points
0 votes

To delete manually:

  1. Shutdown the cluster
  2. Clean kafka log dir (specified by the log.dir attribute in kafka config file ) as well the zookeeper data
  3. Restart the cluster
answered Nov 27, 2018 by Kailash
Want to delete consumed topic programatically
Not a good way

How we know that which topic consumed which not and only consumed should be deleted ?

is there any way to delete topics after consume?
0 votes

Follow these steps:

  1. Stop kafka
  2. Clean kafka log specific to partition, kafka stores its log file in a format of "logDir/topic-partition" so for a topic named "MyTopic" the log for partition id 0 will be stored in /tmp/kafka-logs/MyTopic-0 where /tmp/kafka-logs is specified by the log.dirattribute
  3. Restart kafka
answered Nov 27, 2018 by Kesh
+1 vote

Add one line to server.properties file under config folder:

delete.topic.enable=true

then, run this command:

bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
answered Nov 27, 2018 by Lavanya
0 votes

This script will empty a topic:

#!/bin/bash
echo "Enter name of topic to empty:"
read topicName
/$Kafka_Home/bin/kafka-configs --zookeeper localhost:2181 --alter --entity-type topics --entity-name $topicName --add config retention.ms=1000
sleep 5
/$Kafka_Home/bin/kafka-configs --zookeeper localhost:2181 --alter --entity-type topics --entity-name $topicName --delete-config retention.ms
answered Nov 27, 2018 by Ninja
0 votes

Try this:

  • stop kafka
  • delete the files rm -rf /tmp/kafka-logs/newTopic-*
answered Nov 27, 2018 by Alison
0 votes

One of the possible reason for this error could be your consumer's fetch size is smaller than the largest message it is trying to consume. Try increasing your fetch.size to make it larger than the max.message.size on your producers

answered Nov 27, 2018 by Bilal
0 votes

Adding to @Bilal's answer, there could be another possible reason for this error, the log on the server could be corrupted. To verify this, try running the
DumpLogSegments tool on the log segments for that topic/partition and see if it reports a problem

answered Nov 27, 2018 by Haider
0 votes
sh kafka-run-class.sh kafka.admin.DeleteTopicCommand --topic yourtopic --zookeeper localhost:2181
answered Dec 10, 2018 by Bhargav
0 votes

What I did was change the per-topic run-tine retention:

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my_topic --config retention.bytes=1
answered Dec 10, 2018 by Lalla

Follow these steps:

  1. Stop kafka
  2. Clean kafka log specific to partition, kafka stores its log file in a format of "logDir/topic-partition" so for a topic named "MyTopic" the log for partition id 0 will be stored in /tmp/kafka-logs/MyTopic-0 where /tmp/kafka-logs is specified by the log.dirattribute
  3. Restart kafka
0 votes
I dont know.
answered Jan 30, 2019 by anonymous
+1 vote

As of kafka 2.3.0 version, there is an alternate way to soft deletion of Kafka (old approach are deprecated ).

Update retention.ms to 1 sec (1000ms) then set it again after a min, to default setting i.e 7 days (168 hours, 604,800,000 in ms )

Soft deletion:- (rentention.ms=1000) (using kafka-configs.sh).

bin/kafka-configs.sh --zookeeper 192.168.1.10:2181 --alter --entity-name kafka_topic3p3r --entity-type topics  --add-config retention.ms=1000
Completed Updating config for entity: topic 'kafka_topic3p3r'.


Setting to default:- 7 days (168 hours , retention.ms= 604800000)

bin/kafka-configs.sh --zookeeper 192.168.1.10:2181 --alter --entity-name kafka_topic3p3r --entity-type topics  --add-config retention.ms=604800000
answered Sep 8, 2019 by Brajkishore
• 220 points
+1 vote

To delete all the messages from a Kafka topic. 

There are many approach : - 

Approach 1: Using Retention Time ( We can also say Soft Deletion without interruption. ) By Default Retention Time is : 7 Days

bin/kafka-configs.sh --zookeeper localhost:2181 --alter --entity-name kafka_keyword --entity-type topics  --add-config retention.ms=1000.  [ Here Identified for 1 Sec only ] 1 second = .001 ms
Approach 2 : Soft Delete ( Just Delete Topic and create new one with same name)
bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic keyword
Approach 3: Hard Delete 
- Stop the server
- Clean kafka log dir (specified by the log.dir attribute in kafka config file ) as well the zookeeper data
- Restart the broker
I hope it is enough to get idea. 

answered Jun 19 by Pawan
• 380 points

Related Questions In Apache Kafka

0 votes
1 answer

How can I create a topic in apache kafka?

Hi@akhtar, To create kafka topic you can use ...READ MORE

answered Feb 6 in Apache Kafka by MD
• 67,420 points
138 views
0 votes
1 answer

how can we delete the topic in kafka in kafkatool application ?

Hi@sreeveena, I am not sure but there may be ...READ MORE

answered May 13 in Apache Kafka by MD
• 67,420 points
212 views
0 votes
1 answer

How to delete a topic in Kafka 0.8.1.1?

Deleting topic isn't always working in 0.8.1.1 Deletion ...READ MORE

answered Sep 4, 2018 in Apache Kafka by nitinrawat895
• 10,950 points
611 views
0 votes
0 answers

how to delete the topic in kafka

How to stop Kafka server from crashing ...READ MORE

May 13 in Apache Kafka by sreeveena
• 140 points
136 views
+1 vote
0 answers

Want to delete consumed topic programatically

directly delete kafka log folder and zeekeeper ...READ MORE

Jun 4, 2019 in Apache Kafka by sarvesh
334 views
0 votes
2 answers

How can I create multiple consumers in apache kafka?

First of all, Consumer is not thread ...READ MORE

answered Jun 17 in Apache Kafka by Pawan
• 380 points
1,250 views
0 votes
1 answer

How to know leader information in kafka?

Hi@akhtar, To know leader information you can use ...READ MORE

answered Feb 10 in Apache Kafka by MD
• 67,420 points
1,912 views
0 votes
1 answer

How to create Kafka consumer API?

Hi@akhtar, If you know the kafka producer API ...READ MORE

answered Feb 11 in Apache Kafka by MD
• 67,420 points
299 views
0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,450 points
8,531 views
0 votes
1 answer

Re-balancing error while reading messages from Kafka.

rebalance.backoff.ms defines the time for which Kafka ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,450 points
1,254 views