Writing the Kafka consumer output to a file

+1 vote

I want to write the messages which I am consuming using console consumer to a text file which I can keep for the future reference. Can anyone tell me how to do that?

Jul 9, 2018 in Apache Kafka by coldcode
• 2,050 points
12,964 views

10 answers to this question.

+2 votes

Use the pipe operator when you are running the console consumer. It will log all the messages which are getting consumed, to a file.

kafka-console-producer.sh --broker-list localhost:9092 --topic Topic < abc.txt
answered Jul 9, 2018 by Shubham
• 13,380 points

Isn't that backward?  The question is about outputting consumer messages to a text file. 

kafka-console-producer.sh --broker-list localhost:9092 --topic Topic < abc.txt

That line is a producer loading the events from a file.

This is great post. This comment is nice and excellent. 
+1 vote

You can use a simple > for this:

kafka-console-consumer > file.txt
answered Dec 7, 2018 by Sachin
+2 votes

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, "UTF-8"));
outputmessage += new String(bytes, "UTF-8")+"\n";

writeDataToFile(outputmessage);

answered Dec 7, 2018 by Harsh
Can you plz ping me the full code , Im in learning of kafka
+1 vote
$KAFKA_HOME/bin/connect-standalone.sh my-standalone.properties my-file.properties
answered Dec 7, 2018 by Ilegar
+1 vote
tail -n0 -F my_file.txt | kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic
answered Dec 7, 2018 by anonymous
+1 vote
echo "message goes here" | kafka-console-consumer.sh --broker-list localhost:9092 --topic my_topic
answered Dec 7, 2018 by Raga
+1 vote
answered Dec 7, 2018 by Aslam
+1 vote

You can create topic and then cat it to output file:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 
-partitions 1 --topic topic-name
answered Dec 7, 2018 by Logita
+1 vote
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Multibrokerapplication > output.txt
answered Dec 7, 2018 by Samanth
+1 vote
answered Dec 7, 2018 by Renku

Related Questions In Apache Kafka

0 votes
1 answer

[Mac] Writing the Kafka consumer output to a CSV file

Hi@ I don't know it will work or ...READ MORE

answered Feb 18 in Apache Kafka by MD
• 23,580 points
452 views
0 votes
1 answer

Is there any change in consumer offsets if a new partition(s) is added to a Kafka topic?

Yes, it stays the same. An offset is ...READ MORE

answered Jul 9, 2018 in Apache Kafka by nitinrawat895
• 10,920 points
576 views
0 votes
1 answer

Explain to me the functionality of Kafka in a Big-Data Cluster

Let me explain to you about Apache ...READ MORE

answered Apr 30, 2019 in Apache Kafka by ravikiran
• 4,600 points
236 views
+1 vote
1 answer

Kafka consumer not able to start reading messages from the last uncommitted offsets

Hi@prasant, I am not sure it will work ...READ MORE

answered Feb 18 in Apache Kafka by MD
• 23,580 points
914 views
0 votes
1 answer

How can I create multiple consumers in apache kafka?

Hi@akhtar, If you already created multiple producers then ...READ MORE

answered Feb 6 in Apache Kafka by MD
• 23,580 points
374 views
0 votes
1 answer

How to know leader information in kafka?

Hi@akhtar, To know leader information you can use ...READ MORE

answered Feb 10 in Apache Kafka by MD
• 23,580 points
312 views
0 votes
1 answer

What is the need of serializer in Kafka Producer API?

Hi@akhtar, If we try to use String key ...READ MORE

answered Feb 11 in Apache Kafka by MD
• 23,580 points
110 views
0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,380 points
5,899 views
+1 vote
12 answers

How can I delete all the messages from a Kafka topic?

As of kafka 2.3.0 version, there is ...READ MORE

answered Sep 8, 2019 in Apache Kafka by Brajkishore
• 240 points
88,113 views