Writing the Kafka consumer output to a file

+2 votes

I want to write the messages which I am consuming using console consumer to a text file which I can keep for the future reference. Can anyone tell me how to do that?

Jul 10, 2018 in Apache Kafka by coldcode
• 2,090 points
38,289 views

10 answers to this question.

+2 votes

Use the pipe operator when you are running the console consumer. It will log all the messages which are getting consumed, to a file.

kafka-console-producer.sh --broker-list localhost:9092 --topic Topic < abc.txt


Hope this helps!

To know more about Kafka, I would recommend you to enroll with Kafka training online today.

Thanks.

answered Jul 10, 2018 by Shubham
• 13,490 points

Isn't that backward?  The question is about outputting consumer messages to a text file. 

kafka-console-producer.sh --broker-list localhost:9092 --topic Topic < abc.txt

That line is a producer loading the events from a file.

This is great post. This comment is nice and excellent. 
+1 vote

You can use a simple > for this:

kafka-console-consumer > file.txt
answered Dec 7, 2018 by Sachin
+2 votes

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, "UTF-8"));
outputmessage += new String(bytes, "UTF-8")+"\n";

writeDataToFile(outputmessage);

answered Dec 7, 2018 by Harsh
Can you plz ping me the full code , Im in learning of kafka
+1 vote
$KAFKA_HOME/bin/connect-standalone.sh my-standalone.properties my-file.properties
answered Dec 7, 2018 by Ilegar
+1 vote
tail -n0 -F my_file.txt | kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic
answered Dec 7, 2018 by anonymous
+1 vote
echo "message goes here" | kafka-console-consumer.sh --broker-list localhost:9092 --topic my_topic
answered Dec 7, 2018 by Raga
+1 vote
answered Dec 7, 2018 by Aslam
+1 vote

You can create topic and then cat it to output file:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 
-partitions 1 --topic topic-name
answered Dec 7, 2018 by Logita
+1 vote
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Multibrokerapplication > output.txt
answered Dec 7, 2018 by Samanth
+1 vote
answered Dec 7, 2018 by Renku

Related Questions In Apache Kafka

0 votes
1 answer

[Mac] Writing the Kafka consumer output to a CSV file

Hi@ I don't know it will work or ...READ MORE

answered Feb 18, 2020 in Apache Kafka by MD
• 95,460 points
4,046 views
0 votes
1 answer

Is there any change in consumer offsets if a new partition(s) is added to a Kafka topic?

Yes, it stays the same. An offset is ...READ MORE

answered Jul 9, 2018 in Apache Kafka by nitinrawat895
• 11,380 points
2,338 views
0 votes
1 answer

Explain to me the functionality of Kafka in a Big-Data Cluster

Let me explain to you about Apache ...READ MORE

answered Apr 30, 2019 in Apache Kafka by ravikiran
• 4,620 points
1,056 views
+1 vote
1 answer

Kafka consumer not able to start reading messages from the last uncommitted offsets

Hi@prasant, I am not sure it will work ...READ MORE

answered Feb 18, 2020 in Apache Kafka by MD
• 95,460 points
9,614 views
0 votes
2 answers

How can I create multiple consumers in apache kafka?

First of all, Consumer is not thread ...READ MORE

answered Jun 18, 2020 in Apache Kafka by PAWAN
• 380 points
3,485 views
0 votes
1 answer

How to know leader information in kafka?

Hi@akhtar, To know leader information you can use ...READ MORE

answered Feb 10, 2020 in Apache Kafka by MD
• 95,460 points
16,165 views
0 votes
1 answer

What is the need of serializer in Kafka Producer API?

Hi@akhtar, If we try to use String key ...READ MORE

answered Feb 11, 2020 in Apache Kafka by MD
• 95,460 points
1,150 views
0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 10, 2018 in Apache Kafka by Shubham
• 13,490 points
13,963 views
+1 vote
14 answers

How can I delete all the messages from a Kafka topic?

To delete all the messages from a Kafka topic.  There are many approach ...READ MORE

answered Jun 19, 2020 in Apache Kafka by PAWAN
• 380 points
228,879 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP