Writing the Kafka consumer output to a file

Question

I want to write the messages which I am consuming using console consumer to a text file which I can keep for the future reference. Can anyone tell me how to do that?

Answer 1

Use the pipe operator when you are running the console consumer. It will log all the messages which are getting consumed, to a file.

kafka-console-producer.sh --broker-list localhost:9092 --topic Topic < abc.txt

Comments

Isn't that backward?  The question is about outputting consumer messages to a text file. 

kafka-console-producer.sh --broker-list localhost:9092 --topic Topic < abc.txt

That line is a producer loading the events from a file.

Answer 2

You can use a simple > for this:

kafka-console-consumer > file.txt

Answer 3

System.out.println(String.valueOf(output.offset()) + ": " + new String(bytes, "UTF-8"));
outputmessage += new String(bytes, "UTF-8")+"\n";

writeDataToFile(outputmessage);

Comments

Can you plz ping me the full code , Im in learning of kafka

Answer 4

$KAFKA_HOME/bin/connect-standalone.sh my-standalone.properties my-file.properties

Answer 5

tail -n0 -F my_file.txt | kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic

Answer 6

echo "message goes here" | kafka-console-consumer.sh --broker-list localhost:9092 --topic my_topic

Answer 7

You can refer this blog to do this: http://bigdatums.net/2017/06/22/writing-data-from-apache-kafka-to-text-file/

Answer 8

You can create topic and then cat it to output file:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 
-partitions 1 --topic topic-name

Answer 9

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Multibrokerapplication > output.txt

Answer 10

http://training.data-artisans.com/exercises/toFromKafka.html