How can I write a text file in HDFS not from an RDD in Spark program

Question

I am executing a command in Spark, where I am using saveAsTextFile to save my RDD. I am able to save the RDD in both my local filesystem as well as in HDFS present on my cluster. Both work fine.

Now I  also have to write some more additional files generated during processing, which I am writing to local filesystem. I want to send them to my HDFS cluster.

Can anyone help me out?

Answer 1

Yes, you can go ahead and write a text file into HDFS using Spark.

val filesystem = FileSystem.get(sparkContext.hadoopConfiguration);
val output_stream = filesystem.create(new Path(file));
val buffered_output = BufferedOutputStream(output_stream)
buffered_output.write("My Text".getBytes("UTF-8"))
buffered_output.close()

Important thing to note down is FSDataOutputStream, is object output stream not a text output stream.