How can I write a text file in HDFS not from an RDD, in Spark program?

0 votes
I am executing a command in Spark, where I am using saveAsTextFile to save my RDD. I am able to save the RDD in both my local filesystem as well as in HDFS present on my cluster. Both work fine.

Now I  also have to write some more additional files generated during processing, which I am writing to local filesystem. I want to send them to my HDFS cluster.

Can anyone help me out?
May 29, 2018 in Apache Spark by code799
710 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

Yes, you can go ahead and write a text file into HDFS using Spark.

val filesystem = FileSystem.get(sparkContext.hadoopConfiguration);
val output_stream = filesystem.create(new Path(file));
val buffered_output = BufferedOutputStream(output_stream)
buffered_output.write("My Text".getBytes("UTF-8"))
buffered_output.close()

Important thing to note down is FSDataOutputStream, is object output stream not a text output stream.

answered May 29, 2018 by Shubham
• 12,710 points

Related Questions In Apache Spark

0 votes
1 answer

How is RDD in Spark different from Distributed Storage Management? Can anyone help me with this ?

Some of the key differences between an RDD and ...READ MORE

answered Jul 26, 2018 in Apache Spark by zombie
• 3,690 points
81 views
0 votes
1 answer
0 votes
0 answers
0 votes
2 answers

How can I convert Spark Dataframe to Spark RDD?

Assuming your RDD[row] is called rdd, you ...READ MORE

answered Jul 9, 2018 in Apache Spark by zombie
• 3,690 points
307 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 9,310 points
1,818 views
0 votes
10 answers

hadoop fs -put command?

copy command can be used to copy files ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Sujay
8,988 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
646 views
+1 vote
1 answer
0 votes
1 answer

In a Spark DataFrame how can I flatten the struct?

You can go ahead and use the ...READ MORE

answered May 24, 2018 in Apache Spark by Shubham
• 12,710 points
403 views
0 votes
1 answer

How to save and retrieve the Spark RDD from HDFS?

You can save the RDD using saveAsObjectFile and saveAsTextFile method. ...READ MORE

answered May 29, 2018 in Apache Spark by Shubham
• 12,710 points
1,181 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.