Copy file from local to hdfs from the spark job in yarn mode

0 votes

How can I copy the file from local to hdfs from the spark job in yarn mode? Means, hdfs dfs -put command equivalent for spark. Because I have a file in local I need to preprocess it the need to put the file in hdfs and then apply the transformation logic. 

Jul 16 in Big Data Hadoop by Kriti
47 views

1 answer to this question.

0 votes

Please refer to the below code:

import org.apache.hadoop.conf.Configuration

import org.apache.hadoop.fs.FileSystem

import org.apache.hadoop.fs.Path

val hadoopConf = new Configuration()

val hdfs = FileSystem.get(hadoopConf)


val srcPath = new Path("/home/edureka/Documents/data")

val destPath = new Path("hdfs:///tranferrred_data")


hdfs.copyFromLocalFile(srcPath, destPath)

Any Spark Job that you are executing, you might want to include the above code snippet according to your requirement use spark-submit to deploy your code in the cluster. Every time you deploy your spark application, the data in your local gets transferred to the hdfs and then you can perform your transformations accordingly.

You might use the below dependencies:

libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.6.0"

libraryDependencies += "org.apache.commons" % "commons-io" % "1.3.2"

libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.6.0"

answered Jul 16 by Raj

Related Questions In Big Data Hadoop

0 votes
1 answer

Copy file from HDFS to the local file system

There are two possible ways to copy ...READ MORE

answered Mar 27, 2018 in Big Data Hadoop by nitinrawat895
• 10,490 points
4,194 views
0 votes
1 answer

Error while copying the file from local to HDFS

Well, the reason you are getting such ...READ MORE

answered May 2, 2018 in Big Data Hadoop by Ashish
• 2,630 points
397 views
0 votes
1 answer
0 votes
1 answer

How to print the content of a file in console present in HDFS?

Yes, you can use hdfs dfs command ...READ MORE

answered Apr 19, 2018 in Big Data Hadoop by Shubham
• 13,290 points
419 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,490 points
2,341 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,490 points
238 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
12,016 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
885 views
0 votes
2 answers

hadoop copy a local file system folder to HDFS

There's a typo in your command: "hadopp". ...READ MORE

answered Feb 4 in Big Data Hadoop by Lohith
2,650 views
0 votes
1 answer

How to check the size of a file in Hadoop HDFS?

You can use the  hadoop fs -ls command to ...READ MORE

answered Nov 21, 2018 in Big Data Hadoop by Omkar
• 67,290 points
655 views