Moving data to hdfs using copyFromLocal command

0 votes
I am trying to copy a simple file present in my local filesystem to the directory specified for hdfs. I don't know what I'm doing wrong?
I have specified in my hdfs-site.xml that the directory for hdfs will be /home/user1/Hadoop/dataNodeHadoopData using the following properties -
<name>dfs.data.dir</name>
<value>/home/user1/Hadoop/dataNodeHadoopData/</value>

and

<name>dfs.name.dir</name>
<value>/home/user1/Hadoop/dataNodeHadoopData/</value>
 

I am using the following command -
bin/hadoop dfs -copyFromLocal /home/user1/ml-100k/sample.data /home/user1/Hadoop/dataNodeHadoopData
to copy the file sample.data from it's local filesystem location to the HDFS directory that I specified. But when I run this command, I'm not able to copy the data from local filesystem to hdfs. Can anyone please help?
Mar 27, 2018 in Big Data Hadoop by Shubham
• 13,490 points
10,176 views

1 answer to this question.

0 votes

Actually dfs.data.dir and dfs.name.dir have to point to two different directories. 
Also make sure you have formatted the namenode FS after changing the directories in the configuration.
In your case while copying to HDFS you're incorrectly specifying the target. 

The correct syntax for copying a local file to HDFS is:
bin/hadoop dfs -copyFromLocal <local_FS_filename> <target_on_HDFS>

Correct syntax in your case
bin/hadoop dfs -copyFromLocal /home/user1/ml-100k/sample.data <hdfs directory>

answered Mar 27, 2018 by nitinrawat895
• 11,380 points

Hi 

I am trying to copy local file to hdfs with below command

hdfs dfs -copyFromLocal /Desktop/edureka/475_m3_county_facts.csv /user/edureka_372192/test_dir/country_facts.csv

getting error No such file or directory 

Please help me for this

Hey @Kishori.

Firstly, your command is wrong. The destination should be a path to a directory but in your command, the destination is a file:

hdfs dfs -copyFromLocal /Desktop/edureka/475_m3_county_facts.csv /user/edureka_372192/test_dir/country_facts.csv

 The right command should be:

hdfs dfs -copyFromLocal /Desktop/edureka/475_m3_county_facts.csv /user/edureka_372192/test_dir/

If you still get this error, then there are 3 possible problems:

  1. The file does not exist in the path mentioned
  2. The source path is wrong
  3. The destination path is wrong

Troubleshooting:

Open terminal and change the directory to the directory where the file is present. Now, run this command to check the present working directory

$pwd

The output you get should be the source path in your hadoop command.

Next, check if the destination path is right. 

Hope this helps..

Don't mention file name in destination

Related Questions In Big Data Hadoop

0 votes
1 answer

How to transfer data from Netezza to HDFS using Apache Sqoop?

Remove the --direct option. It gives issue ...READ MORE

answered Apr 23, 2018 in Big Data Hadoop by kurt_cobain
• 9,350 points
1,770 views
0 votes
1 answer

How to securely transfer data from rdms to hdfs using sqoop?

Sqoop stores metadata in a repository and ...READ MORE

answered Dec 18, 2018 in Big Data Hadoop by Omkar
• 69,220 points
1,395 views
0 votes
1 answer

Sqoop - moving data from DB2 to HDFS ( hive partition table)

sqoop import --driver com.ibm.db2.jcc.DB2Driver --connect jdbc:db2://db2.my.com:50000/databaseName --username ...READ MORE

answered Aug 9, 2019 in Big Data Hadoop by Payal
4,615 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,350 points
4,629 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
11,059 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
108,989 views
0 votes
1 answer
0 votes
1 answer

Is there any way to load data from MySql to HDFS?

The generic command i.e used to import ...READ MORE

answered Apr 10, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
3,663 views
0 votes
1 answer

How to extract only few lines of data from HDFS?

Here also in case of Hadoop, it is ...READ MORE

answered May 2, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
11,549 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP