Moving data to hdfs using copyFromLocal command

0 votes
I am trying to copy a simple file present in my local filesystem to the directory specified for hdfs. I don't know what I'm doing wrong?
I have specified in my hdfs-site.xml that the directory for hdfs will be /home/user1/Hadoop/dataNodeHadoopData using the following properties -
<name>dfs.data.dir</name>
<value>/home/user1/Hadoop/dataNodeHadoopData/</value>

and

<name>dfs.name.dir</name>
<value>/home/user1/Hadoop/dataNodeHadoopData/</value>
 

I am using the following command -
bin/hadoop dfs -copyFromLocal /home/user1/ml-100k/sample.data /home/user1/Hadoop/dataNodeHadoopData
to copy the file sample.data from it's local filesystem location to the HDFS directory that I specified. But when I run this command, I'm not able to copy the data from local filesystem to hdfs. Can anyone please help?
Mar 27, 2018 in Big Data Hadoop by Shubham
• 13,190 points
1,151 views

1 answer to this question.

0 votes

Actually dfs.data.dir and dfs.name.dir have to point to two different directories. 
Also make sure you have formatted the namenode FS after changing the directories in the configuration.
In your case while copying to HDFS you're incorrectly specifying the target. 

The correct syntax for copying a local file to HDFS is:
bin/hadoop dfs -copyFromLocal <local_FS_filename> <target_on_HDFS>

Correct syntax in your case
bin/hadoop dfs -copyFromLocal /home/user1/ml-100k/sample.data <hdfs directory>

answered Mar 27, 2018 by nitinrawat895
• 10,110 points

Hi 

I am trying to copy local file to hdfs with below command

hdfs dfs -copyFromLocal /Desktop/edureka/475_m3_county_facts.csv /user/edureka_372192/test_dir/country_facts.csv

getting error No such file or directory 

Please help me for this

Hey @Kishori.

Firstly, your command is wrong. The destination should be a path to a directory but in your command, the destination is a file:

hdfs dfs -copyFromLocal /Desktop/edureka/475_m3_county_facts.csv /user/edureka_372192/test_dir/country_facts.csv

 The right command should be:

hdfs dfs -copyFromLocal /Desktop/edureka/475_m3_county_facts.csv /user/edureka_372192/test_dir/

If you still get this error, then there are 3 possible problems:

  1. The file does not exist in the path mentioned
  2. The source path is wrong
  3. The destination path is wrong

Troubleshooting:

Open terminal and change the directory to the directory where the file is present. Now, run this command to check the present working directory

$pwd

The output you get should be the source path in your hadoop command.

Next, check if the destination path is right. 

Hope this helps..

Don't mention file name in destination

Related Questions In Big Data Hadoop

0 votes
1 answer

How to transfer data from Netezza to HDFS using Apache Sqoop?

Remove the --direct option. It gives issue ...READ MORE

answered Apr 23, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
232 views
0 votes
1 answer

How to securely transfer data from rdms to hdfs using sqoop?

Sqoop stores metadata in a repository and ...READ MORE

answered Dec 18, 2018 in Big Data Hadoop by Omkar
• 67,120 points
62 views
0 votes
5 answers
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
765 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,110 points
2,054 views
0 votes
10 answers

hadoop fs -put command?

copy command can be used to copy files ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Sujay
10,503 views
0 votes
1 answer
0 votes
1 answer

Is there any way to load data from MySql to HDFS?

The generic command i.e used to import ...READ MORE

answered Apr 10, 2018 in Big Data Hadoop by nitinrawat895
• 10,110 points
400 views
0 votes
1 answer

How to extract only few lines of data from HDFS?

Here also in case of Hadoop, it is ...READ MORE

answered May 2, 2018 in Big Data Hadoop by nitinrawat895
• 10,110 points
727 views