How to copy file to multiple hdfs directory

0 votes
I have a file in local system that I want to copy to many hdfs directories. I had to copy it 3 directories so I did it manually. I was just wondering, what if I had to copy the file into more directories, say 100. Is there a faster way to do this?
Jan 25, 2019 in Big Data Hadoop by Rusva
1,568 views

1 answer to this question.

0 votes

You can use loop to copy the file to multiple directories, I found a shell script which does the same:

m=10
for (( i = 0; i < 100; i++ )); do
   sh hdfs_cp_script partition$(($i*$m + 1)) & sh hdfs_cp_script partition$(($i*$m + 2) & ... & sh hdfs_cp_script partition$(($i*$m + 10))
done
answered Jan 25, 2019 by Omkar
• 69,150 points

Related Questions In Big Data Hadoop

+1 vote
1 answer

How to copy file from Local file system to HDFS?

Hi@akhtar, You can copy files from your local ...READ MORE

answered Oct 19, 2020 in Big Data Hadoop by MD
• 95,240 points
349 views
0 votes
1 answer
0 votes
1 answer

Copy file from HDFS to the local file system

There are two possible ways to copy ...READ MORE

answered Mar 27, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
13,722 views
0 votes
1 answer

How can we transfer a PDF file to HDFS?

You can easily upload any file to ...READ MORE

answered Apr 13, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
241 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
7,599 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,257 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
57,866 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
2,860 views
0 votes
1 answer

Hadoop: How to copy directory from local system to hdfs using Java code?

Just use the FileSystem's copyFromLocalFile method. If the source Path ...READ MORE

answered Nov 14, 2018 in Big Data Hadoop by Omkar
• 69,150 points
3,150 views
0 votes
1 answer

How to check the size of a file in Hadoop HDFS?

You can use the  hadoop fs -ls command to ...READ MORE

answered Nov 21, 2018 in Big Data Hadoop by Omkar
• 69,150 points
5,128 views