I need to copy data from one HDFS to another HDFS Can you help me do so

Question

I have a setup which includes two Hadoop file systems, HDFS-x and HDFS-y. Now, I need to copy a few tables from HDFS-x to HDFS-y. I wish to do it without having to do a move or migrate operation. It is possible to do so via any of the commands?

Answer 1

I understood your issue.

Let me help you with a tool that can do the job for you.

• DistCp is a tool named as a distributed copy which is used to copy data in between clusters. 
• This tool basically uses MapReduce in the background to do the job.
• This tool can also manage the jobs of Data Recovery, Error Handling, and Report generation.

Usage Method:

$ hadoop distcp <src> <dst>

Example:

$ hadoop distcp hdfs://loc1:8020/file-x hdfs://loc2:8020/file-y

Here file -x from loc1 HDFS is copied to file-y in loc2 of the second HDFS.

There are two versions of this software which are available, Namely:

• DistCp
• DistCp2