How can I put file to HDFS directly without copying it local disk

Question

There is a dataset present on web location which is around 31 GB and has been compressed in .gz format. I have a wordcount program which I want to execute over it. I have a remote Hadoop cluster and I am connecting to it using ssh.

The main problem is my home directory cannot hold this dataset on the remote machine due to disk usage quota. So , I was wondering if there is a way to wget the dataset to my HDFS directory Can anyone help me out ?

Answer 1

Can use pipe from wget to hdfs.

You might face problem as gz files are not splittable, this will stop you from running distributed MapReduce code over it.

I would suggest to download file in a local system, then unzip the file and then use pipe operator.

cat test123.txt | ssh uname@master "hadoop dfs -put - FolderName/test123.txt"