How to send curl downloaded data in hdfs

Question

I am using this code:

curl -o filename.txt url | xargs hdfs dfs -moveFromLocal $1 /path/to/dir/

When I run this code, the values from request are saved inside fileName.txt and then that file is moved to HDFS. I wanna know if I can, mantain the curl output in memory?

Answer 1

Try using put command with stdin as input. Something like this:

curl -sS https://www.google.com/robots.txt | hdfs dfs -put - /robots.txt