Differentiate hadoop fs -text vs hadoop fs -cat vs hadoop fs -get

Question

As per my knowledge all the following commands can be used for copying the hdfs files to local file system. So what is the difference between below commands-

hadoop fs -text /hdfs_dir/* >> /local_dir/localfile.txt
hadoop fs -cat /hdfs_dir/* >> /local_dir/localfile.txt
hadoop fs -get /hdfs_dir/* >> /local_dir/

My thumb-rule is to avoid using 'text' and 'cat' for big files

Answer 1

The main difference between -cat and -text is that text detects the encoding of the file and decodes it to plain text whenever possible whereas cat doesnt do it.

for eg take the example of this lzo compressed file.

using text:

hadoop fs -text hdfs://namenode:8020/user/hive/warehouse/database/000000_0.lzo_deflate
1,bye
2,back
3,baz
4,hey
5,country

using cat:

 hadoop fs -cat hdfs://namenode:8020/user/hive/warehouse/database/000000_0.lzo_deflate 
    ίiW3�I���2�IJ,�2�U\&:�99�\�:��E9)\֙��"

dfs -get command is used to copy files to local-filesystem.