How to check size of HDFS directory

Question

In case of Linux filesystems we use du -sh. But is there any way to check directory size in case of HDFS?

Answer 1

You can view the size of the files and directories in a specific directory with the du command. The command will show you the space (in bytes) used by the files that match the file pattern you specify. If it’s a file, you’ll get the length of the file. The syntax of the du command is as follows:

hdfs dfs -du -h /"path to specific hdfs directory"

Note the following about the output of the du –h command shown here:

The first column shows the actual size (raw size) of the files that users have placed in the various HDFS directories.

The second column shows the actual space consumed by those files in HDFS.

Hope this will answer your query to some extent.

For details, You can even check out Hadoop Ecosystem tools with the Hadoop big data course.

Answer 2

hdfs dfs -du [-s] [-h] URI [URI …]

Answer 3

hadoop fs -du -s -h /path/to/dir

Answer 4

To get the size in Gb, you can try this:

hdfs dfs -du PATHTODIRECTORY | awk '/^[0-9]+/ { print int($1/(1024**3)) " [GB]\t" $2 }'

Answer 5

hadoop fs -du /user/hadoop/dir1 \
    /user/hadoop/file1 \
    hdfs://domain.com/user/hadoop/dir1 

Answer 6

hdfs dfs -du -s -h /$DirectoryName

Answer 7

Using the following command, you'll get the size in %:

sudo -u hdfs hadoop fs –df

Answer 8

To check the size under a particular directory:

sudo -u hdfs hadoop fs -du -h /user

Answer 9

hdfs dfs -du -s dir_name

Answer 10

Another way to show size in GB:

hadoop fs -dus  /path/to/dir  |   awk '{print $2/1024**3 " G"}'