Hadoop HDFS How to delete old files from HDFS

0 votes

I want to delete older files from hdfs, say files that are older than 10 days. If I had to do this in linux, I would do something like this:

find /path/to/directory/ -type f -mtime +10 -name '*.txt' -execdir rm -- {} \;

So how to do this for hdfs?

Nov 15, 2018 in Big Data Hadoop by digger
• 26,700 points
6,417 views

1 answer to this question.

0 votes

You can use commands like this:

hdfs dfs -ls /file/Path    |   tr -s " "    |    cut -d' ' -f6-8    |     grep "^[0-9]"    |    awk 'BEGIN{ MIN=14400; LAST=60*MIN; "date +%s" | getline NOW } { cmd="date -d'\''"$1" "$2"'\'' +%s"; cmd | getline WHEN; DIFF=NOW-WHEN; if(DIFF > LAST){ print "Deleting: "$3; system("hdfs dfs -rm -r "$3) }}'

Or you can use shell script:

today=`date +'%s'`
hdfs dfs -ls /file/Path/ | grep "^d" | while read line ; do
dir_date=$(echo ${line} | awk '{print $6}')
difference=$(( ( ${today} - $(date -d ${dir_date} +%s) ) / ( 24*60*60 ) ))
filePath=$(echo ${line} | awk '{print $8}')

if [ ${difference} -gt 10 ]; then
    hdfs dfs -rm -r $filePath
fi
done
answered Nov 15, 2018 by Omkar
• 69,130 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to delete a directory from Hadoop cluster which is having comma(,) in its name?

Just try the following command: hadoop fs -rm ...READ MORE

answered May 7, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
814 views
0 votes
1 answer

Hadoop: How to copy directory from local system to hdfs using Java code?

Just use the FileSystem's copyFromLocalFile method. If the source Path ...READ MORE

answered Nov 14, 2018 in Big Data Hadoop by Omkar
• 69,130 points
2,577 views
0 votes
1 answer

How to copy files from HDFS to local system?

Hi@akhtar, You can use the put command in ...READ MORE

answered Oct 1, 2020 in Big Data Hadoop by MD
• 94,990 points
94 views
0 votes
1 answer

How to remove files from HDFS system?

Hi@akhtar, You will find rm command in your ...READ MORE

answered Oct 1, 2020 in Big Data Hadoop by MD
• 94,990 points
96 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
6,634 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,046 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
45,308 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
2,508 views
0 votes
1 answer

How to delete a file from Hadoop hdfs using java?

Try this: FileSystem fs = FileSystem.get(getConf()); fs.delete(new ...READ MORE

answered Nov 28, 2018 in Big Data Hadoop by Omkar
• 69,130 points
2,818 views
0 votes
1 answer

How to delete hdfs folder from java app?

Try this: Configuration conf = new Configuration(); ...READ MORE

answered Oct 30, 2018 in Big Data Hadoop by Omkar
• 69,130 points
870 views