Hadoop HDFS How to delete old files from HDFS

0 votes

I want to delete older files from hdfs, say files that are older than 10 days. If I had to do this in linux, I would do something like this:

find /path/to/directory/ -type f -mtime +10 -name '*.txt' -execdir rm -- {} \;

So how to do this for hdfs?

Nov 15, 2018 in Big Data Hadoop by digger
• 26,740 points
14,168 views

1 answer to this question.

+1 vote

You can use commands like this:

hdfs dfs -ls /file/Path    |   tr -s " "    |    cut -d' ' -f6-8    |     grep "^[0-9]"    |    awk 'BEGIN{ MIN=14400; LAST=60*MIN; "date +%s" | getline NOW } { cmd="date -d'\''"$1" "$2"'\'' +%s"; cmd | getline WHEN; DIFF=NOW-WHEN; if(DIFF > LAST){ print "Deleting: "$3; system("hdfs dfs -rm -r "$3) }}'

Or you can use shell script:

today=`date +'%s'`
hdfs dfs -ls /file/Path/ | grep "^d" | while read line ; do
dir_date=$(echo ${line} | awk '{print $6}')
difference=$(( ( ${today} - $(date -d ${dir_date} +%s) ) / ( 24*60*60 ) ))
filePath=$(echo ${line} | awk '{print $8}')

if [ ${difference} -gt 10 ]; then
    hdfs dfs -rm -r $filePath
fi
done
answered Nov 15, 2018 by Omkar
• 69,220 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to delete a directory from Hadoop cluster which is having comma(,) in its name?

Just try the following command: hadoop fs -rm ...READ MORE

answered May 7, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
3,083 views
0 votes
1 answer

Hadoop: How to copy directory from local system to hdfs using Java code?

Just use the FileSystem's copyFromLocalFile method. If the source Path ...READ MORE

answered Nov 14, 2018 in Big Data Hadoop by Omkar
• 69,220 points
5,879 views
0 votes
1 answer

How to copy files from HDFS to local system?

Hi@akhtar, You can use the put command in ...READ MORE

answered Oct 1, 2020 in Big Data Hadoop by MD
• 95,460 points
4,351 views
0 votes
1 answer

How to remove files from HDFS system?

Hi@akhtar, You will find rm command in your ...READ MORE

answered Oct 1, 2020 in Big Data Hadoop by MD
• 95,460 points
25,683 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
11,071 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,569 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
109,025 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,350 points
4,635 views
0 votes
1 answer

How to delete a file from Hadoop hdfs using java?

Try this: FileSystem fs = FileSystem.get(getConf()); fs.delete(new ...READ MORE

answered Nov 28, 2018 in Big Data Hadoop by Omkar
• 69,220 points
5,847 views
0 votes
1 answer

How to delete hdfs folder from java app?

Try this: Configuration conf = new Configuration(); ...READ MORE

answered Oct 30, 2018 in Big Data Hadoop by Omkar
• 69,220 points
2,144 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP