How to find previous records from a data set in Pig

+1 vote
Hi everyone

I'm trying to find previous records from a large dataset in Pig. Say I want to find the data of last 1 month.

Which command will be used for that.

Can anyone please answer me earliest.

Thank you
Jan 17, 2020 in Big Data Hadoop by Hasid
• 370 points
362 views

2 answers to this question.

0 votes

hi @Nadeem,

Convert your date field to DateTime data type using the ToDate() function. Use the CurrentTime() and get the difference between the two dates using DaysBetween() and filter accordingly.

Hope this helps :)

answered Jan 20, 2020 by Kalgi
• 52,350 points
Thanks for your reply.
Can you please share the structure of your dataset or your dataset. I'll help you with it.
0 votes

Hi,

You can use ToDate() and SubtractDuration() function to find the previous records.

Say, You want to find the previous 100 days records from a file. 

$ Filter data by ToDate(Inspection_Date,'MM/dd/yyyy') > SubtractDuration(CurrentTime(),'P100D');

Hope this will work.

Thank You

answered Jan 23, 2020 by MD
• 95,340 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to create a parquet table in hive and store data in it from a hive table?

Please use the code attached below for ...READ MORE

answered Jan 28, 2019 in Big Data Hadoop by Omkar
• 69,190 points
15,684 views
0 votes
1 answer

How to delete a directory from Hadoop cluster which is having comma(,) in its name?

Just try the following command: hadoop fs -rm ...READ MORE

answered May 7, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
1,876 views
0 votes
1 answer

How to create a Hive table from sequence file stored in HDFS?

There are two SerDe for SequenceFile as ...READ MORE

answered Dec 18, 2018 in Big Data Hadoop by Omkar
• 69,190 points
3,289 views
0 votes
1 answer

How to load data from HDFS into pig relation?

Hey, To load data from HDFS to pig ...READ MORE

answered May 7, 2019 in Big Data Hadoop by Gitika
• 65,850 points
1,696 views
0 votes
1 answer

How to create a file in Linux from terminal window?

Hey, Nothing to worry about creating any file ...READ MORE

answered May 13, 2019 in Big Data Hadoop by Gitika
• 65,850 points
158 views
0 votes
1 answer

How to import data in sqoop as a Parquet file?

Sqoop allows you to import the file ...READ MORE

answered May 15, 2019 in Big Data Hadoop by Nanda
7,431 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
8,643 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,525 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
76,441 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
3,312 views
webinar REGISTER FOR FREE WEBINAR X
Send OTP
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP