How to find previous records from a data set in Pig??

+1 vote
Hi everyone

I'm trying to find previous records from a large dataset in Pig. Say I want to find the data of last 1 month.

Which command will be used for that.

Can anyone please answer me earliest.

Thank you
Jan 17 in Big Data Hadoop by Hasid
• 330 points
68 views

2 answers to this question.

0 votes

hi @Nadeem,

Convert your date field to DateTime data type using the ToDate() function. Use the CurrentTime() and get the difference between the two dates using DaysBetween() and filter accordingly.

Hope this helps :)

answered Jan 20 by Kalgi
• 51,830 points
Thanks for your reply.
Can you please share the structure of your dataset or your dataset. I'll help you with it.
0 votes

Hi,

You can use ToDate() and SubtractDuration() function to find the previous records.

Say, You want to find the previous 100 days records from a file. 

$ Filter data by ToDate(Inspection_Date,'MM/dd/yyyy') > SubtractDuration(CurrentTime(),'P100D');

Hope this will work.

Thank You

answered Jan 23 by MD
• 8,510 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to create a parquet table in hive and store data in it from a hive table?

Please use the code attached below for ...READ MORE

answered Jan 28, 2019 in Big Data Hadoop by Omkar
• 69,000 points
7,152 views
0 votes
1 answer

How to create a Hive table from sequence file stored in HDFS?

There are two SerDe for SequenceFile as ...READ MORE

answered Dec 17, 2018 in Big Data Hadoop by Omkar
• 69,000 points
1,016 views
0 votes
1 answer

How to load data from HDFS into pig relation?

Hey, To load data from HDFS to pig ...READ MORE

answered May 7, 2019 in Big Data Hadoop by Gitika
• 26,270 points
288 views
0 votes
1 answer

How to create a file in Linux from terminal window?

Hey, Nothing to worry about creating any file ...READ MORE

answered May 13, 2019 in Big Data Hadoop by Gitika
• 26,270 points
39 views
0 votes
1 answer

How to import data in sqoop as a Parquet file?

Sqoop allows you to import the file ...READ MORE

answered May 15, 2019 in Big Data Hadoop by Nanda
2,002 views
0 votes
1 answer

How to delete a column family from table in HBase?

Hey, You can delete a column family from ...READ MORE

answered Jun 20, 2019 in Big Data Hadoop by Gitika
• 26,270 points
400 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,870 points
4,541 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,870 points
643 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
25,681 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,310 points
1,707 views