How to find previous records from a data set in Pig

+1 vote
Hi everyone

I'm trying to find previous records from a large dataset in Pig. Say I want to find the data of last 1 month.

Which command will be used for that.

Can anyone please answer me earliest.

Thank you
Jan 17, 2020 in Big Data Hadoop by Hasid
• 370 points
214 views

2 answers to this question.

0 votes

hi @Nadeem,

Convert your date field to DateTime data type using the ToDate() function. Use the CurrentTime() and get the difference between the two dates using DaysBetween() and filter accordingly.

Hope this helps :)

answered Jan 20, 2020 by Kalgi
• 52,310 points
Thanks for your reply.
Can you please share the structure of your dataset or your dataset. I'll help you with it.
0 votes

Hi,

You can use ToDate() and SubtractDuration() function to find the previous records.

Say, You want to find the previous 100 days records from a file. 

$ Filter data by ToDate(Inspection_Date,'MM/dd/yyyy') > SubtractDuration(CurrentTime(),'P100D');

Hope this will work.

Thank You

answered Jan 23, 2020 by MD
• 95,140 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to create a parquet table in hive and store data in it from a hive table?

Please use the code attached below for ...READ MORE

answered Jan 28, 2019 in Big Data Hadoop by Omkar
• 69,110 points
13,122 views
0 votes
1 answer

How to delete a directory from Hadoop cluster which is having comma(,) in its name?

Just try the following command: hadoop fs -rm ...READ MORE

answered May 7, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
1,069 views
0 votes
1 answer

How to create a Hive table from sequence file stored in HDFS?

There are two SerDe for SequenceFile as ...READ MORE

answered Dec 17, 2018 in Big Data Hadoop by Omkar
• 69,110 points
2,301 views
0 votes
1 answer

How to load data from HDFS into pig relation?

Hey, To load data from HDFS to pig ...READ MORE

answered May 7, 2019 in Big Data Hadoop by Gitika
• 65,870 points
918 views
0 votes
1 answer

How to create a file in Linux from terminal window?

Hey, Nothing to worry about creating any file ...READ MORE

answered May 13, 2019 in Big Data Hadoop by Gitika
• 65,870 points
91 views
0 votes
1 answer

How to import data in sqoop as a Parquet file?

Sqoop allows you to import the file ...READ MORE

answered May 15, 2019 in Big Data Hadoop by Nanda
5,226 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
7,082 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,138 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
51,108 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
2,679 views