How to find previous records from a data set in Pig

+1 vote
Hi everyone

I'm trying to find previous records from a large dataset in Pig. Say I want to find the data of last 1 month.

Which command will be used for that.

Can anyone please answer me earliest.

Thank you
Jan 17, 2020 in Big Data Hadoop by Hasid
• 370 points
181 views

2 answers to this question.

0 votes

hi @Nadeem,

Convert your date field to DateTime data type using the ToDate() function. Use the CurrentTime() and get the difference between the two dates using DaysBetween() and filter accordingly.

Hope this helps :)

answered Jan 20, 2020 by Kalgi
• 52,290 points
Thanks for your reply.
Can you please share the structure of your dataset or your dataset. I'll help you with it.
0 votes

Hi,

You can use ToDate() and SubtractDuration() function to find the previous records.

Say, You want to find the previous 100 days records from a file. 

$ Filter data by ToDate(Inspection_Date,'MM/dd/yyyy') > SubtractDuration(CurrentTime(),'P100D');

Hope this will work.

Thank You

answered Jan 23, 2020 by MD
• 94,990 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to create a parquet table in hive and store data in it from a hive table?

Please use the code attached below for ...READ MORE

answered Jan 28, 2019 in Big Data Hadoop by Omkar
• 69,130 points
12,060 views
0 votes
1 answer

How to delete a directory from Hadoop cluster which is having comma(,) in its name?

Just try the following command: hadoop fs -rm ...READ MORE

answered May 7, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
823 views
0 votes
1 answer

How to create a Hive table from sequence file stored in HDFS?

There are two SerDe for SequenceFile as ...READ MORE

answered Dec 17, 2018 in Big Data Hadoop by Omkar
• 69,130 points
2,085 views
0 votes
1 answer

How to load data from HDFS into pig relation?

Hey, To load data from HDFS to pig ...READ MORE

answered May 7, 2019 in Big Data Hadoop by Gitika
• 65,850 points
805 views
0 votes
1 answer

How to create a file in Linux from terminal window?

Hey, Nothing to worry about creating any file ...READ MORE

answered May 13, 2019 in Big Data Hadoop by Gitika
• 65,850 points
70 views
0 votes
1 answer

How to import data in sqoop as a Parquet file?

Sqoop allows you to import the file ...READ MORE

answered May 15, 2019 in Big Data Hadoop by Nanda
4,626 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
6,653 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,048 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
45,488 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
2,514 views