TRIM HORIZON AWS definition for DynamoDB Triggers Streams Lambda

0 votes
I have an IoT device sending irregular updates that I store in a DynamoDB Table. Now, I want to process just the recent updates onto another DynamoDB table and save it there. So, I need to use 'n' number of latest updates from the first table to compute an update in my second Table for that same device and ensure that they are in sync with all my original updates, sort of like a sliding window. And, while the DynamoDB Triggers (Streams + Lambda) suit my needs perfectly, I just can't seem to find any clear definition or documentation of TRIM_HORIZON. After referring to a few other docs, I figured it could be the oldest data in my first Table, which is a lot. But some docs also suggest that it is just for the past 24hr. Now, I don't know if it means the oldest data in the stream for the last 24hr, but I can only guess. So, can anyone please tell me which is actually true about TRIM_HORIZON, like will I be able to configure it?

Now, if it's not possible and I cannot use TRIM_HORIZON, I'd have to switch to using LATEST for performing a query on the first Table. But, that would just defeat the entire purpose of my streams. Please, advice!
Nov 15, 2018 in IoT (Internet of Things) by Bharani
• 4,660 points
5,473 views

1 answer to this question.

0 votes
Here's what I found in the DynamoDB's documentation:

(https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html)
All data in DynamoDB Streams is subject to a 24 hour lifetime. You can retrieve and analyze the last 24 hours of activity for any given table

(https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/Welcome.html)
TRIM_HORIZON - Start reading at the last (untrimmed) stream record, which is the oldest record in the shard. In DynamoDB Streams, there is a 24 hour limit on data retention. Stream records whose age exceeds this limit are subject to removal (trimming) from the stream.

This means if you have a Lambda continuously processing your stream updates, you could just go on with using LATEST. And, because you must use 'n' last updates to compute an update in your second Table, you'd as it is, need to query the first Table for each and every update in order to 'merge' current update with previous ones from your device. So, I think its highly unlikely for just about anyone to do something that using TRIM_HORIZON.
answered Nov 15, 2018 by DataKing99
• 8,250 points

Related Questions In IoT (Internet of Things)

0 votes
0 answers
0 votes
1 answer

AWS Lambda : JSON object undefined

Change event.TagData.Time; To event.TagData[0].Time; This sh ...READ MORE

answered Jan 25, 2019 in IoT (Internet of Things) by Shubham
• 13,490 points
2,268 views
0 votes
1 answer

AWS : updating DynamoDB

DynamoDb range key cannot have null attributes. ...READ MORE

answered Jan 28, 2019 in IoT (Internet of Things) by Upasana
• 8,620 points
835 views
0 votes
1 answer
0 votes
1 answer
+2 votes
3 answers
0 votes
1 answer

RPMs for IoT Agents of Backend Device Management GE in FIWARE IoT ecosystem

The RPMs for IDAS component are availaible. ...READ MORE

answered Jul 30, 2018 in IoT (Internet of Things) by DataKing99
• 8,250 points
803 views
0 votes
2 answers

AWS-MQTT Connection

Link --> https://docs.aws.amazon.com/greengrass/latest/developerguide/module3-I.html This is Very Helpful to you.If ...READ MORE

answered Apr 3, 2019 in IoT (Internet of Things) by KanaguRaj
• 160 points
1,598 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP