TRIM_HORIZON: AWS definition for DynamoDB Triggers (Streams + Lambda)

0 votes
I have an IoT device sending irregular updates that I store in a DynamoDB Table. Now, I want to process just the recent updates onto another DynamoDB table and save it there. So, I need to use 'n' number of latest updates from the first table to compute an update in my second Table for that same device and ensure that they are in sync with all my original updates, sort of like a sliding window. And, while the DynamoDB Triggers (Streams + Lambda) suit my needs perfectly, I just can't seem to find any clear definition or documentation of TRIM_HORIZON. After referring to a few other docs, I figured it could be the oldest data in my first Table, which is a lot. But some docs also suggest that it is just for the past 24hr. Now, I don't know if it means the oldest data in the stream for the last 24hr, but I can only guess. So, can anyone please tell me which is actually true about TRIM_HORIZON, like will I be able to configure it?

Now, if it's not possible and I cannot use TRIM_HORIZON, I'd have to switch to using LATEST for performing a query on the first Table. But, that would just defeat the entire purpose of my streams. Please, advice!
Nov 15, 2018 in IoT (Internet of Things) by Bharani
• 4,550 points
612 views

1 answer to this question.

0 votes
Here's what I found in the DynamoDB's documentation:

(https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html)
All data in DynamoDB Streams is subject to a 24 hour lifetime. You can retrieve and analyze the last 24 hours of activity for any given table

(https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/Welcome.html)
TRIM_HORIZON - Start reading at the last (untrimmed) stream record, which is the oldest record in the shard. In DynamoDB Streams, there is a 24 hour limit on data retention. Stream records whose age exceeds this limit are subject to removal (trimming) from the stream.

This means if you have a Lambda continuously processing your stream updates, you could just go on with using LATEST. And, because you must use 'n' last updates to compute an update in your second Table, you'd as it is, need to query the first Table for each and every update in order to 'merge' current update with previous ones from your device. So, I think its highly unlikely for just about anyone to do something that using TRIM_HORIZON.
answered Nov 15, 2018 by DataKing99
• 8,130 points

Related Questions In IoT (Internet of Things)

0 votes
0 answers
0 votes
1 answer

AWS Lambda : JSON object undefined

Change event.TagData.Time; To event.TagData[0].Time; This sho ...READ MORE

answered Jan 24 in IoT (Internet of Things) by Shubham
• 13,310 points
179 views
0 votes
1 answer

AWS : updating DynamoDB

DynamoDb range key cannot have null attributes. ...READ MORE

answered Jan 27 in IoT (Internet of Things) by Upasana
• 8,550 points
60 views
0 votes
1 answer
0 votes
1 answer
+2 votes
3 answers
0 votes
1 answer

RPMs for IoT Agents of Backend Device Management GE in FIWARE IoT ecosystem

The RPMs for IDAS component are availaible. ...READ MORE

answered Jul 29, 2018 in IoT (Internet of Things) by DataKing99
• 8,130 points
56 views
0 votes
2 answers

AWS-MQTT Connection

Link --> https://docs.aws.amazon.com/greengrass/latest/developerguide/module3-I.html This is Very Helpful to you.If ...READ MORE

answered Apr 3 in IoT (Internet of Things) by KanaguRaj
• 160 points
255 views