Kinesis Analytics Destination Guidance: Lambda vs Kinesis Stream to Lambda

0 votes

After Kinesis Analytics does it's job, the next step is to send that information off to a destination. AWS currently offers 3 destination choices:

  • Kinesis stream

  • Kinesis Firehose delivery stream

  • AWS Lambda function

For my use case, Kinesis Firehose delivery stream is not what I want so I am left with:

  • Kinesis stream

  • AWS Lambda function

If I set the destination to a Kinesis Stream, I would then attach a Lambda to that stream to process the records.

AWS also offers the ability to set the destination to a Lambda, bypassing the Kinesis Stream step of this process. In doing some digging for docs I found this:

Using a Lambda Function as Output

Specifically in those docs under Lambda Output Invocation Frequency it says:

If records are emitted to the destination in-application stream within the data analytics application as a continuous query or a sliding window, the AWS Lambda destination function is invoked approximately once per second.

My Kinesis Analytics output qualifies under this scenario. So I can assume that my Lambda will be invoked, "approximately once per second".

I'm trying to understand the difference between using these 2 destinations as it pertains to using a Lambda.

Using AWS Lambda with Kinesis states that:

You can subscribe Lambda functions to automatically read batches of records off your Kinesis stream and process them if records are detected on the stream. AWS Lambda then polls the stream periodically (once per second) for new records.

So it sounds like the the invocation interval is the same in either case; approximately 1 second.

So I think the guidence is:

If the next stage in the pipeline only needs one consumer, then use the AWS Lambda functiondestination. If however, you need to use multiple different consumers to do different things for the same data sent to the destination, the a Kinesis Stream is more appropriate.

Is this a correct assumption on how to choose a destination? Again, for my use case I am excluding the Kinesis Firehose delivery stream.

Aug 21, 2018 in AWS by bug_seeker
• 15,350 points
492 views

1 answer to this question.

0 votes

If this does not answer your question, please clarify it. There is a little ambiguity.

I would always use Kinesis Stream with one shard and batch size = 1 (for example) if I wanted the items to be consumed one by one with no concurrency.

If there are multiple consumers, increase the number of shards, one lambda is launched in parallel for each shard when there are items to process. If it makes sense, also increase the batch size.

But read again at the highlighted phrase below:

If however, you need to use multiple different consumers to do different things for the same data sent to the destination, the a Kinesis Stream is more appropriate.

If you have one or more producers and many consumers of the exactly same item, I guess you need to use SNS. The producer writes the item on one topic, then all the lambdas listening to the topic will process that item.

answered Aug 21, 2018 by Priyaj
• 56,900 points

Related Questions In AWS

0 votes
1 answer

How do I invoke another lambda async and pass context to it?

I tried several times trying to pass ...READ MORE

answered Apr 27, 2018 in AWS by Cloud gunner
• 4,280 points
1,712 views
0 votes
2 answers
0 votes
1 answer

Trying to upload data stream in Amazon S3

Try adding the ContentType & ACL in ...READ MORE

answered Jun 28, 2018 in AWS by Cloud gunner
• 4,280 points
386 views
0 votes
1 answer
0 votes
1 answer
+1 vote
3 answers

Not able to pass params POST to AWS Lambda from Amazon API Gateway

For this template : application/x-www-form-urlencoded  This line below will ...READ MORE

answered Jun 13, 2018 in AWS by Cloud gunner
• 4,280 points
2,223 views
+2 votes
3 answers
0 votes
1 answer