Subscribe kinesis stream to listen cloudwatch

Short description of the problem

Recently I need to find the way to subscribe kinesis stream to listen all aws log groups: any log group that was created had to send logs directly into kinesis.

One solution that can help to achieve this goal is to:

• create cloudwatch rule which will be triggered by logs.amazon.com event with name CreateLogGroup
• create lambda function that will be triggered by that cloudwatch rule. This lambda function:
  • fetches the list of all log groups
  • iterates through this list and subscribes log groups to the kinesis stream if they were not yet subscribed

So in general it is mostly the same as the approach described here The only difference is that you have to know log group name to create subscription filter for specific group. In our case we need to do it dynamically for all possible groups: existed and that will be created in the future.

Everything looked fine, but I stuck with such problems as:

• which event we need to listen in cloudwatch rule
• lambda did not have proper rights to do the proper link between cloudwatch group and kinesis

Which event we need to listen

The proper event is aws.logs and eventName should be CreateLogGroup. In cloudformation template (sam template to be more precise) the lambda template has to have Events block as the following:

Events:
  CloudWatchEvent:
    Type: CloudWatchEvent
    Properties:
      Pattern:
        source:
          - aws.logs
        detail:
          eventName:
            - CreateLogGroup

What permissions lambda should have to be able to create proper subscription filter

This question is a bit trickier as for me. By itself lambda should be able to have role to:

• describe log groups
• describe subscription filters
• put subscription filters
• create log group
• creat log stream
• put log events

But the most important that this role has to allow to pass role (action iam:PassRole) to the cloudwatch service. The role that will be passed should allow to put records in the kinesis stream.

NOTE: when lambda executes put subscription filter operation it passes the specified role to cloudwatch and from that moment cloudwatch is allowed to put records to kinesis.

Disadvantages

Of course it is not really clean approach and it has it’s disadvantages like:

• we need lambda to add subscription filters. In perfect world would be great to have general config that allows to subscribe log groups to specific resource by default
• logs that appear before subscription was added to listen log group will be lost
• big amount of resources involved in this solution