Short description of the problem

This error appeared when I need to create such architecture:

  • in s3 bucket we collect logs
  • lambda has s3 bucket trigger which will be invoked once the file will be in place
  • lambda has a role which allows it to talk to the same s3 bucket and get files from it

When I described it as usual I got an error which says that these resources (bucket, lambda and role) were involved in circular dependency.

Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Circular dependency between resources: [HelloWorldFunctionSendLogEventPermission, ArchivedLogsBucket, RoleLambdaAccessS3Bucket, HelloWorldFunction]

The strange thing here is that bucket is not dependent on lambda or role. You can take a look at this yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  AWS

  Sample SAM Template for AWS

Globals:
  Function:
    Timeout: 3

Resources:
  ArchivedLogsBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub ${AWS::AccountId}-cloudwatch-monitoring
  RoleLambdaAccessS3Bucket:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub RoleLambdaAccessS3Bucket${AWS::AccountId}
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: PolicyLambdaAccessS3Bucket
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: arn:aws:logs:*:*:*
              - Effect: Allow
                Action:
                  - s3:GetBucketLocation
                  - s3:GetObject
                  - s3:ListBucket
                Resource:
                  - !Join
                    - ''
                    - - 'arn:aws:s3:::'
                      - !Ref ArchivedLogsBucket
                  - !Join
                    - ''
                    - - 'arn:aws:s3:::'
                      - !Ref ArchivedLogsBucket
                      - '/*'
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.7
      Role: !GetAtt 'RoleLambdaAccessS3Bucket.Arn'
      Events:
        SendLogEvent:
          Type: S3
          Properties:
            Bucket: !Ref ArchivedLogsBucket
            Events: s3:ObjectCreated:Put

The reason of the circular dependency

As it turned out S3 bucket has the dependency on the Lambda function even without direct link to it in cloudformation script. The thing is that for s3 bucket to be able to notify lambda about some events that happened special permission has to be created, something like this:

1
2
3
4
5
MyS3BucketPermission
  Type: AWS::Lambda::Permission
  Properties:
    Action: lambda:InvokeFunction
    SourceArn: !Ref MyS3Bucket

The problem is that s3 bucket needs permission resource that will be created after lambda creation, at the same time lambda function is dependent on Role which in its turn dependent on the bucket. This situation creates circular dependency.

It looks like two points are crucial here: bucket permission which will be created automatically and lambda’s role that has dependency on the bucket.

The way to fix circular dependency between bucket and lambda function

The dependency appears because we are using such function as Ref which returns the logical id of the resource, but apparently this id can not be accessed before resource will be created. In order to fix this problem we need some abstraction between resources that are dependent.

We know that Ref returns the name of the bucket and at the same time we can use any string for this so this can be our abstraction point. Instead of using Ref function we can use just the name, in my case I wanted to make the bucket name unique by adding account id to it. In order to do this I used Sub function to concatenate with sum suffix.

Despite Sub function is used, the idea is that instead of using Ref or GetAtt functions we have to use the string which represents the bucket name. So our final script will look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  AWS

  Sample SAM Template for AWS

Globals:
  Function:
    Timeout: 3

Resources:
  ArchivedLogsBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub ${AWS::AccountId}-cloudwatch-monitoring
  RoleLambdaAccessS3Bucket:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub RoleLambdaAccessS3Bucket${AWS::AccountId}
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: PolicyLambdaAccessS3Bucket
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: arn:aws:logs:*:*:*
              - Effect: Allow
                Action:
                  - s3:GetBucketLocation
                  - s3:GetObject
                  - s3:ListBucket
                Resource:
                  - !Sub ${AWS::AccountId}-cloudwatch-monitoring
                  - !Join
                    - ''
                    - - !Sub ${AWS::AccountId}-cloudwatch-monitoring
                      - '/*'
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.7
      Role: !GetAtt 'RoleLambdaAccessS3Bucket.Arn'
      Events:
        SendLogEvent:
          Type: S3
          Properties:
            Bucket: !Ref ArchivedLogsBucket
            Events: s3:ObjectCreated:Put

Pay attention to the Resource sections in RoleLambdaAccessS3Bucket which now contains Sub function. So right now Role is dependent only on the bucket name instead of the reference to the bucket.

Try to run the cloudformation package and deploy commands - should work right now.

The only question that I did not get is why then Ref to the bucket in Lambda function does not create the dependency, but I did not find the answer yet.

Updated: