Short description of the problem
This error appeared when I need to create such architecture:
- in s3 bucket we collect logs
- lambda has s3 bucket trigger which will be invoked once the file will be in place
- lambda has a role which allows it to talk to the same s3 bucket and get files from it
When I described it as usual I got an error which says that these resources (bucket, lambda and role) were involved in circular dependency.
Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Circular dependency between resources: [HelloWorldFunctionSendLogEventPermission, ArchivedLogsBucket, RoleLambdaAccessS3Bucket, HelloWorldFunction]The strange thing here is that bucket is not dependent on lambda or role. You can take a look at this yaml:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
AWS
Sample SAM Template for AWS
Globals:
Function:
Timeout: 3
Resources:
ArchivedLogsBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub ${AWS::AccountId}-cloudwatch-monitoring
RoleLambdaAccessS3Bucket:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub RoleLambdaAccessS3Bucket${AWS::AccountId}
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: PolicyLambdaAccessS3Bucket
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: arn:aws:logs:*:*:*
- Effect: Allow
Action:
- s3:GetBucketLocation
- s3:GetObject
- s3:ListBucket
Resource:
- !Join
- ''
- - 'arn:aws:s3:::'
- !Ref ArchivedLogsBucket
- !Join
- ''
- - 'arn:aws:s3:::'
- !Ref ArchivedLogsBucket
- '/*'
HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: hello_world/
Handler: app.lambda_handler
Runtime: python3.7
Role: !GetAtt 'RoleLambdaAccessS3Bucket.Arn'
Events:
SendLogEvent:
Type: S3
Properties:
Bucket: !Ref ArchivedLogsBucket
Events: s3:ObjectCreated:Put
The reason of the circular dependency
As it turned out S3 bucket has the dependency on the Lambda function even without direct link
to it in cloudformation script. The thing is that for s3 bucket to be able to notify lambda about
some events that happened special permission has to be created, something like this:
1
2
3
4
5
MyS3BucketPermission
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
SourceArn: !Ref MyS3Bucket
The problem is that s3 bucket needs permission resource that will be created after lambda creation, at the same time lambda function is dependent on Role which in its turn dependent on the bucket. This situation creates circular dependency.
It looks like two points are crucial here: bucket permission which will be created automatically and lambda’s role that has dependency on the bucket.
The way to fix circular dependency between bucket and lambda function
The dependency appears because we are using such function as Ref which returns the logical id
of the resource, but apparently this id can not be accessed before resource will be created. In
order to fix this problem we need some abstraction between resources that are dependent.
We know that Ref returns the name of the bucket and at the same time we can use any string for
this so this can be our abstraction point. Instead of using Ref function we can use just the
name, in my case I wanted to make the bucket name unique by adding account id to it. In order to
do this I used Sub function to concatenate with sum suffix.
Despite Sub function is used, the idea is that instead of using Ref or GetAtt functions we
have to use the string which represents the bucket name. So our final script will look like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
AWS
Sample SAM Template for AWS
Globals:
Function:
Timeout: 3
Resources:
ArchivedLogsBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub ${AWS::AccountId}-cloudwatch-monitoring
RoleLambdaAccessS3Bucket:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub RoleLambdaAccessS3Bucket${AWS::AccountId}
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: PolicyLambdaAccessS3Bucket
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: arn:aws:logs:*:*:*
- Effect: Allow
Action:
- s3:GetBucketLocation
- s3:GetObject
- s3:ListBucket
Resource:
- !Sub ${AWS::AccountId}-cloudwatch-monitoring
- !Join
- ''
- - !Sub ${AWS::AccountId}-cloudwatch-monitoring
- '/*'
HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: hello_world/
Handler: app.lambda_handler
Runtime: python3.7
Role: !GetAtt 'RoleLambdaAccessS3Bucket.Arn'
Events:
SendLogEvent:
Type: S3
Properties:
Bucket: !Ref ArchivedLogsBucket
Events: s3:ObjectCreated:Put
Pay attention to the Resource sections in RoleLambdaAccessS3Bucket which now contains Sub
function. So right now Role is dependent only on the bucket name instead of the reference to the
bucket.
Try to run the cloudformation package and deploy commands - should work right now.
The only question that I did not get is why then Ref to the bucket in Lambda function does not
create the dependency, but I did not find the answer yet.
