(@aws-cdk/aws-apigatewayv2-alpha): After adding around 30 routes to a http api, I got policy PolicyLengthExceededException

[tgjorgoski]

What is the problem?

I’m creating http api using: const api = new HttpApi (from '@aws-cdk/aws-apigatewayv2-alpha') , and then I’m adding routes to it, something like this:

api.addRoutes({
      path: '/jobs',
      methods: [apiGateway.HttpMethod.PUT, apiGateway.HttpMethod.GET, apiGateway.HttpMethod.POST],
      integration: lambdaIntegration
    })

I got to around 30 routes, all the routes are connected to one and the same lambda. At which point I got the following exception from the CloudFormation:

22:28:54 | CREATE_FAILED        | AWS::Lambda::Permission             | UserServiceAPIGETn...Permission1A32E9FF
The final policy size (20937) is bigger than the limit (20480). (Service: AWSLambdaInternal; Status Code: 400; Error Code: PolicyLengthExceededException; Request ID: d673
c975-ff84-43da-9120-4be039578371; Proxy: null)

If I understand right, CDK adds policy statements for each route to the resource-based policy of the lambda. The statements look like this:

{
      "Sid": "UserServiceStack-UserServiceAPIPUTjobapplicationsUserServiceAPIPermission1CBB1DA2-fds",
      "Effect": "Allow",
      "Principal": {
        "Service": "apigateway.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:eu-central-1:fds:function:UserServiceStack-UserServiceAPIhandler4CB02A91-rGHPeEZcjJTY",
      "Condition": {
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:execute-api:eu-central-1:fsd:j057ar4qy1/*/*/job-applications"
        }
      }
    },

So, it seems they quickly get the total length of the policy document to the max.
I think I will create two CDK lambdas from the same code, and split it among the routes, to somehow solve the immediate problem, but it might be good if the CDK could automatically solve this.

Reproduction Steps

1. Create a HTTP API
2. Add +30 routes to it all connected to the same lambda (through lambda integration)

What did you expect to happen?

The routes are proprely created

What actually happened?

Got: Service: AWSLambdaInternal; Status Code: 400; Error Code: PolicyLengthExceededException

CDK CLI Version

2.17.0 (build f9cd009)

Framework Version

No response

Node.js Version

14

OS

Mac OS

Language

Typescript

Language Version

No response

Other information

No response

[JPLemelin]

@tgjorgoski take a look at this issues #9327 it's for apigateway v1, but your are facing a very similar issue

[peterwoodworth]

Hey, we've actually just recently merged a PR which aims to tackle all sorts of policy size limit issues. Look forward to the next v2 release and let me know if this is still an issue 🙂

#19114

[tgjorgoski]

@peterwoodworth , I updated all cdk npm packages (including alpha ones) to 2.18.0 , turned on the feature flag and redeployed. I saw bunch of policies being squished, but unfortunately in the case of the apigateway v2 routes, the optimisation isn't enough. It seems to have done something because the new message says The final policy size (20541)..., and the previous one was 20937, however it is still bigger than the limit, and I will probably add more routes.
I guess in this case it is not just simple duplication, but bunch of similar policy statements.

[tgjorgoski]

@JPLemelin , thanks for the pointer! I will look into it, on a first look seems as a good workaround!

[tgjorgoski]

I used the (somewhat modified) solution from #9327 , and it seems to work:

Instead of HttpLambdaIntegration, I used this modified class...

export class HttpLambdaIntegrationNoPermission extends HttpLambdaIntegration {
       
  constructor(id: string, handler: IFunction, props?: HttpLambdaIntegrationProps) {
    super(id, handler, props)
  }

  protected completeBind(options: HttpRouteIntegrationBindOptions): void {
    const permissions = options.route.node.children.filter(child => child instanceof CfnPermission)
    permissions.forEach(permission => options.route.node.tryRemoveChild(permission.node.id))
  }
}

And then in the end I added one general policy for all the routes:

    apiHandlerLambda.addPermission(id + "ApiGWPermissions", {
      action: 'lambda:InvokeFunction',
      principal: new ServicePrincipal('apigateway.amazonaws.com'),
      sourceArn: `arn:aws:execute-api:${props.env!.region!}:${props.env!.account!}:${api.apiId}/*/*/*`
    });

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[peterwoodworth]

Didn't mean to close this, sorry 😅

[peterwoodworth]

Thanks for letting me know the issue is still occurring. A bit disappointing to hear

[magidnadav]

HttpLambdaIntegration caused the same problem for me as well.
The following worked for me:

export class HttpLambdaIntegrationNoPermission extends HttpLambdaIntegration {
  constructor(id: string, handler: IFunction, props?: HttpLambdaIntegrationProps) {
    super(id, handler, props);
  }

  completeBind(): void {
    return;
  }
}

[rib]

I think to some extent there is some overlap with issue #9327 here.

I recently looked at switching a project from Serverless Framework to cdk and ended up hitting this issue and as a result I wasn't able to deploy anything while the generated policy was too large.

It was quite a pain I'd say but I did manage to figure out a (non-trivial, hacky) workaround which I posted here: #9327 (comment) in case that helps anyone else.

For reference I'm using cdk 2.52.0 (cli and libs)