Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cdk-pipelines: pipelines.ConfirmPermissionsBroadening not skipping changes without Permission Broadening #32482

Open
1 task
ChristophrK opened this issue Dec 11, 2024 · 2 comments
Labels
@aws-cdk/pipelines CDK Pipelines library bug This issue is a bug. effort/medium Medium work item – several days of effort p2

Comments

@ChristophrK
Copy link

ChristophrK commented Dec 11, 2024

Describe the bug

In the past if the cdk diff didnt show any security changes the ConfirmPermissionsBroadening (CodeBuild Action) would call the CDKPipelinesAutoApprove Lambda Function which automatically would approve the Lambda Function. This is not happening anymore and a manual approval is always required.

We noticed that this feature stopped working for us in multiple indepent Accounts and Repositories without any updates (cdk-Version ,Bootstrapping etc.) on our end. The last time that we saw the step working as expected was November 4th.

Regression Issue

  • Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

I would expect that the CDK diff detects that there are no changes and calls the Lambda Function. This was the behaviour in the past. The Logs of the CodeBuild look like this:

[Container] 2024/11/04 16:54:49.319085 Running on CodeBuild On-demand
[Container] 2024/11/04 16:54:49.319098 Waiting for agent ping
[Container] 2024/11/04 16:54:49.520531 Waiting for DOWNLOAD_SOURCE
[Container] 2024/11/04 16:54:50.777909 Phase is DOWNLOAD_SOURCE
[Container] 2024/11/04 16:54:50.815397 CODEBUILD_SRC_DIR=/codebuild/output/srcXXXXXXX/src
[Container] 2024/11/04 16:54:50.815950 YAML location is /codebuild/readonly/buildspec.yml
[Container] 2024/11/04 16:54:50.819309 Setting HTTP client timeout to higher timeout for S3 source
[Container] 2024/11/04 16:54:50.819422 Processing environment variables
[Container] 2024/11/04 16:54:51.007034 No runtime version selected in buildspec.
[Container] 2024/11/04 16:54:51.059621 Moving to directory /codebuild/output/srcXXXXXXX/src
[Container] 2024/11/04 16:54:51.061258 Unable to initialize cache download: no paths specified to be cached
[Container] 2024/11/04 16:54:51.178746 Configuring ssm agent with target id: codebuild:XXXXXXX
[Container] 2024/11/04 16:54:51.179779 Successfully updated ssm agent configuration
[Container] 2024/11/04 16:54:51.180166 Registering with agent
[Container] 2024/11/04 16:54:51.216167 Phases found in YAML: 1
[Container] 2024/11/04 16:54:51.216237  BUILD: 11 commands
[Container] 2024/11/04 16:54:51.216572 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2024/11/04 16:54:51.216585 Phase context status code:  Message: 
[Container] 2024/11/04 16:54:51.285608 Entering phase INSTALL
[Container] 2024/11/04 16:54:51.319698 Phase complete: INSTALL State: SUCCEEDED
[Container] 2024/11/04 16:54:51.319714 Phase context status code:  Message: 
[Container] 2024/11/04 16:54:51.355941 Entering phase PRE_BUILD
[Container] 2024/11/04 16:54:51.361266 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2024/11/04 16:54:51.361281 Phase context status code:  Message: 
[Container] 2024/11/04 16:54:51.396937 Entering phase BUILD
[Container] 2024/11/04 16:54:51.433396 Running command npm install -g aws-cdk
added 1 package in 9s
[Container] 2024/11/04 16:55:06.075130 Running command export PIPELINE_NAME="$(node -pe '`${process.env.CODEBUILD_INITIATOR}`.split("/")[1]')"
[Container] 2024/11/04 16:55:06.110247 Running command payload="$(node -pe 'JSON.stringify({ "PipelineName": process.env.PIPELINE_NAME, "StageName": process.env.STAGE_NAME, "ActionName": process.env.ACTION_NAME })' )"
[Container] 2024/11/04 16:55:06.143489 Running command ARN=$CODEBUILD_BUILD_ARN
[Container] 2024/11/04 16:55:06.148583 Running command REGION="$(node -pe '`${process.env.ARN}`.split(":")[3]')"
[Container] 2024/11/04 16:55:06.181179 Running command ACCOUNT_ID="$(node -pe '`${process.env.ARN}`.split(":")[4]')"
[Container] 2024/11/04 16:55:06.214130 Running command PROJECT_NAME="$(node -pe '`${process.env.ARN}`.split(":")[5].split("/")[1]')"
[Container] 2024/11/04 16:55:06.247115 Running command PROJECT_ID="$(node -pe '`${process.env.ARN}`.split(":")[6]')"
[Container] 2024/11/04 16:55:06.279713 Running command export LINK="https://$REGION.console.aws.amazon.com/codesuite/codebuild/$ACCOUNT_ID/projects/$PROJECT_NAME/build/$PROJECT_NAME:$PROJECT_ID/?region=$REGION"
[Container] 2024/11/04 16:55:06.322494 Running command export PIPELINE_LINK="https://$REGION.console.aws.amazon.com/codesuite/codepipeline/pipelines/$PIPELINE_NAME/view?region=$REGION"
[Container] 2024/11/04 16:55:06.327848 Running command if cdk diff -a . --security-only --fail $STAGE_PATH/\*; then aws lambda invoke --function-name MyPipeline-PipelinePipelinesSecurit-tjblx3z6zu4g3 --invocation-type Event --cli-binary-format raw-in-base64-out --payload "$payload" lambda.out; export MESSAGE="No security-impacting changes detected."; else [ -z "${NOTIFICATION_ARN}" ] || aws sns publish --topic-arn $NOTIFICATION_ARN --subject "$NOTIFICATION_SUBJECT" --message "An upcoming change would broaden security changes in $PIPELINE_NAME.
Review and approve the changes in CodePipeline to proceed with the deployment.
Review the changes in CodeBuild:
$LINK
Approve the changes in CodePipeline (stage $STAGE_NAME, action $ACTION_NAME):
$PIPELINE_LINK"; export MESSAGE="Deployment would make security-impacting changes. Click the link below to inspect them, then click Approve if all changes are expected."; fi
current credentials could not be used to assume 'arn:aws:iam::012345678901:role/cdk-cgh650dgw-lookup-role-012345678901-eu-central-1', but are for the right account. Proceeding anyway.
Lookup role exists but was not assumed. Proceeding with default credentials.
Lookup role exists but was not assumed. Proceeding with default credentials.
start: Building ckMp6awVd6qv9dKdgTJE3DBZcy0CjwNdWryeX7HaR4C48kjQD8Q4CGp6N0EW3RJF:012345678901-eu-central-1
success: Built ckMp6awVd6qv9dKdgTJE3DBZcy0CjwNdWryeX7HaR4C48kjQD8Q4CGp6N0EW3RJF:012345678901-eu-central-1
start: Publishing ckMp6awVd6qv9dKdgTJE3DBZcy0CjwNdWryeX7HaR4C48kjQD8Q4CGp6N0EW3RJF:012345678901-eu-central-1
current credentials could not be used to assume 'arn:aws:iam::012345678901:role/cdk-cgh650dgw-file-publishing-role-012345678901-eu-central-1', but are for the right account. Proceeding anyway.
fail: Bucket named 'cdk-cgh650dgw-assets-012345678901-eu-central-1' exists, but we dont have access to it.
Could not create a change set, will base the diff on template differences (run again with -v to see the reason)
Stack Pipeline/Pipepline/MyStack (MyStack)
✨  Number of stacks with differences: 0

[Container] 2024/11/04 16:55:20.275824 Phase complete: BUILD State: SUCCEEDED
[Container] 2024/11/04 16:55:20.275843 Phase context status code:  Message: 
[Container] 2024/11/04 16:55:20.307476 Entering phase POST_BUILD
[Container] 2024/11/04 16:55:20.312519 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2024/11/04 16:55:20.312536 Phase context status code:  Message: 
[Container] 2024/11/04 16:55:20.355916 Set report auto-discover timeout to 5 seconds
[Container] 2024/11/04 16:55:20.355993 Expanding base directory path:  .
[Container] 2024/11/04 16:55:20.357593 Assembling file list
[Container] 2024/11/04 16:55:20.357608 Expanding .
[Container] 2024/11/04 16:55:20.359236 Expanding file paths for base directory .
[Container] 2024/11/04 16:55:20.359247 Assembling file list
[Container] 2024/11/04 16:55:20.359251 Expanding **/*
[Container] 2024/11/04 16:55:20.361148 No matching auto-discover report paths found
[Container] 2024/11/04 16:55:20.361165 Report auto-discover file discovery took 0.005249 seconds
[Container] 2024/11/04 16:55:20.361191 Phase complete: UPLOAD_ARTIFACTS State: SUCCEEDED
[Container] 2024/11/04 16:55:20.361197 Phase context status code:  Message: 

Current Behavior

Currently the CDK diff always fails which causes the Lambda to not be triggered. The Logs of the CodeBuild look like this:

[Container] 2024/12/10 12:42:52.550106 Running on CodeBuild On-demand
[Container] 2024/12/10 12:42:52.550117 Waiting for agent ping
[Container] 2024/12/10 12:42:52.651078 Waiting for DOWNLOAD_SOURCE
[Container] 2024/12/10 12:42:53.677653 Phase is DOWNLOAD_SOURCE
[Container] 2024/12/10 12:42:53.712676 CODEBUILD_SRC_DIR=/codebuild/output/srcXXXXX/src
[Container] 2024/12/10 12:42:53.713236 YAML location is /codebuild/readonly/buildspec.yml
[Container] 2024/12/10 12:42:53.717394 Setting HTTP client timeout to higher timeout for S3 source
[Container] 2024/12/10 12:42:53.717516 Processing environment variables
[Container] 2024/12/10 12:42:53.940972 No runtime version selected in buildspec.
[Container] 2024/12/10 12:42:53.959129 Moving to directory /codebuild/output/srcXXXX/src
[Container] 2024/12/10 12:42:53.960662 Unable to initialize cache download: no paths specified to be cached
[Container] 2024/12/10 12:42:54.060708 Configuring ssm agent with target id: codebuild:XXXXXX
[Container] 2024/12/10 12:42:54.061492 Successfully updated ssm agent configuration
[Container] 2024/12/10 12:42:54.061825 Registering with agent
[Container] 2024/12/10 12:42:54.095927 Phases found in YAML: 1
[Container] 2024/12/10 12:42:54.095942  BUILD: 11 commands
[Container] 2024/12/10 12:42:54.096324 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2024/12/10 12:42:54.096336 Phase context status code:  Message: 
[Container] 2024/12/10 12:42:54.163936 Entering phase INSTALL
[Container] 2024/12/10 12:42:54.203398 Phase complete: INSTALL State: SUCCEEDED
[Container] 2024/12/10 12:42:54.203414 Phase context status code:  Message: 
[Container] 2024/12/10 12:42:54.239046 Entering phase PRE_BUILD
[Container] 2024/12/10 12:42:54.244181 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2024/12/10 12:42:54.244194 Phase context status code:  Message: 
[Container] 2024/12/10 12:42:54.280869 Entering phase BUILD
[Container] 2024/12/10 12:42:54.315902 Running command npm install -g aws-cdk

added 1 package in 8s

[Container] 2024/12/10 12:43:07.462099 Running command export PIPELINE_NAME="$(node -pe '`${process.env.CODEBUILD_INITIATOR}`.split("/")[1]')"

[Container] 2024/12/10 12:43:07.494760 Running command payload="$(node -pe 'JSON.stringify({ "PipelineName": process.env.PIPELINE_NAME, "StageName": process.env.STAGE_NAME, "ActionName": process.env.ACTION_NAME })' )"

[Container] 2024/12/10 12:43:07.525573 Running command ARN=$CODEBUILD_BUILD_ARN

[Container] 2024/12/10 12:43:07.530026 Running command REGION="$(node -pe '`${process.env.ARN}`.split(":")[3]')"

[Container] 2024/12/10 12:43:07.560674 Running command ACCOUNT_ID="$(node -pe '`${process.env.ARN}`.split(":")[4]')"

[Container] 2024/12/10 12:43:07.592502 Running command PROJECT_NAME="$(node -pe '`${process.env.ARN}`.split(":")[5].split("/")[1]')"

[Container] 2024/12/10 12:43:07.630729 Running command PROJECT_ID="$(node -pe '`${process.env.ARN}`.split(":")[6]')"

[Container] 2024/12/10 12:43:07.662314 Running command export LINK="https://$REGION.console.aws.amazon.com/codesuite/codebuild/$ACCOUNT_ID/projects/$PROJECT_NAME/build/$PROJECT_NAME:$PROJECT_ID/?region=$REGION"

[Container] 2024/12/10 12:43:07.666499 Running command export PIPELINE_LINK="https://$REGION.console.aws.amazon.com/codesuite/codepipeline/pipelines/$PIPELINE_NAME/view?region=$REGION"

[Container] 2024/12/10 12:43:07.670348 Running command if cdk diff -a . --security-only --fail $STAGE_PATH/\*; then aws lambda invoke --function-name Pipeline-PipelinePipelinesSecurit-XXXXX --invocation-type Event --cli-binary-format raw-in-base64-out --payload "$payload" lambda.out; export MESSAGE="No security-impacting changes detected."; else [ -z "${NOTIFICATION_ARN}" ] || aws sns publish --topic-arn $NOTIFICATION_ARN --subject "$NOTIFICATION_SUBJECT" --message "An upcoming change would broaden security changes in $PIPELINE_NAME.
Review and approve the changes in CodePipeline to proceed with the deployment.

Review the changes in CodeBuild:

$LINK

Approve the changes in CodePipeline (stage $STAGE_NAME, action $ACTION_NAME):

$PIPELINE_LINK"; export MESSAGE="Deployment would make security-impacting changes. Click the link below to inspect them, then click Approve if all changes are expected."; fi

current credentials could not be used to assume 'arn:aws:iam::012345678901:role/cdk-cgh650dgw-lookup-role-012345678901-eu-central-1', but are for the right account. Proceeding anyway.
Lookup role arn:aws:iam::012345678901:role/cdk-cgh650dgw-lookup-role-012345678901-eu-central-1 was not assumed. Proceeding with default credentials.
User: arn:aws:sts::012345678901:assumed-role/Pipeline-PipelinePipelinesSecurity-XXXX/AWSCodeBuild-XXXXXX is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:eu-central-1:012345678901:stack/MyStack/* because no identity-based policy allows the cloudformation:DescribeStacks action

[Container] 2024/12/10 12:43:09.086719 Phase complete: BUILD State: SUCCEEDED
[Container] 2024/12/10 12:43:09.086737 Phase context status code:  Message: 
[Container] 2024/12/10 12:43:09.125476 Entering phase POST_BUILD
[Container] 2024/12/10 12:43:09.130289 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2024/12/10 12:43:09.130305 Phase context status code:  Message: 
[Container] 2024/12/10 12:43:09.178086 Set report auto-discover timeout to 5 seconds
[Container] 2024/12/10 12:43:09.178184 Expanding base directory path:  .
[Container] 2024/12/10 12:43:09.179824 Assembling file list
[Container] 2024/12/10 12:43:09.179838 Expanding .
[Container] 2024/12/10 12:43:09.181468 Expanding file paths for base directory .
[Container] 2024/12/10 12:43:09.181477 Assembling file list
[Container] 2024/12/10 12:43:09.181481 Expanding **/*
[Container] 2024/12/10 12:43:09.183695 Found 1 file(s)
[Container] 2024/12/10 12:43:09.183764 Report auto-discover file discovery took 0.005679 seconds
[Container] 2024/12/10 12:43:09.184395 Phase complete: UPLOAD_ARTIFACTS State:

It seems like creating it without the IAM Permissions based on template differences isnt working anymore. I checked via Cloud-Trail that there were no Changes to the IAM Permissions of the Role. Original this was deployed with package version aws-cdk-lib==2.151.0, later I updatet to aws-cdk-lib==2.163.0 and aws-cdk-lib==2.164.0. But there was no change

Reproduction Steps

Create a Pipeline with Manual Permission Check. Rough Example of what this looks like:

code_pipeline = pipelines.CodePipeline(
            self,
            "Pipeline",
            code_build_defaults=pipelines.CodeBuildOptions(
                build_environment=aws_codebuild.BuildEnvironment(
                    build_image=aws_codebuild.LinuxBuildImage.AMAZON_LINUX_2_5,
                    compute_type=aws_codebuild.ComputeType.SMALL,
                ),
            ),
            synth=pipelines.ShellStep(
                "Synth",
                input=pipelines.CodePipelineSource.connection(
                    repo_string=repo_string,
                    branch="master",
                    action_name="Source",
                    connection_arn=connection_arn,
                ),
                commands=[
                    "pip install -r requirements.txt",
                    "npm install -g aws-cdk",
                    "cdk synth",
                ],
            ),
        )

        the_stage = PipelineStage(self, id, account_var=account_var, env=env)
        code_pipeline.add_stage(
            stage=the_stage,
            pre=[
                pipelines.ConfirmPermissionsBroadening(
                    "PermissionCheck",
                    stage=the_stage,
                )
            ],
        )

Possible Solution

If I manually add these Permissions to the Role of CDKSecurityCheck and than it starts working:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "cloudformation:ListStacks",
                "cloudformation:DescribeStacks",
                "cloudformation:GetTemplate"
            ],
            "Resource": "*"
        }
    ]
}

But since the Pipeline are self Mutating this manual change just get its overwritten. I found #27605 (comment), but im not able to access Pipeline/PipelinesSecurityCheck/CDKSecurityCheck/Role in this way since the Construct doesnt expose the Node.

It would be great if somebody could show me how to do the same as done in #27605 (comment) but for the Construct, so I can apply my workarround. Or if the underlying bug (?) gets fixed.

Additional Information/Context

No response

CDK CLI Version

2.163.1 (build c14a1ff)

Framework Version

No response

Node.js Version

20

OS

Amazon Linux 2023

Language

Python

Language Version

3.9.20

Other information

No response

@ChristophrK ChristophrK added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Dec 11, 2024
@github-actions github-actions bot added the @aws-cdk/pipelines CDK Pipelines library label Dec 11, 2024
@ashishdhingra ashishdhingra self-assigned this Dec 13, 2024
@ashishdhingra ashishdhingra added p2 investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed needs-triage This issue or PR still needs to be triaged. labels Dec 13, 2024
@ashishdhingra
Copy link
Contributor

ashishdhingra commented Dec 13, 2024

@ChristophrK Good afternoon. I'm unsure if it was working for you in CDK version 2.151.0, since using the same version for code below (or version 2.140.0):

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as pipelines from 'aws-cdk-lib/pipelines';
import * as codebuild from 'aws-cdk-lib/aws-codebuild';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as snssubscriptions from 'aws-cdk-lib/aws-sns-subscriptions';
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';

export class CdktestStackNew extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const pipeline = new pipelines.CodePipeline(this, 'Pipeline', {
      codeBuildDefaults: {
        buildEnvironment: {
          buildImage: codebuild.LinuxBuildImage.AMAZON_LINUX_2_5,
          computeType: codebuild.ComputeType.SMALL
        }
      },
      synth: new pipelines.ShellStep('Synth', {
        input: pipelines.CodePipelineSource.connection('ashishdhingra/demorepo', 'main', {
          connectionArn:
            'arn:aws:codestar-connections:us-east-2:<<ACCOUNT-ID>>:connection/<<GUID>>',
        }),
        commands: ['npm ci', 'npm run build', 'npx cdk synth'],
      })
    });

    const topic = new sns.Topic(this, 'SecurityChangesTopic');
    topic.addSubscription(new snssubscriptions.EmailSubscription('<<someemail>>'));

    const stage = new MyApplicationStage(this, 'MyApplication');
    pipeline.addStage(stage, {
      pre: [
        new pipelines.ConfirmPermissionsBroadening('Check', {
          stage,
          notificationTopic: topic,
        }),
      ],
    });
  }
}

class MyApplicationStage extends cdk.Stage {
  constructor(scope: Construct, id: string, props?: cdk.StageProps) {
    super(scope, id, props);

    const dbStack = new DatabaseStack(this, 'Database');
  }
}

class DatabaseStack extends cdk.Stack {
  public readonly table: dynamodb.TableV2;

  constructor(scope: Construct, id: string) {
    super(scope, id);
    this.table = new dynamodb.TableV2(this, 'Table', {
      partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
    });
  }
}

const app = new cdk.App();

new CdktestStackNew(app, 'CdktestStackNew', {
  env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
});

also displays below message in log for Check.Check step:


[Container] 2024/12/13 22:48:39.094067 Running on CodeBuild On-demand
--
2 | [Container] 2024/12/13 22:48:39.094077 Waiting for agent ping
3 | [Container] 2024/12/13 22:48:39.194984 Waiting for DOWNLOAD_SOURCE
4 | [Container] 2024/12/13 22:48:40.477305 Phase is DOWNLOAD_SOURCE
5 | [Container] 2024/12/13 22:48:40.510989 CODEBUILD_SRC_DIR=/codebuild/output/src1648033198/src
6 | [Container] 2024/12/13 22:48:40.511563 YAML location is /codebuild/readonly/buildspec.yml
7 | [Container] 2024/12/13 22:48:40.514546 Setting HTTP client timeout to higher timeout for S3 source
8 | [Container] 2024/12/13 22:48:40.514830 Processing environment variables
9 | [Container] 2024/12/13 22:48:40.725780 No runtime version selected in buildspec.
10 | [Container] 2024/12/13 22:48:40.772415 Moving to directory /codebuild/output/src1648033198/src
11 | [Container] 2024/12/13 22:48:40.774901 Unable to initialize cache download: no paths specified to be cached
12 | [Container] 2024/12/13 22:48:40.857009 Configuring ssm agent with target id: codebuild:7a352ab2-2e84-4d80-93c2-6c0b15894c2c
13 | [Container] 2024/12/13 22:48:40.881270 Successfully updated ssm agent configuration
14 | [Container] 2024/12/13 22:48:40.881620 Registering with agent
15 | [Container] 2024/12/13 22:48:40.918112 Phases found in YAML: 1
16 | [Container] 2024/12/13 22:48:40.918132  BUILD: 11 commands
17 | [Container] 2024/12/13 22:48:40.918377 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
18 | [Container] 2024/12/13 22:48:40.918394 Phase context status code:  Message:
19 | [Container] 2024/12/13 22:48:40.974417 Entering phase INSTALL
20 | [Container] 2024/12/13 22:48:41.016334 Phase complete: INSTALL State: SUCCEEDED
21 | [Container] 2024/12/13 22:48:41.016346 Phase context status code:  Message:
22 | [Container] 2024/12/13 22:48:41.048688 Entering phase PRE_BUILD
23 | [Container] 2024/12/13 22:48:41.054923 Phase complete: PRE_BUILD State: SUCCEEDED
24 | [Container] 2024/12/13 22:48:41.054936 Phase context status code:  Message:
25 | [Container] 2024/12/13 22:48:41.085131 Entering phase BUILD
26 | [Container] 2024/12/13 22:48:41.116779 Running command npm install -g aws-cdk
27 |  

added 1 package in 9s
--
29 |  
30 | [Container] 2024/12/13 22:48:55.265852 Running command export PIPELINE_NAME="$(node -pe '`${process.env.CODEBUILD_INITIATOR}`.split("/")[1]')"
31 |  
32 | [Container] 2024/12/13 22:48:55.300261 Running command payload="$(node -pe 'JSON.stringify({ "PipelineName": process.env.PIPELINE_NAME, "StageName": process.env.STAGE_NAME, "ActionName": process.env.ACTION_NAME })' )"
33 |  
34 | [Container] 2024/12/13 22:48:55.333814 Running command ARN=$CODEBUILD_BUILD_ARN
35 |  
36 | [Container] 2024/12/13 22:48:55.337649 Running command REGION="$(node -pe '`${process.env.ARN}`.split(":")[3]')"
37 |  
38 | [Container] 2024/12/13 22:48:55.371083 Running command ACCOUNT_ID="$(node -pe '`${process.env.ARN}`.split(":")[4]')"
39 |  
40 | [Container] 2024/12/13 22:48:55.404686 Running command PROJECT_NAME="$(node -pe '`${process.env.ARN}`.split(":")[5].split("/")[1]')"
41 |  
42 | [Container] 2024/12/13 22:48:55.442517 Running command PROJECT_ID="$(node -pe '`${process.env.ARN}`.split(":")[6]')"
43 |  
44 | [Container] 2024/12/13 22:48:55.482989 Running command export LINK="https://$REGION.console.aws.amazon.com/codesuite/codebuild/$ACCOUNT_ID/projects/$PROJECT_NAME/build/$PROJECT_NAME:$PROJECT_ID/?region=$REGION"
45 |  
46 | [Container] 2024/12/13 22:48:55.487491 Running command export PIPELINE_LINK="https://$REGION.console.aws.amazon.com/codesuite/codepipeline/pipelines/$PIPELINE_NAME/view?region=$REGION"
47 |  
48 | [Container] 2024/12/13 22:48:55.494473 Running command if cdk diff -a . --security-only --fail $STAGE_PATH/\*; then aws lambda invoke --function-name CdktestStackNew-PipelinePipelinesSecurityCheckCDKP-V3gvFD65aZy9 --invocation-type Event --cli-binary-format raw-in-base64-out --payload "$payload" lambda.out; export MESSAGE="No security-impacting changes detected."; else [ -z "${NOTIFICATION_ARN}" ] \|\| aws sns publish --topic-arn $NOTIFICATION_ARN --subject "$NOTIFICATION_SUBJECT" --message "An upcoming change would broaden security changes in $PIPELINE_NAME.
49 | Review and approve the changes in CodePipeline to proceed with the deployment.
50 |  
51 | Review the changes in CodeBuild:
52 |  
53 | $LINK
54 |  
55 | Approve the changes in CodePipeline (stage $STAGE_NAME, action $ACTION_NAME):
56 |  
57 | $PIPELINE_LINK"; export MESSAGE="Deployment would make security-impacting changes. Click the link below to inspect them, then click Approve if all changes are expected."; fi
58 | current credentials could not be used to assume 'arn:aws:iam::139480602983:role/cdk-hnb659fds-lookup-role-139480602983-us-east-2', but are for the right account. Proceeding anyway.
59 | Lookup role arn:aws:iam::<<ACCOUNT_ID>>:role/cdk-hnb659fds-lookup-role-<<ACCOUNT_ID>>-us-east-2 was not assumed. Proceeding with default credentials.
60 | User: arn:aws:sts::<<ACCOUNT_ID>>:assumed-role/CdktestStackNew-PipelinePipelinesSecurityCheckCDKSe-WJDJVVk7zBJq/AWSCodeBuild-7a352ab2-2e84-4d80-93c2-6c0b15894c2c is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:us-east-2:<<ACCOUNT_ID>>:stack/MyApplication-Database/* because no identity-based policy allows the cloudformation:DescribeStacks action
61 | {
62 | "MessageId": "6e03d452-c6c8-5d12-ae00-8b847aee29fc"
63 | }
64 |  
65 | [Container] 2024/12/13 22:49:10.420284 Phase complete: BUILD State: SUCCEEDED
66 | [Container] 2024/12/13 22:49:10.420335 Phase context status code:  Message:
67 | [Container] 2024/12/13 22:49:10.454750 Entering phase POST_BUILD
68 | [Container] 2024/12/13 22:49:10.459884 Phase complete: POST_BUILD State: SUCCEEDED
69 | [Container] 2024/12/13 22:49:10.459900 Phase context status code:  Message:
70 | [Container] 2024/12/13 22:49:10.505909 Set report auto-discover timeout to 5 seconds
71 | [Container] 2024/12/13 22:49:10.505996 Expanding base directory path:  .

The most likely reason is that in Pipeline, command npm install -g aws-cdk is executed, which would install the latest version and the previous self update pipeline step might have modified role permissions.

Are you able to share which role did you modify to add missing permissions after which cdk diff and auto-approval started working?

I cannot certainly rule out the issue here since cdk diff is perhaps failing.

Thanks,
Ashish

@ashishdhingra ashishdhingra added effort/medium Medium work item – several days of effort p1 p2 and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. p2 p1 labels Dec 13, 2024
@ashishdhingra ashishdhingra removed their assignment Dec 19, 2024
@ChristophrK
Copy link
Author

Hi @ashishdhingra, sorry for the late reply. I agree it seems like that its not really dependent on the version and the only common thing in our pipelines across usecases is that they started braking after 4th of November.

The way to "make it work" with adjusting the IAM Permissions is the following:

  1. Start Pipeline Execution as normally
  2. Decline the Deployment via the Manual Confirmation (PermissionCheck.Confirm), this puts the Pipeline in a failed state
  3. Go to IAM and adjust the Permissions (Addition of the Policies mentioned under Possible Solution) of the following Role:
    • arn:aws:sts::012345678901:assumed-role/Pipeline-PipelinePipelinesSecurity-XXXX/AWSCodeBuild-XXXXXX (my example)
    • CdktestStackNew-PipelinePipelinesSecurityCheckCDKSe-WJDJVVk7zBJq/AWSCodeBuild-7a352ab2-2e84-4d80-93c2-6c0b15894c2c (your example)
  4. Select Retry Stage on the Failed Pipeline Execution this will restart the failed Stage of the Pipeline and run with the adjusted IAM Permissions.

The Logs than look like this:

[Container] 2024/12/11 14:13:19.143396 Running on CodeBuild On-demand
[Container] 2024/12/11 14:13:19.143408 Waiting for agent ping
[Container] 2024/12/11 14:13:19.244372 Waiting for DOWNLOAD_SOURCE
[Container] 2024/12/11 14:13:20.375180 Phase is DOWNLOAD_SOURCE
[Container] 2024/12/11 14:13:20.419142 CODEBUILD_SRC_DIR=/codebuild/output/srcXXXXX/src
[Container] 2024/12/11 14:13:20.419647 YAML location is /codebuild/readonly/buildspec.yml
[Container] 2024/12/11 14:13:20.421901 Setting HTTP client timeout to higher timeout for S3 source
[Container] 2024/12/11 14:13:20.422062 Processing environment variables
[Container] 2024/12/11 14:13:20.617669 No runtime version selected in buildspec.
[Container] 2024/12/11 14:13:20.632255 Moving to directory /codebuild/output/srcXXXXX/src
[Container] 2024/12/11 14:13:20.633774 Unable to initialize cache download: no paths specified to be cached
[Container] 2024/12/11 14:13:20.719291 Configuring ssm agent with target id: codebuild:XXXXX
[Container] 2024/12/11 14:13:20.720003 Successfully updated ssm agent configuration
[Container] 2024/12/11 14:13:20.720290 Registering with agent
[Container] 2024/12/11 14:13:20.755857 Phases found in YAML: 1
[Container] 2024/12/11 14:13:20.755873  BUILD: 11 commands
[Container] 2024/12/11 14:13:20.756234 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2024/12/11 14:13:20.756246 Phase context status code:  Message: 
[Container] 2024/12/11 14:13:20.820609 Entering phase INSTALL
[Container] 2024/12/11 14:13:20.864567 Phase complete: INSTALL State: SUCCEEDED
[Container] 2024/12/11 14:13:20.864581 Phase context status code:  Message: 
[Container] 2024/12/11 14:13:20.897716 Entering phase PRE_BUILD
[Container] 2024/12/11 14:13:20.902195 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2024/12/11 14:13:20.902209 Phase context status code:  Message: 
[Container] 2024/12/11 14:13:20.937448 Entering phase BUILD
[Container] 2024/12/11 14:13:20.973568 Running command npm install -g aws-cdk
added 1 package in 7s
[Container] 2024/12/11 14:13:32.771881 Running command export PIPELINE_NAME="$(node -pe '`${process.env.CODEBUILD_INITIATOR}`.split("/")[1]')"
[Container] 2024/12/11 14:13:32.803727 Running command payload="$(node -pe 'JSON.stringify({ "PipelineName": process.env.PIPELINE_NAME, "StageName": process.env.STAGE_NAME, "ActionName": process.env.ACTION_NAME })' )"
[Container] 2024/12/11 14:13:32.833570 Running command ARN=$CODEBUILD_BUILD_ARN
[Container] 2024/12/11 14:13:32.837479 Running command REGION="$(node -pe '`${process.env.ARN}`.split(":")[3]')"
[Container] 2024/12/11 14:13:32.866707 Running command ACCOUNT_ID="$(node -pe '`${process.env.ARN}`.split(":")[4]')"
[Container] 2024/12/11 14:13:32.895890 Running command PROJECT_NAME="$(node -pe '`${process.env.ARN}`.split(":")[5].split("/")[1]')"
[Container] 2024/12/11 14:13:32.925110 Running command PROJECT_ID="$(node -pe '`${process.env.ARN}`.split(":")[6]')"
[Container] 2024/12/11 14:13:32.954128 Running command export LINK="https://$REGION.console.aws.amazon.com/codesuite/codebuild/$ACCOUNT_ID/projects/$PROJECT_NAME/build/$PROJECT_NAME:$PROJECT_ID/?region=$REGION"
[Container] 2024/12/11 14:13:32.957721 Running command export PIPELINE_LINK="https://$REGION.console.aws.amazon.com/codesuite/codepipeline/pipelines/$PIPELINE_NAME/view?region=$REGION"
[Container] 2024/12/11 14:13:32.987721 Running command if cdk diff -a . --security-only --fail $STAGE_PATH/\*; then aws lambda invoke --function-name MyPipeline-PipelinePipelinesSecurit-tjblx3z6zu4g3 --invocation-type Event --cli-binary-format raw-in-base64-out --payload "$payload" lambda.out; export MESSAGE="No security-impacting changes detected."; else [ -z "${NOTIFICATION_ARN}" ] || aws sns publish --topic-arn $NOTIFICATION_ARN --subject "$NOTIFICATION_SUBJECT" --message "An upcoming change would broaden security changes in $PIPELINE_NAME.
Review and approve the changes in CodePipeline to proceed with the deployment.
Review the changes in CodeBuild:
$LINK
Approve the changes in CodePipeline (stage $STAGE_NAME, action $ACTION_NAME):
$PIPELINE_LINK"; export MESSAGE="Deployment would make security-impacting changes. Click the link below to inspect them, then click Approve if all changes are expected."; fi
current credentials could not be used to assume 'arn:aws:iam::012345678901:role/cdk-cgh650dgw-lookup-role-012345678901-eu-central-1', but are for the right account. Proceeding anyway.
Lookup role arn:aws:iam::012345678901:role/cdk-cgh650dgw-lookup-role-012345678901-eu-central-1 was not assumed. Proceeding with default credentials.
Lookup role arn:aws:iam::012345678901:role/cdk-cgh650dgw-lookup-role-012345678901-eu-central-1 was not assumed. Proceeding with default credentials.
start: Building ckMp6awVd6qv9dKdgTJE3DBZcy0CjwNdWryeX7HaR4C48kjQD8Q4CGp6N0EW3RJF:012345678901-eu-central-1
success: Built ckMp6awVd6qv9dKdgTJE3DBZcy0CjwNdWryeX7HaR4C48kjQD8Q4CGp6N0EW3RJF:012345678901-eu-central-1
start: Publishing ckMp6awVd6qv9dKdgTJE3DBZcy0CjwNdWryeX7HaR4C48kjQD8Q4CGp6N0EW3RJF:012345678901-eu-central-1
current credentials could not be used to assume 'arn:aws:iam::012345678901:role/cdk-cgh650dgw-file-publishing-role-012345678901-eu-central-1', but are for the right account. Proceeding anyway.
fail: Bucket named 'cdk-cgh650dgw-assets-012345678901-eu-central-1' exists, but we dont have access to it.
Could not create a change set, will base the diff on template differences (run again with -v to see the reason)
✨  Number of stacks with differences: 0
{
    "StatusCode": 202
}
[Container] 2024/12/11 14:13:45.475047 Phase complete: BUILD State: SUCCEEDED
[Container] 2024/12/11 14:13:45.475074 Phase context status code:  Message: 
[Container] 2024/12/11 14:13:45.508103 Entering phase POST_BUILD
[Container] 2024/12/11 14:13:45.512628 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2024/12/11 14:13:45.512650 Phase context status code:  Message: 
[Container] 2024/12/11 14:13:45.554836 Set report auto-discover timeout to 5 seconds
[Container] 2024/12/11 14:13:45.554881 Expanding base directory path:  .
[Container] 2024/12/11 14:13:45.556345 Assembling file list
[Container] 2024/12/11 14:13:45.556357 Expanding .
[Container] 2024/12/11 14:13:45.557904 Expanding file paths for base directory .
[Container] 2024/12/11 14:13:45.557918 Assembling file list
[Container] 2024/12/11 14:13:45.557922 Expanding **/*
[Container] 2024/12/11 14:13:45.559771 No matching auto-discover report paths found
[Container] 2024/12/11 14:13:45.559855 Report auto-discover file discovery took 0.005019 seconds
[Container] 2024/12/11 14:13:45.559916 Phase complete: UPLOAD_ARTIFACTS State: SUCCEEDED
[Container] 2024/12/11 14:13:45.559925 Phase context status code:  Message: 

And in the Manual Approval Step (PermissionCheck.Confirm) I can see the expected Auto Approval Message:

Status
Approved
Comments about this action
No security-impacting changes detected.

Due to the self update the fix that changes the Role gets removed with the next Invocation of the Pipeline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/pipelines CDK Pipelines library bug This issue is a bug. effort/medium Medium work item – several days of effort p2
Projects
None yet
Development

No branches or pull requests

2 participants