Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cli: permissions snapshot integration testing #32088

Open
2 tasks
Tracked by #32171
iliapolo opened this issue Nov 11, 2024 · 4 comments · May be fixed by #33066
Open
2 tasks
Tracked by #32171

cli: permissions snapshot integration testing #32088

iliapolo opened this issue Nov 11, 2024 · 4 comments · May be fixed by #33066
Assignees
Labels
effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1 package/tools Related to AWS CDK Tools or CLI

Comments

@iliapolo
Copy link
Contributor

iliapolo commented Nov 11, 2024

Describe the feature

Every CLI integration test will record a file containing all IAM roles it assumed and all IAM actions it performed during the test execution.

This file will serve as a snapshot, which will fail the test in case it contains changes.

Use Case

It is not uncommon for organizations to have very strict IAM policy requirements. So strict in fact that the roles used by the CLI are configured to only allow the actions and principals that are currently required for it to operate.

In such cases, any changes to which roles are assumed, or which actions are performed will break customer deployments. While the CDK doesn't make guarantees it will never do so (it would be close to impossible to add new features if we did), it would still be advantageous to catch those occurrences and handle them with more care.

Examples: #29483, #32219

Proposed Solution

One option to catch these types of changes is configure our own policies to be as strict as possible when we run integration tests. This however can be cumbersome to operate and maintain. Instead, we would like an automatic mechanism that doesn't require infrastructure changes.

Research required, but there must be a way to hook into the aws sdk and intercept service calls, to form a snapshot of the roles and actions used during the test.

In case a test fails because the snapshot changed, we will:

  1. Determine if the change is expected. If it isn't - great, we caught a bug.
  2. If it is, we bypass the validation and proceed. In this case, the snapshot file can serve as data that we can communicate to our customers, notifying them about what exactly changed so they can prepare for it.

The aggregation of all snapshots from all tests can also serve as a public document, detailing which actions are required for each command.

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

CDK version used

ALL

Environment details (OS name and version, etc.)

ALL

@iliapolo iliapolo added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Nov 11, 2024
@github-actions github-actions bot added the package/tools Related to AWS CDK Tools or CLI label Nov 11, 2024
@pahud pahud added p2 effort/medium Medium work item – several days of effort p1 and removed needs-triage This issue or PR still needs to be triaged. p2 labels Nov 11, 2024
@iankhou iankhou self-assigned this Jan 17, 2025
@iankhou
Copy link
Contributor

iankhou commented Jan 17, 2025

I propose the following:

First, we create an iam-tracking interface that does the following:

  • Begins tracking IAM usage through a middleware
  • Stops tracking IAM usage and logs to a file (snapshot)

This interface will be implemented for each cli command.

We will need two different behaviors:

  1. Baseline - run the iam-tracking interface for the first time, or establish a new baseline, and write/overwrite a baseline file
  2. Compare - run the iam-tracking interface and compare the output file to the baseline file

Baseline should only be run with contributor/maintainer intervention (pass a flag like --set-iam-baseline), while compare should run on every PR.

@iankhou
Copy link
Contributor

iankhou commented Jan 17, 2025

The IAMClient used by integration tests can be wrapped with an HOC that tracks its activity. We should then format this and store it into a file for each integration test that uses it.

When we baseline, this should either create a new file or overwrite/version up the baseline file. We can pass a flag to the integ runner like --set-iam-baseline.

When we run integ tests normally, we should expect there to be a baseline file, or fail the test. If there is a mismatch between the newly generated IAM activity and the existing baseline, we should investigate and decide whether it's an acceptable change.

Hoping to discuss this on Tuesday with @iliapolo.

@iankhou
Copy link
Contributor

iankhou commented Jan 22, 2025

Met with @iliapolo.

Working POC pushed to a branch: https://github.com/aws/aws-cdk/tree/iankhou-cli-iam-tracking
Draft PR: #33066

In this implementation, I created a middleware that logs every SDK call for an SDK client (S3, CFn, etc.), then uses an STSClient to get the role used (SDK calls only pass access key ID and secret key, not the role itself). Then the service, operation, and role are written to a file.

This middleware is applied to every SDK client (not just the IAMClient as mentioned above). In my POC branch, I saved a file for the cdk migrate integ test.

@iankhou iankhou closed this as completed Jan 22, 2025
Copy link

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 22, 2025
@iankhou iankhou reopened this Jan 22, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1 package/tools Related to AWS CDK Tools or CLI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants