Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fulfillment Lambda fails to update version/alias when Embedding settings change on stack update #566

Closed
ihmaws opened this issue Mar 14, 2023 · 1 comment
Labels

Comments

@ihmaws
Copy link
Member

ihmaws commented Mar 14, 2023

Describe the bug
Fulfillment Lambda fails to update version/alias when Embedding settings change on stack update (credit to @rstrahan on reporting this bug)

To Reproduce

  1. Create new stack from here: https://solutions-reference.s3.amazonaws.com/qnabot-on-aws/v5.3.0/qnabot-on-aws-main.template
  2. Set parameter EmbeddingsApi to SageMaker
  3. Deploy stack
  4. Once complete, import questions, and run query through client (should be successful)
  5. Update stack and set EmbeddingsApi to Disabled
  6. Once complete, run query again through client (should be successful)
  7. Update stack and set parameter EmbeddingsApi to SageMaker
  8. Once complete, run query again through client (this time should fail)

Expected behavior
When enabling, disabling, and then reenabling embeddings, the client queries should still work.

Please complete the following information about the solution:

  • [v5.3.0] Version: [e.g. v0.0.1]
  • [us-east-1] Region: [e.g. us-east-1]
  • [No] Was the solution modified from the version published on this repository?
  • [N/A] If the answer to the previous question was yes, are the changes available on GitHub?
  • [N/A] Have you checked your service quotas for the sevices this solution uses?
  • [Yes] Were there any errors in the CloudWatch Logs?

{
"errorType": "AccessDeniedException",
"errorMessage": "User: arn:aws:sts::XXXXXX:assumed-role/v530-qnabot-FulfillmentLambdaRole/v530-qnabot-FulfillmentLambda is not authorized to perform: sagemaker:InvokeEndpoint on resource: arn:aws:sagemaker:us-east-1:XXXXXX:endpoint/qnabotsmprovisionedembeddingendpoint because no identity-based policy allows the sagemaker:InvokeEndpoint action",
"code": "AccessDeniedException",
...
}

Screenshots
Lambda Console - new lambda version has not been created, despite 2 lambda updates:
image

Empty response in client due to lambda failure:
image

Additional context
Temporary workaround is to manually point the live alias to $LATEST

@ihmaws ihmaws added the bug label Mar 14, 2023
tabdunabi added a commit that referenced this issue Mar 15, 2023
### Fixed

- Bug causing bot Fulfillment to fail on embeddings updates (#566)

### Updated

- VPC documentation update (SageMaker Serverless is not supported within a VPC)
- Security patches for npm and pip packages
@ihmaws
Copy link
Member Author

ihmaws commented Mar 15, 2023

Fixed as of v5.3.1

@ihmaws ihmaws closed this as completed Mar 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant