select_object_content gives access denied when SSECustomerAlgorithm is enabled

[HassanAhamed24]

Hi,

i am having couple of objcts in s3 out of which some of them are encrypted using customerKey and some are not.

When i use select_object_content api on normal files which are not encrypted, it works like a charm, however when i use the api on encrypted file, i get access denied. I was under the impression that my ekey would have been wrong,
To counter that i used get_object api on encrypted file with the same SSECustomerAlgorithm and SSECustomerKey, and it worked fine. Also to be on safe side used AWS-SDK in javascript and i was able to successfully run selectObjectContent api on encrypted file.

Not sure if this there are some additional paramaters that i am missing here. but according to the documentation i am using all the required paramaters.

tested it on boto3 - 1.9.18 and 1.21.21

code is as follows-

response1 = s3_client.select_object_content(
        Expression= query, 
        ExpressionType= 'SQL', 
        InputSerialization= { 
            'JSON': {
                'Type': 'DOCUMENT'
            },
            'CompressionType': "GZIP"
        },
        Key= myBucketKey, 
        OutputSerialization= { 
            'JSON': {
                'RecordDelimiter': ','
            }
        },
        SSECustomerAlgorithm= 'AES256',
        SSECustomerKey= ekey,
        Bucket= myBucketName,
     )

running above code gives me error as follows-

raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the SelectObjectContent operation: Access Denied

when the api works fine on files that are not encrypted, it should adhere to same behaviour on encrypted files when passed SSECustomerAlgorithm and SSECustomerKey paramaters. Let me know if my understanding is wrong.

[tim-finnigan]

Hi @HassanAhamed24 here is the select_object_content documentation for reference: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.select_object_content

If you have the s3:GetObject permission assigned then you should have access. There are also some requirements in that documentation for querying objects that are protected with server-side encryption. Can you verify that those requirements are met?

[HassanAhamed24]

Hi @tim-finnigan ,
i have uploaded a file using boto as follows-

s3_client = boto3.client("s3")
s3_client.head_bucket(Bucket=bucket)
steam = json.dumps({some json data})
stream = gzip.compress(bytes(steam, 'utf-8'))
s3_client.put_object(Key="1", Bucket=bucket, Body=stream, ContentType='application/x-gzip',SSECustomerAlgorithm='AES256',SSECustomerKey=eKey)

and when i try to perform select_object_content on above uploaded encrypted file, it gives access denied.
Note the ekey is 32 length long base64 encoded string

can anyone confirm if this is a bug, as i tried this on multiple buckets and i am getting same error.
Further, get_object api works on above encrypted file.

[tim-finnigan]

Hi @HassanAhamed24 can I ask how you generated the SSE key? Here is documentation on that: https://boto3.amazonaws.com/v1/documentation/api/1.9.46/reference/services/s3.html#uploading-downloading-files-using-sse-customer-keys

[HassanAhamed24]

Hi @tim-finnigan , SSE keys are generated as per the standards mentioned in the doc.
I dont think that should be a problem as i am able to successfully call get_object api with the same key.

Please let me know your thoughts.

[tim-finnigan]

Hi @HassanAhamed24 I'm still looking into this issue and plan to bring this up for further discussion with the team. I will let you know when I have more information. If you have anything new to report please let us know. Thanks for your patience.

[HassanAhamed24]

Thanks @tim-finnigan , i would like to update you that i am following up with aws support as well, so will update once i get to some resolution if any.
Its kind of weird as aws cli too is giving the error. and intrestingly its working in asw-sdk of nodejs.
So its pretty weird.

Will keep you updated.

Thanks

[nateprewitt]

Hi @HassanAhamed24, this appears to be an issue with handler registration specifically for SSE in S3. You can see here we have a customization that runs on certain operations such as PutObject to auto-encode your encryption key for you. The key MUST be a base64-encoded string as noted in the API docs for S3. The customization is a courtesy function to reduce complexity but doesn't appear to be applied consistently.

We're going to need to do some testing, but we should be able to ship this in a future version of Boto3 to add this to the subset of APIs auto-encoding the SSE key.

[HassanAhamed24]

Hey @nateprewitt thanks for the help.

I was able to solve it by performing few hacks as per the customization that you have pointed out.

But i hope this gets resolved in future release, Also the same has to be done for AWS CLI.

Regards
Hassan

[HassanAhamed24]

For anyone looking for the hack, please do as follows-

`key_as_bytes = ekey.encode('utf-8')

key_b64_encoded = base64.b64encode(key_as_bytes).decode('utf-8')`

now pass the key_b64_encoded in SSECustomerKey. This should work for now.
@nateprewitt please let me know if this should be good for now.

[nateprewitt]

Yes, that would be the correct way to pass the key without the helper function. It will be forwards compatible after the fix as well.