S3Boto3Storage raises a misleading exception on bucket creation

[ainterr]

S3Boto3Storage assumes any HTTP response besides a 301 or a 404 indicates that the bucket could not be created and AWS_AUTO_CREATE_BUCKET is False and raises an exception saying this. However, if AWS_S3_ENDPOINT_URL is set to point to a custom S3 host that happens to return some other HTTP error code, this exception is misleading.

Example (using Minio S3 instance running at http://localhost:9000 with no buckets initially configured):

settings.py

...
AWS_ACCESS_KEY_ID = '****'
AWS_SECRET_ACCESS_KEY = '****'
AWS_STORAGE_BUCKET_NAME = 'bucket'

AWS_S3_ENDPOINT_URL = 'http://localhost:9000'

AWS_AUTO_CREATE_BUCKET = True
AWS_S3_FILE_OVERWRITE = True
AWS_S3_USE_SSL = False

STATIC_URL = '%s/minio/%s/' % (AWS_S3_ENDPOINT_URL, AWS_STORAGE_BUCKET_NAME)
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
...

models.py

from django.db import models

class Document(models.Model):
    upload = models.FileField(upload_to='documents')

error

(env) ainterr@dev ~/g/test> python manage.py shell
Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from app import models
>>> from django.core.files import File
>>> d = models.Document()
>>> d.upload.save('test.txt', File(open('test.txt')))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/django/db/models/fields/files.py", line 95, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/django/core/files/storage.py", line 54, in save
    return self._save(name, content)
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 436, in _save
    obj = self.bucket.Object(encoded_name)
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 279, in bucket
    self._bucket = self._get_or_create_bucket(self.bucket_name)
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 353, in _get_or_create_bucket
    "``True``." % name)
ImproperlyConfigured: Bucket bucket does not exist. Buckets can be automatically created by setting AWS_AUTO_CREATE_BUCKET to ``True``.
>>> from django.conf import settings
>>> settings.AWS_AUTO_CREATE_BUCKET
True

further

>>> r.upload.storage.auto_create_bucket
True
>>> s = r.upload.storage
>>> bucket = s.connection.Bucket(s.bucket_name)
>>> bucket.meta.client.head_bucket(Bucket=s.bucket_name)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (502) when calling the HeadBucket operation (reached max retries: 4): Bad Gateway

In this case a bad proxy configuration caused the Minio instance at http://localhost:9000 to return a 502 but I had to dig through the django-storages source code to make sure I wasn't going crazy and actually had set AWS_AUTO_CREATE_BUCKET to True.

[jschneier]

Yep, it does not make much sense. While you’re here: can you expand a bit on why you are using AWS_AUTO_CREATE_BUCKET? I know it’s convenient but given the additional permissions required I’m considering deprecating and removing or further discouraging it.

…

On Aug 31, 2018, at 2:14 PM, Alex Interrante-Grant ***@***.***> wrote: S3Boto3Storage assumes any HTTP response besides a 301 or a 404 indicates that the bucket could not be created and AWS_AUTO_CREATE_BUCKET is False and raises an exception saying this. However, if AWS_S3_ENDPOINT_URL is set to point to a custom S3 host that happens to return some other HTTP error code, this exception is misleading. Example (using Minio S3 instance running at http://localhost:9000 with no buckets initially configured): settings.py ... AWS_ACCESS_KEY_ID = '****' AWS_SECRET_ACCESS_KEY = '****' AWS_STORAGE_BUCKET_NAME = 'bucket' AWS_S3_ENDPOINT_URL = 'http://localhost:9000' AWS_AUTO_CREATE_BUCKET = True AWS_S3_FILE_OVERWRITE = True AWS_S3_USE_SSL = False STATIC_URL = '%s/minio/%s/' % (AWS_S3_ENDPOINT_URL, AWS_STORAGE_BUCKET_NAME) STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage' DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage' ... models.py from django.db import models class Document(models.Model): upload = models.FileField(upload_to='documents') error (env) ***@***.*** ~/g/test> python manage.py shell Python 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609] on linux2 Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> from app import models >>> from django.core.files import File >>> d = models.Document() >>> d.upload.save('test.txt', File(open('test.txt'))) Traceback (most recent call last): File "<console>", line 1, in <module> File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/django/db/models/fields/files.py", line 95, in save self.name = self.storage.save(name, content, max_length=self.field.max_length) File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/django/core/files/storage.py", line 54, in save return self._save(name, content) File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 436, in _save obj = self.bucket.Object(encoded_name) File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 279, in bucket self._bucket = self._get_or_create_bucket(self.bucket_name) File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 353, in _get_or_create_bucket "``True``." % name) ImproperlyConfigured: Bucket bucket does not exist. Buckets can be automatically created by setting AWS_AUTO_CREATE_BUCKET to ``True``. >>> from django.conf import settings >>> settings.AWS_AUTO_CREATE_BUCKET True further >>> r.upload.storage.auto_create_bucket True >>> s = r.upload.storage >>> bucket = s.connection.Bucket(s.bucket_name) >>> bucket.meta.client.head_bucket(Bucket=s.bucket_name) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call return self._make_api_call(operation_name, kwargs) File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (502) when calling the HeadBucket operation (reached max retries: 4): Bad Gateway In this case a bad proxy configuration caused the Minio instance at http://localhost:9000 to return a 502 but I had to dig through the django-storages source code to make sure I wasn't going crazy and actually had set AWS_AUTO_CREATE_BUCKET to True. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#574>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACJB2Jx1S_7yf15rBLHqbyMSe2g5DeIAks5uWXzvgaJpZM4WVpsU>.

[ainterr]

Purely convenience in our case. Though it looks like to me like a 502 from the configured S3 endpoint would raise this exception regardless.

[jschneier]

Right. When the Boto3 backend was written a lot of language was taken directly from the older Boto backend which handles auto-creation slightly differently. Since they are different the language from the exception should have been changed.

…

On Aug 31, 2018, at 2:58 PM, Alex Interrante-Grant ***@***.***> wrote: Purely convenience in our case. Though it looks like to me like a 502 from the configured S3 endpoint would raise this exception regardless. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#574 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACJB2PcFpgH8j2THw1v6cIsZVD1nPTFvks5uWYdrgaJpZM4WVpsU>.