Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write mode option for Dropbox #873

Merged
merged 12 commits into from
Apr 18, 2020
1 change: 1 addition & 0 deletions AUTHORS
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ By order of apparition, thanks:
* Shaung Cheng (S3 docs)
* Andrew Perry (Bug fixes in SFTPStorage)
* Manuel Kaufmann (humitos)
* Taras Petriichuk (Dropbox overwrite option)
* Zoe Liao (S3 docs)


Expand Down
8 changes: 8 additions & 0 deletions docs/backends/dropbox.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,13 @@ To use DropBoxStorage set::
Timeout in seconds for making requests to the API. If ``None``, the client will wait forever.
The default is ``100`` seconds which is the current default in the official SDK.

``DROPBOX_FILE_OVERWRITE`` (optional)
petriichuk marked this conversation as resolved.
Show resolved Hide resolved
Overwrite an existing file when it has the same name as the file being uploaded.
Otherwise, rename it. Default is ``False``

``DROPBOX_WRITE_MODE`` (optional)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you default this to whatever the current default is (not overwrite). Don't want to break backwards compat. Can you also link out to the Dropbox docs for this property.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed to 'add'

Allow to set Dropbox WriteMode strategy.
Default is ``overwrite``

.. _`tutorial`: https://www.dropbox.com/developers/documentation/python#tutorial
.. _`Dropbox SDK for Python`: https://www.dropbox.com/developers/documentation/python#tutorial
39 changes: 34 additions & 5 deletions storages/backends/dropbox.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

from __future__ import absolute_import

import posixpath
from io import BytesIO
from shutil import copyfileobj
from tempfile import SpooledTemporaryFile
Expand All @@ -21,9 +22,11 @@
from django.utils.deconstruct import deconstructible
from dropbox import Dropbox
from dropbox.exceptions import ApiError
from dropbox.files import CommitInfo, FolderMetadata, UploadSessionCursor
from dropbox.files import (
CommitInfo, FolderMetadata, UploadSessionCursor, WriteMode,
)

from storages.utils import setting
from storages.utils import get_available_overwrite_name, setting

_DEFAULT_TIMEOUT = 100

Expand Down Expand Up @@ -69,15 +72,20 @@ class DropBoxStorage(Storage):
location = setting('DROPBOX_ROOT_PATH', '/')
oauth2_access_token = setting('DROPBOX_OAUTH2_TOKEN')
timeout = setting('DROPBOX_TIMEOUT', _DEFAULT_TIMEOUT)
file_overwrite = setting('DROPBOX_FILE_OVERWRITE', False)
write_mode = setting('DROPBOX_WRITE_MODE', 'overwrite')

CHUNK_SIZE = 4 * 1024 * 1024

def __init__(self, oauth2_access_token=oauth2_access_token, root_path=location, timeout=timeout):
def __init__(self, oauth2_access_token=oauth2_access_token, root_path=location, timeout=timeout,
file_overwrite=file_overwrite, write_mode=write_mode):
if oauth2_access_token is None:
raise ImproperlyConfigured("You must configure an auth token at"
"'settings.DROPBOX_OAUTH2_TOKEN'.")

self.root_path = root_path
self.file_overwrite = file_overwrite
self.write_mode = write_mode
self.client = Dropbox(oauth2_access_token, timeout=timeout)

def _full_path(self, name):
Expand Down Expand Up @@ -132,7 +140,7 @@ def _open(self, name, mode='rb'):
def _save(self, name, content):
content.open()
if content.size <= self.CHUNK_SIZE:
self.client.files_upload(content.read(), self._full_path(name))
self.client.files_upload(content.read(), self._full_path(name), mode=WriteMode(self.write_mode))
else:
self._chunked_upload(content, self._full_path(name))
content.close()
Expand All @@ -146,7 +154,7 @@ def _chunked_upload(self, content, dest_path):
session_id=upload_session.session_id,
offset=content.tell()
)
commit = CommitInfo(path=dest_path)
commit = CommitInfo(path=dest_path, mode=WriteMode(self.write_mode))

while content.tell() < content.size:
if (content.size - content.tell()) <= self.CHUNK_SIZE:
Expand All @@ -158,3 +166,24 @@ def _chunked_upload(self, content, dest_path):
content.read(self.CHUNK_SIZE), cursor
)
cursor.offset = content.tell()

def _clean_name(self, name):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can just use self._full_path instead here since that seems to work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

"""
Cleans the name so that Windows style paths work
"""
# Normalize Windows style paths
clean_name = posixpath.normpath(name).replace('\\', '/')

# os.path.normpath() can strip trailing slashes so we implement
# a workaround here.
if name.endswith('/') and not clean_name.endswith('/'):
# Add a trailing slash as it was stripped.
clean_name += '/'
return clean_name

def get_available_name(self, name, max_length=None):
"""Overwrite existing file with the same name."""
name = self._clean_name(name)
if self.file_overwrite:
Copy link
Owner

@jschneier jschneier Apr 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you can just check if self.write_mode == 'overwrite'.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jschneier can you review please ?

return get_available_overwrite_name(name, max_length)
return super(DropBoxStorage, self).get_available_name(name, max_length)