Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement write and delete for S3FileSystem #54

Closed
wants to merge 3 commits into from

Conversation

wjones127
Copy link

As part of prototyping apache/datafusion#2246 I am implementing the API for S3 to see what it's like to implement these operations for object stores. (So far, I've learned that AsyncWrite is quite difficult to implement 😢.)

@matthewmturner
Copy link
Collaborator

Very happy to see this - i had it on my to-do list as well. What issues have you run into with AsyncWrite?

@wjones127
Copy link
Author

Basically took me a bit to figure out how to create this state enum:

enum S3MultipartUploadState {
    Ready,
    UploadInProgress(Pin<Box<dyn Future<Output = Result<()>>>>),
    CompleteInProgress(Pin<Box<dyn Future<Output = Result<()>>>>),
    Completed
}

Need to be able to hold onto the futures between the different poll calls. Hopefully, with an example for S3, the implementations for other object stores can just follow the same general pattern.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants