Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add S3 storage importer #150

Closed
morazow opened this issue May 2, 2023 · 0 comments · Fixed by #152
Closed

Add S3 storage importer #150

morazow opened this issue May 2, 2023 · 0 comments · Fixed by #152
Assignees
@morazow @kaklakariada
Labels
feature Product feature

Comments

@morazow
Copy link
Contributor

morazow commented May 2, 2023

Situation

We should add another source definition for the spark-connector that can import from intermediate S3 storage, e.g, from temporary buckets, when reading data from Exasol database.

Acceptance Criteria

  • S3 default source is defined
  • Reading from Exasol using intermediate storage is supported
  • Integration tests added using localstack S3 container
@morazow morazow added the feature Product feature label May 2, 2023
@morazow morazow self-assigned this May 2, 2023
@morazow morazow mentioned this issue May 3, 2023
Merged
morazow added a commit that referenced this issue May 5, 2023
@morazow @kaklakariada
#150: Added S3 intermediate storage layer (#152)
63e9950 
Co-authored-by: Christoph Pirkl <[email protected]>
@morazow morazow closed this as completed May 8, 2023
morazow added a commit that referenced this issue Jul 14, 2023
@morazow @kaklakariada
@ckunki @tkilias @redcatbear
Prepare for release of 2.0.0 (#186)
e522e56 
* #150: Added S3 intermediate storage layer (#152)


Co-authored-by: Christoph Pirkl <[email protected]>

* #149: Added s3 import query generator and runner (#156)


Fixes #149

Co-authored-by: Christoph Kuhnke <[email protected]>
Co-authored-by: KK <[email protected]>

* #160: Add support for writing to Exasol database using S3 as intermediate storage (#161)


Fixes #160

* Address review suggestion from @tkilias

* #159: Added cleanup process to remove intermediate data after job finish (#165)

Fixes #159

---------

Co-authored-by: Christoph Kuhnke <[email protected]>
Co-authored-by: KK <[email protected]>

* #158: Refactored options class (#167)

Fixes #158

---------

Co-authored-by: Torsten Kilias <[email protected]>

* #168: Refactored to add module setup (#170)


Fixes #168

* #171: Refactored artifact packaging and releasing process for module setup (#173)


---------

Co-authored-by: Sebastian Bär <[email protected]>

* #174: Refactored GitHub workflow actions (#175)

Fixes #175

* #176: Fixed sha256sum files upload bug (#179)

* #155: Refactored to unify user parameters (#178)

Fixes #155

---------

Co-authored-by: Christoph Pirkl <[email protected]>

* #164: Added validation check before writing to intermediate location (#180)

Fixes #164

---------

Co-authored-by: Christoph Pirkl <[email protected]>

* #162: Updated developer guide with staged commit behaviour (#182)

Fixes #162

---------

Co-authored-by: Christoph Pirkl <[email protected]>

* #183: Updated user guide to include latest changes (#184)

Fixes #183

---------

Co-authored-by: Christoph Pirkl <[email protected]>

* Updated version to `2.0.0` (#185)

* Changed version
* Improved default options
* Updated changes and user guide docs

---------

Co-authored-by: Christoph Pirkl <[email protected]>
Co-authored-by: Christoph Kuhnke <[email protected]>
Co-authored-by: KK <[email protected]>
Co-authored-by: Torsten Kilias <[email protected]>
Co-authored-by: Sebastian Bär <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@morazow morazow

@kaklakariada kaklakariada

Labels
feature Product feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#150: Added S3 intermediate storage layer exasol/spark-connector
2 participants
@morazow @kaklakariada