Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update OSS secrets handling to match cloud #11095

Closed
Tracked by #9646
cgardens opened this issue Mar 13, 2022 · 3 comments
Closed
Tracked by #9646

Update OSS secrets handling to match cloud #11095

cgardens opened this issue Mar 13, 2022 · 3 comments
Assignees
Labels
area/platform issues related to the platform team/platform-move type/enhancement New feature or request

Comments

@cgardens
Copy link
Contributor

cgardens commented Mar 13, 2022

Tell us about the problem you're trying to solve

Currently our secrets handling changes widely based on whether the user chooses an external secrets store or not (see: https://github.com/airbytehq/airbyte/blob/master/airbyte-config/persistence/src/main/java/io/airbyte/config/persistence/SecretsRepositoryWriter.java#L129-L132). We should update the OSS secrets handling to match the handling we use when external secrets stores are used.

see also @jrhizor 's comment here:

One other part that makes this complicated is not having a separated secrets strategy / migration for this for OSS. It'd be better if we even used a table with globally encrypted values or something so we don't have misleading names depending on the configuration.

Acceptance Criteria:

  • A tech spec captures our short term approach that includes the implementation plan plus migration strategy
@cgardens cgardens added type/enhancement New feature or request area/platform issues related to the platform labels Mar 13, 2022
@pmossman
Copy link
Contributor

Grooming notes:

  • Specifically, connector config secret management. We label individual secret fields, and in Cloud, we create a different version of the config with coordinates to the secrets instead of secrets themselves.

  • In OSS, we just store the whole config in the DB, including secrets

  • It's a problem because:

  • we dont want to store plaintext secrets in databases

  • internally, we don't want there to be different code paths between environments

We want both sides to use the 'secrets coordinate' method

Potential options:

  • use some global encryption key and a database table
  • heavier weight/longer term - using different secret stores (vault, AWS, etc)
  • probably want something shorter term

Any blockers to starting the work on the shorter term option?

  • should spec out the exact mechanism for the database secret store.

@cgardens
Copy link
Contributor Author

Another related user comment: #9646 (comment)

@benmoriceau benmoriceau self-assigned this Mar 22, 2022
@benmoriceau
Copy link
Contributor

https://docs.google.com/document/d/1-WNEwYhCFoiVVspOFXtaz7L0bijpMVTUKeSIOn7TgTM/edit# -> Document that explain the problem and propose different approach to solve it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/platform issues related to the platform team/platform-move type/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants