[EPIC] Refactor Secrets Handling

[cgardens]

Tell us about the problem you're trying to solve

We should audit our handling of secrets anywhere they are present in code. The goal is identify everywhere in the codebase where secrets are access and that our handling meets our desired security posture. This project is focused on de-risking areas where it would be easy for a developer to make a mistake about secrets.

Describe the solution you’d like

Spec

Execution Plan

• Audit secrets handling #9946
• Decrease code paths where secrets can be accessed #9947
• Update OSS secrets handling to match cloud #11095
• Java JSONSchema Traversal #9948
• Migration from no secret to a secret store #11604
• https://github.com/airbytehq/airbyte-internal-issues/issues/404
• https://github.com/airbytehq/airbyte-internal-issues/issues/405
• potentially others after the initial audit

[cgardens]

relates to: #7391

[olivermeyer]

I am hosting the OSS version of Airbyte and have a few questions:

• Can we expect that after this issue is closed, we will no longer see any unencrypted secrets in the Airbyte DB?
• If yes, given the 2022-q1-platform label, can we expect this issue to be release and available for the OSS version by the end of this quarter?
• If no, is there another issue which relates to ensuring that all secrets in the Airbyte DB are encrypted?

Thanks :)

[anand-srinivaas]

• If there is an option to extend external secret services like cloud secret managers or vault, will be very much helpful!
• Appreciate if this can be done in this quarter :)

Thanks

[cgardens]

@anand-srinivaas thanks for your note! I have updated this issue so that the scope of it is more clear. This issue is focused on refactoring secrets code.

We actually already support GCP Secrets Manager(docs). I have created a separate issue #10519 to add support for HashiCorp Vault. I don't think we will be able to devote time to Vault this quarter, but we are open to contribution here. If someone can implement the SecretsPersistence class with Vault, we can bring it the rest of the way home. If there is some other secrets manager you'd like please create an issue and tag me.

[cgardens]

@olivermeyer can you speak a little more about what outcome you'd like. do simply want the the db to be encrypted at rest or are you specifically looking for credentials values to be specifically encrypted?

[olivermeyer]

@olivermeyer can you speak a little more about what outcome you'd like. do simply want the the db to be encrypted at rest or are you specifically looking for credentials values to be specifically encrypted?

@cgardens I am familiar with the Airflow model (which may or may not be a common way of going about this, I don't know): secret values are encrypted in the DB and decrypted with a key which is generated when the Airflow installation is first started (doc). To spell it out, this means that even if someone somehow accesses the DB, they cannot see our secrets.

Right now, if someone gains access to our Airbyte DB, they will also get access to most system to which Airbyte connects. That's a problem.

Does this make sense?

[cgardens]

Closing as the projects we care about for this quarter are complete. The remaining open issues will be prioritized and handled separately.