Ignore KMS when decrypting

[jashandeep-sohi]

Is there someway to prevent sops from attempting to use KMS and prefer PGP instead when decrypting a file?

[autrilla]

Not currently, no. I suppose you could reorder the keys in the encrypted YAML, as SOPS goes through them in order.

[jvehent]

Out of curiosity: why do you think having the ability to ignore KMS is a feature we need?

[mccutchen]

At $JOB we're wrapping sops in some higher-level tooling, which takes care of assuming the right role for a user working with secrets and populating the environment with appropriate AWS credentials before invoking sops.

We also provide a --recover-secrets flag for this tooling, which basically has the effect you're looking for here @jashandeep-sohi. It does the simplest thing we could think of: When --recover-secrets is given, we do not populate the environment with AWS credentials, which means sops's attempts to use KMS will fail and it will automatically fall back to PGP.

So even though this functionality is not built into sops, you can at least simulate it by manipulation of the environment in which sops is running. Hope this helps!

[jvehent]

would a --use-key pgp[i] option, where i is the position of the key, address your need?

[jashandeep-sohi]

@jvehent I work offline/near-offline a lot and sometimes the latency of trying to reach the KMS is too much. I do like the idea of a generic solution like --use-key (pgp|kms)[i].

[jpadilla]

I'm actually running into this too. We're using sops in development with PGP and KMS for production. KMS will always be tried first and fail while in development. A --use-key flag would be pretty useful, I guess you could optionally specify the index of the key to use, but would expect --use-key pgp to just try to decrypt with all PGP keys.

[brad-jones]

Not currently, no. I suppose you could reorder the keys in the encrypted YAML, as SOPS goes through them in order.

@autrilla I did attempt this and it appeared to make no difference what so ever, still attempts to call out to KMS.

You were referring to simply moving sops.pgp to appear above sops.kms?

Whats more the second you edit the file sops then reorders it such that the pgp "YAML" key is last again.

Obviously each and every time a developer decrypts a secret locally thats a KMS call that costs money. If they happened to have a valid PGP key it would make sense to try that first. So the operation is cheaper and faster.

Ideally we would like to keep secrets encrypted at rest at all times and then use something like https://cyberark.github.io/summon/ to inject secrets at runtime. Hence the need to decrypt lots and lots.

[brad-jones]

Update: If you delete sops.kms then it will use PGP to decrypt and not bother trying KMS.

[autrilla]

@brad-jones no, simply moving the entry in a mapping won't work. I was suggesting using key groups and making sure your PGP key group appears first in the file. If you use a Shamir threshold of 1, and then list your PGP key group first, and the KMS key group second, either one of those groups would be enough to decrypt the file, and sops will try the first key group first.

[brad-jones]

Ah nice, I'll give it a shot.

[ajvb]

@autrilla In taking a look at the base AWS client struct in the sdk (https://docs.aws.amazon.com/sdk-for-go/api/aws/client/#Client) - it looks like there is a retryer with a back-off, which is probably why this can take a while if you can't use the AWS KMS key for whatever reason.

It might be worth including an option (--use-key X, --offline), including our own Retryer with a shorter retry window, or changing the decryption functionality to always try pgp first?

My first choice would be the include a shorter retry window, as that is probably the simplest and will be similar to what we want to do for #402

[autrilla]

@ajvb that could be good, yeah.

Another option would be #266 since we'd try PGP at the same time as KMS, and that would succeed before KMS times out.

[ajvb]

@autrilla 👍 sounds good

[SinisterMinister]

I just submitted PR #638 which, while it doesn't allow for specifying specific ones to run, it runs all in parallel with success winning and returning immediately. This solved the problems described as they were similar to ours.

We're using the sops vars plugin in our deploys and with it, Ansible calls sops with each task run. This is no big deal on an AWS machine or one with credentials. Ones without that rely on PGP would literally take a minute for each task to run sops, however. This solved the problem and now sops runs as fast as the quickest successful key.

[dt-darren]

Adding my support for the option to choose which key to use during decrypt. Our use case is we would be using one vault as a backup and using another as the "everyday vault". The backup would have strict access policies so the machines that would do the everyday access wouldn't have access to the backup. So performing all the attempts in parallel is a waste of effort.

If the parallel method is chosen can sops be made to only output error logging if no method succeeds, not for every one that fails even if one works please? :)

[james-callahan]

This is a pretty high usability issue for me. Could we get a higher priority?

[bkreitch]

We are affected by this as well. We use Hashicorp Vault for editing secrets and servers have age private key for decryption (via Flux CD). The problem is that some servers don't have access to the Vault network so they spend 1 minute trying before giving up and continuing to use age.

Using key groups doesn't seem to help too. Shamir threshold is required to be at least 2, so I tried 3 groups - 2 age (same key) and 1 hcvault, but SOPS still went to hcvault even when age was in the first two key groups.

So I'm opening #1121 to put PGP/age keys first in line and hopefully avoid the need to contact remote KMS/Vault.

[lenalebt]

I'm also affected. I usually want to prefer KMS keys, because that's what we use for almost all people involved. However, I sometimes am working on bad internet connections and need to decrypt stuff way more often than encrypt - and my backup PGP key is available. SOPS then always fails, although there is a key it could use. I think re-ordering keys is not an option, because that would affect my teammates as well, and I don't want PGP to be the default option.

For me, it would be enough to have a switch --offline-mode or something like that, which would then just skip over KMS methods (and others requiring an internet connection). Of course, not all operations would work (like re-encrypting), but it would already be a massive improvement.

[bkreitch]

@lenalebt If your teammates don't have correct PGP key then SOPS will just continue to KMS. And if you do have PGP then why not to use it?

[danielwoodz]

If anyone is looking for a stop-gap solution before #1121 is merged, I implemented this on our runners which has alleviated the KMS bottleneck and prevented failures (decrypting hundreds of jsons).

# reorder json keys to put GPG first for decryption providers
echo "[INFO] Reorder VPC jsons"
mapfile -t jsons < <(find conf/vpcs* -name "*.json" -type f | sort)
for json in "${jsons[@]}"; do
    echo "[INFO] Reorder ${json}"
    jq '.sops = {
        "lastmodified": .sops.lastmodified,
        "version": .sops.version,
        "encrypted_regex": .sops.encrypted_regex,
        "mac": .sops.mac,
        "gpg": .sops.gpg,
        "pgp": .sops.pgp,
        "age": .sops.age,
        "hc_vault": .sops.hc_vault,
        "kms": .sops.kms,
        "azure_kv": .sops.azure_kv,
        "gcp_kms": .sops.gcp_kms
    }' "${json}" >"${json}".tmp
    mv "${json}".tmp "${json}"
done

[GMartinez-Sisti]

Instead of trying to sort the keys, I would suggest a --prefer option that could be specified multiple times. We have some SOPS files that are used across clouds, meaning that it will have AWS, GCP and Azure keys on the config, sorting wouldn't work for all, but giving priority to a type of key would include all the scenarios (also the PGP ones referred above). Thoughts?

[felixfontein]

That sounds like a great idea! I'm not sure whether the CLI library allows something to be specified multiple times (Edit: yes, it does, and it's already used in other parts), but you could comma-separate different values. In any case, it should also be possible to provide this option via an environment variable. That way you can globally export this in your ~/.profile instead of having to supply it on every sops edit/decrypt call.

[danielwoodz]

Instead of trying to sort the keys, I would suggest a --prefer option that could be specified multiple times. We have some SOPS files that are used across clouds, meaning that it will have AWS, GCP and Azure keys on the config, sorting wouldn't work for all, but giving priority to a type of key would include all the scenarios (also the PGP ones referred above). Thoughts?

This is a great suggestion that probably deserves its own user story.

Probably not prefer as that's ambiguous. Probably something more like --decryption-order that can take an ordered list, and then just tries the remaining on a failure.

[bkreitch]

--prefer / --decryption-order could be useful but it's a bit complicated to pass it when SOPS is used as a library (for example in FluxCD). Maybe to store it inside SOPS file or make sure keys are stored/used in file in strict order?

[GMartinez-Sisti]

--prefer / --decryption-order could be useful but it's a bit complicated to pass it when SOPS is used as a library (for example in FluxCD). Maybe to store it inside SOPS file or make sure keys are stored/used in file in strict order?

This is a decryption preference for the client, so it shouldn't be part of the SOPS file, otherwise we're back to the initial issue of not working across different clouds/environments. I'm not sure if Flux would pass environment variables to the process running the SOPS decryption, but if it doesn't then I think it's an implementation concern for Flux or it should be able to be configured on import.

[hiddeco]

@bkreitch Flux will not be affected by this as it already has been, and will continue to, prefer offline keys first.

[bkreitch]

Well, basically sorting offline methods could be a private (and default) case as --decryption-order age,pgp but priority could be given to other key types too, i.e. --decryption-order kms,gcpkms. And idea is still to sort the keys according to the given order. I think I can extend the PR to support that.