Support --preferred-chain certbot option

[lukecyca]

Let's Encrypt is changing the default chain of trust (details), which has wide implications for backwards compatibility for clients with old CA root stores. Certbot now supports specifying --preferred-chain so that we can manage this transition (details).

The first part of this transition is going to occur in less than a week, or as soon as they receive their cross-signature from IdentTrust for the new "R3" Intermediate. So this is very time sensitive.

1. Is it possible to specify --preferred-chain in letsencrypt-nginx-proxy-companion currently? Is there a way to override the certbot parameters that are used in order to include this?

2. Can we add a LETSENCRYPT_PREFERRED_CHAIN environment variable so that this can be selected on a per-container basis?

[buchdag]

Hi.

First thing first, thanks for warning us about this change.

As mentioned in the certbot issue you linked to, the first change that'll have a real life impact have been postponed to January 11, 2021. The intermediate certificate change that will happen next week won't impact site visitors :

We will still be making a smaller change to our issuing intermediate this fall. We’ll switch to using our just-issued R3 intermediate. However, that intermediate will be cross-signed by IdenTrust (just like our “Let’s Encrypt Authority X3” intermediate is), so compatibility with your site visitors will not change. Your ACME client should automatically download and configure the correct certificate chain with the next issuance after we make the change.

As you're aware of, this project does not use certbot but simp_le, so support for --preferred-chain will have to be added to simp_le before we can support it in letsencrypt-nginx-proxy-companion.

kuba/simp_le has been abandoned by its creator for years, the currently maintained fork (and the one used by letsencrypt-nginx-proxy-companion) is zenhack/simp_le, so you should re-open you issue on this repo.

[lukecyca]

Thanks for the response. I've opened an issue on zenhack's repo.

Regarding the urgency:

January 11th is when they will switch the default Intermediate to E1 which is no longer signed by Identrust. This will be a big deal for many people, but they say that the R3 Intermediate will still be an option (via --preferred-chain) for those of us requiring this backwards compatibility a bit longer.

In the coming week they are switching from X3 to R3. The latter is still signed by both Identrust and ISRG so in theory it has the same backwards compatibility. What they have not stated is which of those two trust chains will be used by default and end up in our fullchain.pem. For most modern browsers it doesn't matter too much, as the browser will be able to follow either chain by requesting additional intermediates as required. However for embedded devices this is not necessarily the case. Only the chain that we supply in fullchain.pem is viable.

Update: I asked Let's Encrypt and got a great response. Sounds like everything should keep working as is until January 11th. At that point we do need a way to select which intermediate to use.

[buchdag]

@lukecyca the featured has been added to the acme.sh branch.

acme.sh appears to be correctly called with the --preferred-cert flag but I'm unable to verify if this actually work or not.

The acme.sh based Docker image can be pulled at jrcs/letsencrypt-nginx-proxy-companion:acmesh if you want to check it out.

[lukecyca]

I pulled the acme.sh image and tested it out. I can see with DEBUG=1 that it correctly calls acme.sh with --preferred-chain "ISRG Root X1" however the certificate that ends up in chain.pem and fullchain.pem is still signed using the default DST Root CA X3 chain. So it appears to be an acme.sh problem I guess?

...

Indeed, acme.sh#3252 describes what I'm seeing. It fetches both chains, but puts the ISRG one in etc/acme.sh/[email protected]/host.example.com/ca.alt. Connecting to the host with a browser shows exactly the same cert chain as before: Let’s Encrypt Authority X3 → DST Root CA X3. I'm actually a bit confused about what I even expect to be seeing here.

In any case it looks like this repo is doing the right thing now, so this issue can probably be closed.

[buchdag]

Looks like you experienced the same results as me and were equally confused. 😅

[buchdag]

@lukecyca I pushed another version of jrcs/letsencrypt-nginx-proxy-companion:acmesh to Dockerhub, built with the master branch of acme.sh instead of a tagged release (2.8.7), if you want to try again. If this still does not work, we should probably mention in the acme.sh issue that we're experiencing it too.

[lukecyca]

I updated the image, and verified that the /app/acme.sh inside it is version 2.8.8 (same as what it says in their master branch currently).

Forced renewal. Verified the acme.sh parameters in the debug output:

Calling acme.sh --issue with the following parameters : --log /dev/null --debug 2 --server https://acme-v02.api.letsencrypt.org/directory --config-home /etc/acme.sh/webmaster@[example.com] --webroot /usr/share/nginx/html --keylength 4096 --cert-file /etc/nginx/certs/[example.com]/cert.pem --key-file /etc/nginx/certs/[example.com]/key.pem --ca-file /etc/nginx/certs/[example.com]/chain.pem --fullchain-file /etc/nginx/certs/[example.com]/fullchain.pem --preferred-chain "ISRG Root X1" --force --domain [example.com]

The result was exactly the same as before. The intermediate cert that gets included in /etc/nginx/certs/[example.com]/fullchain.pem is Let's Encrypt Authority X3 signed by DST Root CA X3, rather than Let's Encrypt Authority X3 signed by ISRG Root X1.

Edit: Looks like acme.sh has confirmed this as a bug and is working on it. Let's track that and once they have a fix perhaps you can pull that into your acmesh branch and I will re-test. Thanks for all your help BTW!

[buchdag]

The issue is supposed to be fixed in the dev branch of acme.sh, so I built and pushed another acmesh image built with this one.

[lukecyca]

I'm hitting API limits from all this testing, so I'll try again in a day or two.

[lukecyca]

I pulled the latest image and did force_renew and unfortunately the behaviour is exactly the same.

Their fix was in this commit so I manually went into my container and verified that it does indeed have that change. So I can be sure I was testing with this fix.

[lukecyca]

I'm not sure where to go from here. :(

I guess just trace through what's different between proxy-companion and using acme.sh natively.

@Neilpang are you able to offer any suggestions? Can you see our problem from the log file I emailed you?

[buchdag]

From what I recall from my tests two weeks ago I'm starting to doubt that the --preferred-chain flag is correctly passed to or accepted by acme.sh: the --preferred-chain flag should add some additional outputs (at the very least "Try rel: $rel" with info level) but I don't recall finding them in the logs. Do you have those outputs in your logs ?

The flag parameter is the only one to be passed to acme.sh --issue with escaped double quotes, because that's the only parameter we're using that can include whitespace. I wonder if that might be causing the issue.

if [[ "$acme_preferred_chain" != "<no value>" ]]; then
    # Using amce.sh --preferred-chain to select alternate chain.
    params_issue_arr+=(--preferred-chain "\"$acme_preferred_chain\"")
fi

[buchdag]

I pushed yet another version of the acmesh image without those escaped quotes.

[lukecyca]

I just emailed you my logs of my last couple attempts earlier today. I tried both ACME_PREFERRED_CHAIN: "ISRG Root X1" as well as ACME_PREFERRED_CHAIN: "ISRG" because it occurred to me that the spaces might be an issue.

I will try your latest image next.

[lukecyca]

It works! 🎉

Your latest image works for me with with ACME_PREFERRED_CHAIN: "ISRG"

Thanks for all of your help with this. Let me know if there's anything I can do to help get the acmesh branch tested and ready for release.

[buchdag]

@lukecyca just to be sure : do you confirm that the feature work correctly with jrcs/letsencrypt-nginx-proxy-companion:2.0.0 (or :2.0) ?

[lukecyca]

I ran the following three tests over the last hour:

• On my test server that was used for testing the acmesh branch previously in this thread above, I updated the container to 2.0 and did force_renew and verified that the new certs were as expected.

• On another server that was still running 1.x, I pulled the latest docker image (and added a volume for /etc/acme.sh, and allowed it to regenerate a bunch of certificates as it started up. That all seemed to work fine.

• On that new server, I changed one of my sites to ACME_PREFERRED_CHAIN: "ISRG" and then did force_renew, and verified that its certificate was successfully changed over to the other chain.

👍 Everything looks good to me. Thank you for all your effort on this.

[Pierre-Gilles]

Works fine on my side too.

I tried to set ACME_PREFERRED_CHAIN="ISRG Root X1" on a backend container and then did a force renew, and it worked perfectly the certificate was changed to the new chain.

Many thanks for this feature 🙏

[RBarci]

I'm trying to install generated Letsencrypt cert to Zimbra 8.7.11 but verifycrtchain is not working. If you split fullchain.pem to 3 .cer files, by opening the 3rd part, it shows "This certificate has exired or is not yet valid."

Any sugestion?

Best Regards