HTTP/2 support

[s4y]

Submitting a website which supports HTTP/2 fails with "Error: Invalid Certificate Chain".

The message advises running the SSL Labs test, which passes.

It sounds like other folks have experienced the same issue with HTTP/2 sites.

[lgarron]

(Copying my observation from the Chromium bug.)

I can perform the scan just fine from the Google corporate network.

It's exactly the same Go code as on hstspreload.appspot.com
I strongly suspect that Google Cloud is causing a problem. One potential solution is to move to a Google Compute Engine VM, but that requires some bureaucracy for the DNS (and loses the scalability of flexible environments), so I'd prefer to avoid it.

I'll talk to the Cloud team.

[lgarron]

For reference, the Chromium bug initially filed for this was https://crbug.com/637610

[lgarron]

I was contacted about this again today. That makes three reports within 4 days, with no previous issues.

[hpvb]

It happens for my domains now too after enabling http/2 on my frontends.

[lgarron]

It happens for my domains now too after enabling http/2 on my frontends.

Could you post which domain that is?
I'm trying to collect test cases for when it comes time to look at solutions for this.

[hpvb]

You can test using

• https://notabug.org
• https://tmm.cx

Thanks for looking into it!

[jiri-pejchal]

Another case:
https://www.czc.cz

[lgarron]

A lot of these have come up very recently. We should fix this soon.

[Ayesh]

Thanks Lucas.
I worked around it by disabling http/2, submitting the site, and enabling http/2 back. Some of my sites are now in the preload list, and hopefully an automatic recheck wouldn't kick in before it can work with http/2 sites properly.

Once again thanks for your effort in this Lucas!

[lgarron]

The following appears in the logs a lot: transport: http2Client.notifyError got notified that the client transport was broken EOF

[lgarron]

Some more stuff:

• We use Google Cloud Flexible Environment, which seems to compile using Go 1.6.3 in a Docker container. So net/http: Go 1.6 http.Client doesn't support http2 by default golang/go#14391 is probably not the issue, but maybe?
• Some sites generally don't work with Go's HTTP library (regardless of whether the version properly supports HTTP/2). For example, trying to connect to https://www.czc.cz using a Go implementation of httpstat hangs for 5 minutes and then prints failed to read response: Get https://czc.cz: http2: server sent GOAWAY and closed the connection; LastStreamID=0, ErrCode=NO_ERROR, debug=""

I'm currently working on trying to move hosting to a full VM.

[lgarron]

Oh, and Go 1.7 for App Engine/Flexible Environment is under way, but not launching soon enough.

[Ayesh]

I worked around it by disabling http/2, submitting the site, and enabling http/2 back. Some of my sites are now in the preload list, and hopefully an automatic recheck wouldn't kick in before it can work with http/2 sites properly.

#35 will eventually remove the site if we don't support HTTP/2 in the scanner.

[lgarron]

#35 will eventually remove the site if we don't support HTTP/2 in the scanner.

Note that the scanner supports HTTP/2 if you use the proper version of Go.
However, the submission site still fails on HTTP/2 because the deployment environment is restricted to Go 1.6 right now.

[makuser]

However, the submission site still fails on HTTP/2 because the deployment environment is restricted to Go 1.6 right now.

@lgarron what can we do about that?
The sites on my servers (eg. https://mynightout.de/) are affected as well, because I use HTTP/2.

[lgarron]

@lgarron what can we do about that?

From some looking around, it looks like App Engine / Flexible Environment might support Go 1.7 in beta soon (like, very soon).

Barring that, I could use help figuring out a way to adapt https://github.com/chromium/hstspreload to use HTTP/2 in 1.6 (which I've seen indications is possible, but haven't had time to do).

[keystrike]

I'm not sure if you need more test cases but I have a number of domains this affects.

biopsychiatry.com
modafinil.com

I've preloaded mdma.net and hedweb.com prior to moving to HTTP/2 and they are still preloaded at the moment.

All of the above sites are on the same server and get the same results on the SSL Labs test.

[lgarron]

From some looking around, it looks like App Engine / Flexible Environment might support Go 1.7 in beta soon (like, very soon).

After a month of trying to track down this planned change, it appears that this won't be happening. :-/

[Ayesh]

Hi - It's working fine with HTTP2 URLs now! Thanks!
hstspreload.org itself is now on http/2:

• https://http2.pro/check?url=https%3A//hstspreload.org/
• https://hstspreload.org/?domain=hstspreload.org

[lgarron]

Are you sure?
It doesn't seem to work with all. liquidlight.co.uk works for me locally, but not on hstspreload.org

[lgarron]

Also, someone reported scooter-system.fr today: https://twitter.com/Eroan/status/818760236060340226

[Ayesh]

I'm not 100% sure, but please see (but not submit) this:
https://hstspreload.org/?domain=rs-devdemo.host

This site is using h2, but hstspreload shows me the form to add.

[lgarron]

Hmm, interesting. Sounds like some HTTP/2 sites work, and some don't.
https://tools.keycdn.com/http2-test doesn't distinguish between them.

If someone knows a lot about HTTP/2, I could use some help sleuthing to figure out what's different about liquidlight.co.uk and scooter-system.fr.

[Ayesh]

There is a very good chance I'm at wrong here, but I tested the failed domain names (liquidlight.co.uk and others) with h2spec, and none of them passed the test. The domains passing hstspreload test also passed h2spec.

Here is a comparison for your reference: https://gist.github.com/Ayesh/be83385f63699f2946d29bce5cdce7bc

I also tested with curl -Iv --http2 https://hstspreload.org/,and from what I see, the certificate and protocol differences (RSA vs ECDSA, the encryption method/modes, etc) do not make a difference.

For rs-devdemo.host (which did pass the test), I'm using Apache 2.4.25 with ECDSA + RSA2048 certificates if that helps anyone. I tested with RSA2048 alone, and still passed the test.

[lgarron]

Hmm, h2spec returns ERROR: HTTP/2 settings negotiation {failed, timeout} for every site, on three of my computers. Am I holding it wrong?

[Ayesh]

h2spec has a rather short timeout. I got the errors too, but a longer timeout fixed it.

h2spec -h hstspreload.org -p 443 -t true -S -o 50 # -o 50 is timeout

[lgarron]

It seems h2spec does distinguish between the failure cases correctly.

I've sent Brad Fitzpatrick an email about this, and have sent questions liquidlight.co.uk and scooter-system.fr to see what servers they're using.

[theromi]

I'm having the same problem (SSL Labs rates me A+).

-> % nginx -v
nginx version: nginx/1.11.8
-> % openssl version
OpenSSL 1.0.2j 26 Sep 2016
-> % uname -a
Linux ... 4.8.13-1-ARCH #1 SMP PREEMPT Fri Dec 9 07:24:34 CET 2016 x86_64 GNU/Linux

I'm using certs from Letsencrypt. Anything else you want to know?

[lgarron]

I'm using certs from Letsencrypt. Anything else you want to know?

What's your domain?

Also, any chance you can get it to pass h2spec (see above)?

[theromi]

https://romanmichel.de

I'll see about h2spec this evening (about 4 hours from now for me).

[s4y]

Another data point: I was able to submit istheinternetdown.com when I saw the comments from ~11 days ago.

But either something else was broken then, or the test changed — that domain redirects to HTTPS only if the request includes Accept: text/html. The goal is to provide HSTS to browser-like UAs without breaking things like curl istheinternetdown.com (which exits and prints nothing if it gets a redirect). This might not be OK for preloading; separate question.

Today, when I submit the domain, it gets rejected because of a missing redirect. If you didn't change anything, then it working temporarily could be evidence of other mysterious/wrong App Engine behavior. But no more certificate chain gripes.

[lgarron]

I tried working around this in a few ways:

• Set transport.ExpectContinueTimeout = 0
  on the server (, and all of them failed.
• Set the GODEBUG env var to h2client=0 simultaneously for the local build environment and the server.

I might have tried setting transport.TLSNextProto to a value that disables HTTP/2, don't recall.

In any case, I'm running out of ideas short of moving to a custom VM.

For the people on this thread with sites that still fail, could you let me know what stack your server is running?

[theromi]

h2spec for my domain:
Finished in 111.8529 seconds
145 tests, 115 passed, 0 skipped, 30 failed

So some fail, most go well. Full log: http://pastebin.com/cYvN4eXb

For my stack, see above. Here is my nginx-config: http://pastebin.com/FTMqJ4nT

[Ayesh]

@theromi for comparison, here is the results of a site that is passing the hstspreload test as well (it's from https://http2.pro).
http://pastebin.com/CnN7iDFy

145 tests, 144 passed, 0 skipped, 1 failed

FWIW, above server is running Apache 2.4.25, with ECDSA (prime256v1) + RSA (2048 bits) dual certificates. It produces same results over RSA 2048 as well. For me, the hstspreload test started to pass only after I upgraded Apache from 2.4.23 to 2.4.25. This upgrade contained some HTTP2 security fixes. Unless the hsts check had any changes, upgrading to 2.4.25 will most likely fix the h2spec test. No clue about nginx.

[lgarron]

Looks like we're now on Go 1.8; yay!
https://groups.google.com/d/msg/google-appengine-go/QTCsm5aLojU/J75Zc2yIAwAJ

All the examples in this bug seem to work now. If anyone is still seeing issues with HTTP/2, please file a new bug.

[theromi]

Unfortunately, my domain (romanmichel.de) still does not work.

[jiri-pejchal]

Same here, czc.cz returns Invalid certificate chain.

[lgarron]

romanmichel.de and czc.cz have issues with Go clients apart from HTTP/2. I would recommend separate bugs if you think there is something worth fixing on the scanner side (but I'll need help diagnosing the details).

[nogweii]

I'm running into what I think is the same error with my own website - evaryont.me (running nginx on Centos 7)

@lgarron do you have a recommended client to test Go & HTTP/2? Would httpstat suffice?

[lgarron]

I'm running into what I think is the same error with my own website - evaryont.me (running nginx on Centos 7)

I checked locally, and evaryont.me does not have the same error.

However, it seems your headers are malformed. Try removing the second colon after Referrer-Policy:

> https -ph evaryont.me
HTTP/1.1 200 OK
Accept-Ranges: bytes
Connection: keep-alive
Content-Length: 3528
Content-Security-Policy: default-src https: data: 'unsafe-inline' 'unsafe-eval'
Content-Type: text/html
Date: Mon, 26 Jun 2017 19:00:57 GMT
ETag: "594f8032-dc8"
Last-Modified: Sun, 25 Jun 2017 09:19:46 GMT
Referrer-Policy:: strict-origin-when-cross-origin
Server: nginx
Strict-Transport-Security: max-age=15768000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block