HTTP2 support

[meisimo]

I have noted that the discussion about HTTP 2 support has years closed, but I have found that the trend of HTTP 2 has increased, already is over the 50% of the internet. This is my reference :D

Maybe its not the purpose of this lib, but I love to use it for web scrapping and this feature would be great! :D

Relatef issues
#320
#863

[Dreamsorcerer]

Seems you've already found the existing discussion around this.

Basically, we'd all like to have HTTP/2 support available, but nobody is working on it. Opening an issue requesting it again is, unfortunately, not going to make it happen. Until someone volunteers to implement it, I'm afraid we're stuck without.

I'm not sure how much benefit it would provide for your web scraping though. Almost all sites still support HTTP/1.1, and my impression of the performance benefits is that it helps speed up getting extra files that are needed by a browser for the web page. Files that you probably won't need if you are web scraping.

[Granitosaurus]

@Dreamsorcerer sorry to revive a closed issue but to asnwer your question:

I'm not sure how much benefit it would provide for your web scraping though

It's very easy to identify and block web scrapers when most of the real traffic is HTTP2.

[webknjaz]

Just to reiterate: this is an open source project happy to accept contributions. A PR for this feature will probably be submitted by someone who really-really needs it. Since nobody attempted it so far, it doesn't look like an ultimate priority for the end-users 🤷‍♂️
Prove me wrong :)

[lilydjwg]

I understand that this is an open source project, but I don't understand why you close the issue (and lock the other). This makes it harder to find if this project has support it or not (or if a patch will be accepted or not).

I saw this new issue closed and thought it had been implemented finally. It turned out not.

[jameshilliard]

Not sure if anyone has looked into this in detail yet but apparently twisted was able to add http2 support via the h2 library, I wonder if a similar approach may be workable for aiohttp.

[webknjaz]

@jameshilliard we have two implementations for things that tend to be slow — one in pure Python and the other in Cython (using the llhttp bindings in case of the HTTP parser).

I'm aware that h11 and h2 exist, but somebody would still have to implement the C-extension version with a similar functionality. We'd need feature parity for this to be acceptable.

[jameshilliard]

I'm aware that h11 and h2 exist, but somebody would still have to implement the C-extension version with a similar functionality. We'd need feature parity for this to be acceptable.

That may not be too difficult as it appears to have been supported by h2's header parser in the past, h2 itself does not appear to actually do parsing itself, it seems to use the hpack backend for header parsing which could have an optimized backend swapped in.

For hpack it even appears that using the optimized nghttp2 c module was supported at some point, so using the nghttp2 based c module may be one approach for getting optimized us http2 hpack header parsing working with h2.

In terms of how h2 would be wired up it looks to me like we'd potentially want to hook callsites like data_received similar to the twisted dataReceived hook, does that seem like it might be viable?

Would we need to pull h2 into the aiohttp codebase so that it can customized or would it make sense to try and use it as is potentially?

[Dreamsorcerer]

@jameshilliard we have two implementations for things that tend to be slow — one in pure Python and the other in Cython (using the llhttp bindings in case of the HTTP parser).

Just to put another option out there that's worth considering. mypyc can compile typed code to C extensions automatically, so that could allow us to have only 1 version of code to maintain.

[jameshilliard]

mypyc can compile typed code to C extensions automatically, so that could allow us to have only 1 version of code to maintain.

Autotranslated code like that tends to not be as good performance wise as native c libraries. Also alpha quality libraries/tools aren't probably a great option here.

[Dreamsorcerer]

Well, we wouldn't know the performance unless we tested it. I don't mind which way we go, but might save a lot of time using mypyc, and make it easier for contributors who wouldn't need to learn Cython to work on these parts of the code.

That alpha warning is 2 years out of date. The project seems to be in a stable position these days. If mypyc produces an output for your code, it should be reliable. The main issue currently is that there are still several features unsupported (e.g. async generators won't compile currently), but those missing features will probably not be used in the code we're talking about.

[FeldrinH]

Would it be possible to reopen this issue to make it clear for those interested that HTTP2 support is currently not implemented? I think it will be easier to attract interested contributors if there is a clear open issue requesting HTTP2 support.