Working towards 0.19...

[tomchristie]

Hi all,

There's some work I'm putting in at the moment that I need to raise the visibility of, so y'all can see where things are headed.
Right now httpx and httpcore are in pretty good shape, but the quality of httpcore isn't quite up to the same level as we've got in httpx just yet.

We've got a couple of critical-level bugs that need addressing...

• PoolTimeout when num tasks in asyncio.gather() exceeds client max_connections #1171
• NewConnectionRequired exception raised when running behind a proxy httpcore#335

...both of which cannot be resolved without some significant refactoring.

Also httpcore doesn't yet meet up to the 100% code coverage that we have with httpx, plus I think it's apparent that managing the code/issues across two repos isn't going to be the best long term outcome, simply because it leads to too many places where things could be.

I think it's been a benefit to our design that we've had them separated, but I having tried both I think it's going to be time to have a single unified repo. Have a unified package will also mean that we have cleaner tracebacks for networking error cases, without having the httpx-httpcore jump.

Right now I'm working on a refresh of httpcore, that I'm going to tie into 100% coverage, and that rejigs some of the design in order to properly tackle the two issues above.

Currently this work is going on at https://github.com/tomchristie/httpcore-the-directors-cut

My intent is to keep working on this for a while, and then graft it into httpx, as the implementation the httpx.HTTPTransport() uses by default, and have that as the 0.19 release.

At the moment everything I'm adding would remain as private API, and switching to a direct-in-httpx implementation wouldn't require any API changes, so from the end user POV nothing ought to be changing here, other than resolving a couple of long-standing bugs.

However, longer term, the design will be working towards being able to expose some lower-level bits of API for when you really want to dig into things. Eg. Allow users to do interesting things with the NetworkBackends, such as mock network backends, record/replay backends. Allowing users to directly inspect the HTTP connections that are present in the connection pool. Allowing for alternate connection pooling implementations to exists. And just generally starting to expose the nuts and bolts on which httpx is built.

None of that would come in a hurry, tho. Likely not until after a 1.0 release. (We're talking about adding API here, not changing API, so it's something we can defer and take our time on.)

One other thing to note...

We do a fantastic job in httpx of decoupling things neatly, so that our tests are able to give us great coverage and reliability, but without actually making full-to-the-network requests throughout. Generally the tests in httpx use the MockTransport, and there's only a limited number of cases where we're actually hitting a server. We can do this, and still know that we've got great testing because we've got a really clean neat interface split at the Transport API.

My current work on the httpcore-refresh has a similar approach, but with the Network Backend API being a neat interface split, that will allow the majority of tests to work against a mock network, so that they are fast, reliable, and give us great coverage, with some other limited subset of integration tests.

Once I've integrated httpx and the latest httpcore-based work, We'll end up with a test suite that looks like...

• Client tests, which test httpx, using a MockTransport.
• Core tests, which test httpx._core, using a MockNetworkBackend.
• Integration tests, which test everything all the way through, making requests against a local server.

They'll be fast, while still giving us great coverage and confidence-in-correctness.

Anyways, enough chit-chat for now. 👋😄

[florimondmanca]

Sounds fantastic, @tomchristie. I'm following along from afar at the moment, but thank you so much for writing this up. Very solid path forward. :-) Bringing back httpcore into httpx._core now that we have a bag of experience with the interface split and real-world usage sounds fair.

[danieldjewell]

Just as a general comment (which may already be on your [collective] minds) - I (along with a not-insignificant number of others) use Termux on Android quite a bit. Android (regardless of Termux or not) uses Bionic libc (https://android.googlesource.com/platform/bionic/) ... One notable limitation of this is the complete lack of a functioning semaphore system (e.g. bionic-libc has a complete lack of sem_open()). As a result, anything that requires sem_open() fails -- multiprocessing.Queue is a good example.

I recognize that this isn't the main case, but I just wanted to bring this to your attention - if there's a way to have things gracefully fall-back, that'd be really good. (And maybe they already do, I had a short glance at the code base but couldn't be 100% sure.)

[tomchristie]

@danieldjewell - That's a really helpful comment thanks! I think we ought to escalate that into a proper issue in order to track it. You're welcome to do that if you like, or otherwise I'll aim to get onto it.