HttpxHttpClient always follows redirects

[2tunnels]

I've encountered an issue with the redirection handling logic in HttpxHttpClient. Currently, there's no straightforward way to disable automatic redirect following when initializing HttpxHttpClient. Ideally, it should be possible to configure it like this:

crawler = HttpCrawler(http_client=HttpxHttpClient(follow_redirects=False))

While the follow_redirects argument is correctly passed to the underlying AsyncClient, HttpxHttpClient overrides this behavior in the crawl method by explicitly setting follow_redirects=True:

crawlee-python/src/crawlee/http_clients/_httpx.py

Line 168 in 179ec93

response = await client.send(http_request, follow_redirects=True)

I assume this was done because AsyncClient defaults to follow_redirects=False, ensuring that HTTP-based crawlers follow redirects by default. However, would it be preferable to handle this differently? For example adding a default follow_redirects argument in kwargs in _get_client method?