InvalidHeader exception when trying to add a ":method" header field (breaking change in 2.28.0)

[Alex8768765]

My app is making a POST request enforcing some specific header fields. Among them, I was able to explicitly enforce a ":method" header field. But since 2.28.0, this now raises the following error:

requests.exceptions.InvalidHeader: Invalid leading whitespace, reserved character(s), or returncharacter(s) in header name: ':method'

Enforcing this header field is possibly a widespread practice as this is part of the requirements for sending notifications through Apple's push notification servers.

[nateprewitt]

Hi @Alex8768765,

Can you please provide the info we originally requested in the issue template? The header you're using isn't valid in the HTTP spec, so it seems unlikely this would be widespread. Python itself rejects sending a header like this in our supported versions.

>>> requests.__version__
'2.27.1'
>>> r = requests.get('https://httpbin.org/get', headers={':method': 'test'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/nateprewitt/.pyenv/versions/3.7.12/lib/python3.7/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/nateprewitt/.pyenv/versions/3.7.12/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/nateprewitt/.pyenv/versions/3.7.12/lib/python3.7/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/nateprewitt/.pyenv/versions/3.7.12/lib/python3.7/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/Users/nateprewitt/.pyenv/versions/3.7.12/lib/python3.7/site-packages/requests/adapters.py", line 450, in send
    timeout=timeout
  File "/Users/nateprewitt/.pyenv/versions/3.7.12/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
    chunked=chunked,
  File "/Users/nateprewitt/.pyenv/versions/3.7.12/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/nateprewitt/.pyenv/versions/3.7.12/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/Users/nateprewitt/.pyenv/versions/3.7.12/lib/python3.7/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/nateprewitt/.pyenv/versions/3.7.12/lib/python3.7/http/client.py", line 1322, in _send_request
    self.putheader(hdr, value)
  File "/Users/nateprewitt/.pyenv/versions/3.7.12/lib/python3.7/site-packages/urllib3/connection.py", line 224, in putheader
    _HTTPConnection.putheader(self, header, *values)
  File "/Users/nateprewitt/.pyenv/versions/3.7.12/lib/python3.7/http/client.py", line 1249, in putheader
    raise ValueError('Invalid header name %r' % (header,))
ValueError: Invalid header name b':method'

[Alex8768765]

Apologies, I missed that the impacted piece of code was using Requests together with a 3rd party HTTP/2 adapter.

from hyper.contrib import HTTP20Adapter
import requests
s = requests.Session()
s.mount("https://httpbin.org/get", HTTP20Adapter())
r = s.post("https://httpbin.org/get", headers={':method': 'test'})

That 3rd party adapter does not seem to be a maintained project as we speak, so I understand my issue is essentially finding the proper way to make HTTP/2 requests.

[nateprewitt]

Hrmm, well that does pose a problem. While Requests only supports http/1.1, Hyper has been "compatible" for quite a while.

I'll need to take a look at possible escape hatches. Ideally, we can update the hyper example to still use requests and bypass the header checks, but I'm not sure how/if that's accomplished.

[nateprewitt]

After a cursory look, I don’t think we can escape this at the adapter level. The immediate fix is to use the PreparedRequest workflow in the Requests docs and set the headers after the Request is prepared. This can be handed to the Session and should work.

I’d like to see how often hyper is being used with Requests still before we look at merging in a code change to bypass this.

[BlondinkaQ]

replace in header '\r' or '\t' or '\n' to empty symbol, this solved my problem

[sigmavirus24]

For those arriving here, this is not a frequently encountered issue, but the solution is in Nate's comment which I will quote here:

After a cursory look, I don’t think we can escape this at the adapter level. The immediate fix is to use the PreparedRequest workflow in the Requests docs and set the headers after the Request is prepared. This can be handed to the Session and should work.

I’d like to see how often hyper is being used with Requests still before we look at merging in a code change to bypass this.

Please read this and the documentation and do not post comments with "+1" or asking for updates. Further comments like this will result in locking the discussion