Replace `requests` with `httpx` #2576

ammar92 · 2024-02-29T12:31:43Z

Changes

This implementation replaces the requests package with httpx where possible. The following packages have also been replaced:

pytest-requests with httpx-requests
- This change improves compatibility with httpx and enables better testing of requests
- For this a lot of tests have been rewritten
- Most of the Rocky mocks for requests have also been replaced which should lead to less code and better readable and maintainable tests
opentelemetry-instrumentation-requests with opentelemetry-instrumentation-httpx

Issue link

Closes #2299

Testing

Testing this requires just clicking around, making sure tasks start and produce results and KATalogus pages, health page and objects lists work.

Code Checklist

All the commits in this PR are properly PGP-signed and verified.
This PR only contains functionality relevant to the issue; tickets have been created for newly discovered issues.
I have written unit tests for the changes or fixes I made.
For any non-trivial functionality, I have added integration and/or end-to-end tests.
I have performed a self-review of my code and refactored it to the best of my abilities.

Communication

I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
I have made corresponding changes to the documentation, if necessary.
I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

boefjes/boefjes/clients/bytes_client.py

boefjes/boefjes/plugins/kat_rpki/main.py

keiko/pyproject.toml

mula/scheduler/connectors/services/bytes.py

mula/scheduler/connectors/services/services.py

rocky/rocky/scheduler.py

rocky/rocky/settings_test.py

rocky/rocky/bytes_client.py

mula/scheduler/connectors/connector.py

…tarlette's test client

stephanie0x00 · 2024-03-13T10:37:58Z

During QA the following errors were triggered:

Clicking on the Findings page

Environment:


Request Method: GET
Request URL: http://127.0.0.1:8000/en/ee/findings/

Django Version: 4.2.11
Python Version: 3.11.8
Installed Applications:
['whitenoise.runserver_nostatic',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.forms',
 'django_otp',
 'django_otp.plugins.otp_static',
 'django_otp.plugins.otp_totp',
 'two_factor',
 'account',
 'tools',
 'fmea',
 'crisis_room',
 'onboarding',
 'katalogus',
 'django_password_validators',
 'django_password_validators.password_history',
 'rest_framework',
 'tagulous',
 'compressor',
 'reports',
 'knox',
 'csp']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'whitenoise.middleware.WhiteNoiseMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django_otp.middleware.OTPMiddleware',
 'rocky.middleware.auth_required.AuthRequiredMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'rocky.middleware.onboarding.OnboardingMiddleware',
 'csp.middleware.CSPMiddleware']



Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
    return handler(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/list.py", line 174, in get
    context = self.get_context_data()
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/tools/view_helpers.py", line 91, in get_context_data
    context = super().get_context_data(**kwargs)  # type: ignore[misc]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/rocky/views/finding_list.py", line 73, in get_context_data
    context = super().get_context_data(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/account/mixins.py", line 101, in get_context_data
    context = super().get_context_data(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/list.py", line 128, in get_context_data
    paginator, page, queryset, is_paginated = self.paginate_queryset(
                                              
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/list.py", line 73, in paginate_queryset
    page = paginator.page(page_number)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 72, in page
    number = self.validate_number(number)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 53, in validate_number
    if number > self.num_pages:
                ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 99, in num_pages
    if self.count == 0 and not self.allow_empty_first_page:
       ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 91, in count
    c = getattr(self.object_list, "count", None)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/rocky/views/mixins.py", line 215, in count
    return self.octopoes_connector.list_findings(
           
  File "/app/rocky/octopoes/connector/octopoes.py", line 238, in list_findings
    res = self.session.get(f"/{self.client}/findings", params=params)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1054, in get
    return self.request(
           
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
    raise exc
    ^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
    hook(response)
    ^^^^^^^^^^^^^^
  File "/app/rocky/octopoes/connector/octopoes.py", line 54, in _verify_response
    raise error
    ^^^^^^^^^^^
  File "/app/rocky/octopoes/connector/octopoes.py", line 46, in _verify_response
    response.raise_for_status()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 761, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Exception Type: HTTPStatusError at /en/ee/findings/
Exception Value: Client error '422 Unprocessable Entity' for url 'http://octopoes_api/ee/findings?valid_time=2024-03-13%2010%3A13%3A38.858925%2B00%3A00&offset=0&limit=0&severities=set%28%29&exclude_muted=true&only_muted=false'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/422

Clicking on the Tasks page

Environment:


Request Method: GET
Request URL: http://127.0.0.1:8000/en/ee/tasks/

Django Version: 4.2.11
Python Version: 3.11.8
Installed Applications:
['whitenoise.runserver_nostatic',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.forms',
 'django_otp',
 'django_otp.plugins.otp_static',
 'django_otp.plugins.otp_totp',
 'two_factor',
 'account',
 'tools',
 'fmea',
 'crisis_room',
 'onboarding',
 'katalogus',
 'django_password_validators',
 'django_password_validators.password_history',
 'rest_framework',
 'tagulous',
 'compressor',
 'reports',
 'knox',
 'csp']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'whitenoise.middleware.WhiteNoiseMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django_otp.middleware.OTPMiddleware',
 'rocky.middleware.auth_required.AuthRequiredMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'rocky.middleware.onboarding.OnboardingMiddleware',
 'csp.middleware.CSPMiddleware']



Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
    return handler(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/list.py", line 174, in get
    context = self.get_context_data()
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/rocky/views/tasks.py", line 83, in get_context_data
    context = super().get_context_data(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/account/mixins.py", line 101, in get_context_data
    context = super().get_context_data(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/list.py", line 128, in get_context_data
    paginator, page, queryset, is_paginated = self.paginate_queryset(
                                              
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/list.py", line 73, in paginate_queryset
    page = paginator.page(page_number)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 72, in page
    number = self.validate_number(number)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 53, in validate_number
    if number > self.num_pages:
                ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 99, in num_pages
    if self.count == 0 and not self.allow_empty_first_page:
       ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 91, in count
    c = getattr(self.object_list, "count", None)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/rocky/scheduler.py", line 138, in count
    self._count = self.scheduler_client.list_tasks(
                  
  File "/app/rocky/rocky/scheduler.py", line 199, in list_tasks
    return PaginatedTasksResponse.model_validate_json(res.content)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pydantic/main.py", line 538, in model_validate_json
    return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Exception Type: ValidationError at /en/ee/tasks/
Exception Value: 2 validation errors for PaginatedTasksResponse
count
  Field required [type=missing, input_value={'detail': [{'type': 'dat...me_from_date_parsing'}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.6/v/missing
results
  Field required [type=missing, input_value={'detail': [{'type': 'dat...me_from_date_parsing'}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.6/v/missing

When adding a Hostname object manually:

Environment:


Request Method: GET
Request URL: http://127.0.0.1:8000/en/ee/objects/detail/?ooi_id=Hostname%7Cinternet%7Cbbc.co.uk

Django Version: 4.2.11
Python Version: 3.11.8
Installed Applications:
['whitenoise.runserver_nostatic',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.forms',
 'django_otp',
 'django_otp.plugins.otp_static',
 'django_otp.plugins.otp_totp',
 'two_factor',
 'account',
 'tools',
 'fmea',
 'crisis_room',
 'onboarding',
 'katalogus',
 'django_password_validators',
 'django_password_validators.password_history',
 'rest_framework',
 'tagulous',
 'compressor',
 'reports',
 'knox',
 'csp']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'whitenoise.middleware.WhiteNoiseMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django_otp.middleware.OTPMiddleware',
 'rocky.middleware.auth_required.AuthRequiredMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'rocky.middleware.onboarding.OnboardingMiddleware',
 'csp.middleware.CSPMiddleware']



Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
    ^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 233, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
    ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
               
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
    ^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 122, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
    ^^^^^^^^^^^^^^^^^^^^^^^^^^

The above exception ([Errno -3] Temporary failure in name resolution) was the direct cause of the following exception:
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
    return handler(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/rocky/views/ooi_detail_related_object.py", line 79, in get
    return super().get(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/rocky/views/ooi_view.py", line 105, in get
    return super().get(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 226, in get
    context = self.get_context_data(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/rocky/views/ooi_detail.py", line 199, in get_context_data
    context["task_history"] = self.get_task_history()
                              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/rocky/views/ooi_detail.py", line 142, in get_task_history
    return Paginator(task_history, self.task_history_limit).page(page)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 72, in page
    number = self.validate_number(number)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 53, in validate_number
    if number > self.num_pages:
                ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 99, in num_pages
    if self.count == 0 and not self.allow_empty_first_page:
       ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 91, in count
    c = getattr(self.object_list, "count", None)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/rocky/scheduler.py", line 138, in count
    self._count = self.scheduler_client.list_tasks(
                  
  File "/app/rocky/rocky/scheduler.py", line 198, in list_tasks
    res = self._client.get("/tasks", params=kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1054, in get
    return self.request(
           
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 232, in handle_request
    with map_httpcore_exceptions():
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Exception Type: ConnectError at /en/ee/objects/detail/
Exception Value: [Errno -3] Temporary failure in name resolution

Clicking on the 'Details' page for the nmap boefje in the katalogus:

Environment:


Request Method: GET
Request URL: http://127.0.0.1:8000/en/ee/kat-alogus/plugins/boefje/nmap/

Django Version: 4.2.11
Python Version: 3.11.8
Installed Applications:
['whitenoise.runserver_nostatic',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.forms',
 'django_otp',
 'django_otp.plugins.otp_static',
 'django_otp.plugins.otp_totp',
 'two_factor',
 'account',
 'tools',
 'fmea',
 'crisis_room',
 'onboarding',
 'katalogus',
 'django_password_validators',
 'django_password_validators.password_history',
 'rest_framework',
 'tagulous',
 'compressor',
 'reports',
 'knox',
 'csp']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'whitenoise.middleware.WhiteNoiseMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django_otp.middleware.OTPMiddleware',
 'rocky.middleware.auth_required.AuthRequiredMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'rocky.middleware.onboarding.OnboardingMiddleware',
 'csp.middleware.CSPMiddleware']



Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
    ^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 233, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
    ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
               
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
    ^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 122, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
    ^^^^^^^^^^^^^^^^^^^^^^^^^^

The above exception ([Errno -3] Temporary failure in name resolution) was the direct cause of the following exception:
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/katalogus/views/mixins.py", line 57, in dispatch
    return super().dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
    return handler(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/katalogus/views/plugin_settings_list.py", line 27, in get
    return super().get(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 226, in get
    context = self.get_context_data(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/katalogus/views/plugin_detail.py", line 137, in get_context_data
    context = super().get_context_data(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/katalogus/views/plugin_detail.py", line 92, in get_context_data
    context["task_history"] = self.get_task_history()
                              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/katalogus/views/plugin_detail.py", line 72, in get_task_history
    return Paginator(task_history, self.task_history_limit).page(page)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 72, in page
    number = self.validate_number(number)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 53, in validate_number
    if number > self.num_pages:
                ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 99, in num_pages
    if self.count == 0 and not self.allow_empty_first_page:
       ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/paginator.py", line 91, in count
    c = getattr(self.object_list, "count", None)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/rocky/scheduler.py", line 138, in count
    self._count = self.scheduler_client.list_tasks(
                  
  File "/app/rocky/rocky/scheduler.py", line 198, in list_tasks
    res = self._client.get("/tasks", params=kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1054, in get
    return self.request(
           
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 232, in handle_request
    with map_httpcore_exceptions():
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Exception Type: ConnectError at /en/ee/kat-alogus/plugins/boefje/nmap/
Exception Value: [Errno -3] Temporary failure in name resolution

… into fix/replace-requests-with-httpx

dekkers

I missed it the first time, but httpx still needs to be added to the main dependencies in boefjes.

boefjes/boefjes/clients/bytes_client.py

mula/scheduler/connectors/connector.py

mula/scheduler/schedulers/boefje.py

octopoes/octopoes/api/api.py

octopoes/octopoes/api/router.py

rocky/rocky/views/ooi_list.py

rocky/rocky/views/organization_member_list.py

rocky/rocky/views/upload_raw.py

rocky/tools/models.py

stephanie0x00 · 2024-03-14T15:11:35Z

Checklist for QA:

I have checked out this branch, and successfully ran a fresh make reset.
I confirmed that there are no unintended functional regressions in this branch:
- I have managed to pass the onboarding flow
- Objects and Findings are created properly
- Tasks are created and completed properly
I confirmed that the PR's advertised feature or hotfix works as intended.

What works:

Looks good! No error messages are shown and the logs look fine too. Ready for merge.

What doesn't work:

n/a

Bug or feature?:

n/a

…rences

rocky/rocky/views/upload_raw.py

octopoes/octopoes/xtdb/client.py

mula/scheduler/schedulers/normalizer.py

ammar92 added 24 commits February 29, 2024 11:44

Updated cveapi

8596a1a

Removed httpx as a direct dependency from keiko

184472f

Removed requests from mula and replaced it with httpx

86788b4

Added httpx to keiko as this is required by Starlette's TestClient

525ecaa

Fixed tests errors in mula

574df2b

Removed requests as direct dependency and replaced it with httpx

93bd48d

Fixed bytes integration tests

9fb8055

Merge branch 'main' into fix/replace-requests-with-httpx

a658839

Merge branch 'main' into fix/replace-requests-with-httpx

bf2473f

Removed real_http assignment

71c047f

Updated requirements for Octopoes

f4128da

Refactored and updated tests

d599ca3

Fixed tests

56fb0ac

Reverted changes

2f6b15e

Merge branch 'main' into fix/replace-requests-with-httpx

2b074a5

Possible fix

aaa71e9

Merge branch 'main' into fix/replace-requests-with-httpx

068f9d7

Updated dependencies for Rocky

959b49c

Refactored Rocky to use httpx

0ba3b2e

Added pytest-httpx for Rocky

a1c0a21

Some more requests removals

1130e2c

Updated and fixed some Rocky tests

8e7aeb5

Refactored and fixed tests

a7e1218

Merge branch 'main' into fix/replace-requests-with-httpx

3a000b5

ammar92 marked this pull request as ready for review March 8, 2024 11:21

ammar92 requested a review from a team as a code owner March 8, 2024 11:21

dekkers reviewed Mar 11, 2024

View reviewed changes

ammar92 added 3 commits March 12, 2024 09:39

Merge branch 'main' into fix/replace-requests-with-httpx

538ed44

Removed unneeded dependencies from Keiko

d2d31ff

Some rework and some removed some debug leftovers

43b9d53

Added httpx to Keiko's dev requirements as this is a dependency for S…

1a23831

…tarlette's test client

ammar92 mentioned this pull request Mar 12, 2024

Improve testing, coverage and benchmarking #2534

Closed

8 tasks

ammar92 added 3 commits March 12, 2024 13:30

Some rework

6cbf339

Fixed unit test in Rocky

2b756c6

Merge branch 'main' into fix/replace-requests-with-httpx

8902759

ammar92 added 11 commits March 14, 2024 10:05

Merge branch 'main' into fix/replace-requests-with-httpx

e454a6e

Fixes

c600282

Fixed findings page

d25c3bf

Fixed tasks page

7abf135

Merge branch 'main' into fix/replace-requests-with-httpx

731f1cf

Fixed onboarding

32b4ed5

Merge remote-tracking branch 'origin/fix/replace-requests-with-httpx'…

800cd56

… into fix/replace-requests-with-httpx

Merge branch 'main' into fix/replace-requests-with-httpx

2aa9732

Fixed some clients

8f3a26c

Merge remote-tracking branch 'origin/fix/replace-requests-with-httpx'…

228e003

… into fix/replace-requests-with-httpx

Added timeout for Octopoes connector

3d13fb2

dekkers reviewed Mar 14, 2024

View reviewed changes

ammar92 added 4 commits March 14, 2024 15:28

Fixed bytes client

2496dbc

Added httpx as a root dependency for Boefje

6bec69e

Some more rework

eaeee48

Updated language file

52e68f4

Added httpx to the mypy dependencies list and updated some old refe…

c809510

…rences

dekkers reviewed Mar 15, 2024

View reviewed changes

rocky/rocky/views/upload_raw.py Outdated Show resolved Hide resolved

octopoes/octopoes/xtdb/client.py Outdated Show resolved Hide resolved

octopoes/octopoes/xtdb/client.py Outdated Show resolved Hide resolved

mula/scheduler/schedulers/normalizer.py Outdated Show resolved Hide resolved

Rework

b35eaed

dekkers approved these changes Mar 15, 2024

View reviewed changes

dekkers merged commit 65e5e14 into main Mar 15, 2024
40 checks passed

dekkers deleted the fix/replace-requests-with-httpx branch March 15, 2024 14:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace `requests` with `httpx` #2576

Replace `requests` with `httpx` #2576

ammar92 commented Feb 29, 2024 •

edited

Loading

stephanie0x00 commented Mar 13, 2024

dekkers left a comment

stephanie0x00 commented Mar 14, 2024

Replace requests with httpx #2576

Replace requests with httpx #2576

Conversation

ammar92 commented Feb 29, 2024 • edited Loading

Changes

Issue link

Testing

Code Checklist

Communication

Checklist for code reviewers:

Checklist for QA:

stephanie0x00 commented Mar 13, 2024

Clicking on the Findings page

Clicking on the Tasks page

When adding a Hostname object manually:

Clicking on the 'Details' page for the nmap boefje in the katalogus:

dekkers left a comment

Choose a reason for hiding this comment

stephanie0x00 commented Mar 14, 2024

Checklist for QA:

What works:

What doesn't work:

Bug or feature?:

Replace `requests` with `httpx` #2576

Replace `requests` with `httpx` #2576

ammar92 commented Feb 29, 2024 •

edited

Loading