Full URL is sent to DNS resolver

[TheTechromancer]

Hello,

Recently in our testing we have noticed that when provided URLs, httpx sometimes fails to resolve DNS hosts:

Inspecting the traffic in wireshark reveals that httpx is passing the full URL to the DNS server:

[ehsandeep]

@TheTechromancer thanks a lot for pointing this out, we were already investigating something which might be related to this (not confirmed).

[Mzack9999]

@TheTechromancer Thanks for opening the issue. I can confirm that the five consecutive requests with the same request-id of type A and AAAA seem to match the retryabledns default settings. Anyway, a few things are unclear. For example, the full URL https://www.example.com:80/index.html, from the command screenshot you provided, was never piped into httpx. Also, the AAAA shows two times OPT OPT. Could you confirm if the provided command is the exact one associated with the Wireshark screenshot?

[TheTechromancer]

The target corresponding to the wireshark screenshot was https://www.example.com/index.html, as shown in the capture. These were two separate runs; I mixed them up. So the command that generated the wireshark screenshot was:

echo 'http://www.example.com/index.html' | httpx -debug -json -r <dns server>

Just now I went back to verify this, and strangely I am no longer seeing the URL in the capture. I'm unsure why this is, since I'm using the same binary as before (version 1.2.3), but the DNS requests still seem to fail against the custom resolver.

[Mzack9999]

I suspect the problem might be between https://github.com/projectdiscovery/retryabledns and the custom resolver. Would it be possible to provide more information about it: DNS server software used? Any particular configuration?

[TheTechromancer]

Sure. It is an Active Directory domain controller running Windows Server 2012 R2 with the DNS server role and a default configuration. The forwarder (upstream DNS server) is set to 8.8.8.8.

[Mzack9999]

On Hold - Not reproducible, will need a custom setup similar to the described scenario.

[Mzack9999]

The issue can't be reproduced - It seems more related to the OS configuration than the go app itself. By default, we already use system resolvers in retryabledns.

[gitevildelta]

The issue can't be reproduced - It seems more related to the OS configuration than the go app itself. By default, we already use system resolvers in retryabledns.

I had the same issue with the latest version of httpx (1.3.9), curl was working fine but not httpx, I fixed that by using -r 127.0.0.53 to use systemd-resolved resolver (which should be the system resolver)