race condition when running https + http

[mikerott]

I'm using Nuclei as an embedded lib. This code snippet will give you an idea how I'm using it:

protocolAddresses := []string{}
for _, address := range addresses {
	protocolAddresses = append(protocolAddresses, fmt.Sprintf("http://%s", address))
	protocolAddresses = append(protocolAddresses, fmt.Sprintf("https://%s", address))
}

input := &inputs.SimpleInputProvider{Inputs: protocolAddresses}
_ = engine.Execute([]*templates.Template{&template}, input)
engine.WorkPool().Wait() // Wait for the scan to finish

The goal, of course, being that we automatically check http and https against every host or IP the caller passes.

In my integration tests, I have a server listening only on http (not https) that matches the test template in my integration test. Sometimes my test passes, sometimes it fails.

I'm reading down into the Nuclei code to see where/how this might be happening, and theorized that the HostErrorsCache might be configured incorrectly, but that does not seem to be the problem as it is configured with 30 MaxHostError. I thought also Engine.executeModelWithInput might be skipping, but it also does not appear to be the issue. I've put fmt.Print* statements in many places, but cannot figure out where the odd behavior originates.

For completeness, I'm using the hosterrorscache from Nuclei: cache := hosterrorscache.New(30, hosterrorscache.DefaultMaxHostsCount), setting it on the executorOpts, which of course gets passed to

Perhaps you can point me the right direction?

[mikerott]

I have a clue.

I put a fmt.Print* statement in /projectdiscovery/nuclei/v2/pkg/protocols/common/executor/executor.go.Execute in the if err != nil { code block. I see this output:

2022-07-11T21:18:17.584000+00:00 2022/07/11/[LATEST]717d0b11 MIKE: got err: GET https://integration-test-fake-servers/bitbucket/rest/applinks/1.0/manifest giving up after 1 attempts: Get "https://integration-test-fake-servers/bitbucket/rest/applinks/1.0/manifest": dial tcp 192.168.128.5:443: connect: connection refused
2022-07-11T21:18:17.587000+00:00 2022/07/11/[LATEST]717d0b11 MIKE: got err: GET http://integration-test-fake-servers/bitbucket/rest/applinks/1.0/manifest giving up after 1 attempts: Get "http://integration-test-fake-servers/bitbucket/rest/applinks/1.0/manifest": no address found for host

The first makes sense; integration-test-fake-servers is not listening on 443. The latter does not make sense; why would I get no address found for host?

So I studied deeper into Nuclei code, wondering what state the Dialer keeps. So I changed the line from opts.WithDialerHistory = true to opts.WithDialerHistory = false in common/protocolstate/state.go.Init and ran my test many many times. The test passed each time.

So there is some odd way dialer state is kept when the first attempt against the same host is a connection refused. I'll look deeper into it tomorrow, but would be glad to have advice on a workaround if you know of one.

[mikerott]

Some notes... I put some fmt.Print* in a few other places, but the important one being Dialer.dial. The only real difference I notice is the order of the resolvers. In the first lookup, we get an answer ("a"), but in the second lookup we do not.

2022-07-12T01:09:16.362000+00:00 2022/07/12/[LATEST]5c08bbcc MIKE: dial tcp, integration-test-fake-servers:443, true, false
2022-07-12T01:09:16.363000+00:00 2022/07/12/[LATEST]5c08bbcc MIKE: dial tcp, integration-test-fake-servers:80, false, false
2022-07-12T01:09:16.365000+00:00 2022/07/12/[LATEST]5c08bbcc MIKE: dataMarshaled1: {"host":"integration-test-fake-servers","resolver":["1.1.1.1:53","8.8.8.8:53","127.0.0.11:53"],"a":["172.22.0.5"],"soa":["a.root-servers.net","nstld.verisign-grs.com"],"raw":";; opcode: QUERY, status: NXDOMAIN, id: 18847\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 1232\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86400\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 18847\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 512\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86378\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NOERROR, id: 18847\n;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; ANSWER SECTION:\nintegration-test-fake-servers.\t600\tIN\tA\t172.22.0.5\n;; opcode: QUERY, status: NXDOMAIN, id: 18847\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 1232\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n\n;; AUTHORITY SECTION:\n.\t86400\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 18847\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 512\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n\n;; AUTHORITY SECTION:\n.\t86394\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NOERROR, id: 18847\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n","has_internal_ips":false,"status_code":"NOERROR","timestamp":"2022-07-12T01:09:15.959659633Z"}
2022-07-12T01:09:16.366000+00:00 2022/07/12/[LATEST]5c08bbcc MIKE: calling tls.DialWithDialer: tcp, 172.22.0.5:443
2022-07-12T01:09:16.368000+00:00 2022/07/12/[LATEST]5c08bbcc MIKE: dial tcp, integration-test-fake-servers:443, true, false
2022-07-12T01:09:16.370000+00:00 2022/07/12/[LATEST]5c08bbcc MIKE: dataMarshaled1: {"host":"integration-test-fake-servers","resolver":["1.1.1.1:53","8.8.8.8:53","127.0.0.11:53"],"a":["172.22.0.5"],"soa":["a.root-servers.net","nstld.verisign-grs.com"],"raw":";; opcode: QUERY, status: NXDOMAIN, id: 18847\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 1232\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86400\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 18847\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 512\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86378\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NOERROR, id: 18847\n;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; ANSWER SECTION:\nintegration-test-fake-servers.\t600\tIN\tA\t172.22.0.5\n;; opcode: QUERY, status: NXDOMAIN, id: 18847\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 1232\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n\n;; AUTHORITY SECTION:\n.\t86400\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 18847\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 512\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n\n;; AUTHORITY SECTION:\n.\t86394\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NOERROR, id: 18847\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n","has_internal_ips":false,"status_code":"NOERROR","timestamp":"2022-07-12T01:09:15.959659633Z"}
2022-07-12T01:09:16.371000+00:00 2022/07/12/[LATEST]5c08bbcc MIKE: calling tls.DialWithDialer: tcp, 172.22.0.5:443
2022-07-12T01:09:16.373000+00:00 2022/07/12/[LATEST]5c08bbcc MIKE: got err: GET https://integration-test-fake-servers/bitbucket/rest/applinks/1.0/manifest giving up after 1 attempts: Get "https://integration-test-fake-servers/bitbucket/rest/applinks/1.0/manifest": dial tcp 172.22.0.5:443: connect: connection refused
2022-07-12T01:09:16.374000+00:00 2022/07/12/[LATEST]5c08bbcc MIKE: dataMarshaled1: {"host":"integration-test-fake-servers","resolver":["1.0.0.1:53","8.8.4.4:53","1.1.1.1:53","8.8.8.8:53","127.0.0.11:53"],"soa":["a.root-servers.net","nstld.verisign-grs.com"],"raw":";; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 1232\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86400\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 512\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86399\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 1232\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86400\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 512\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86399\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 1232\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86400\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 1232\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n\n;; AUTHORITY SECTION:\n.\t86400\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 512\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n\n;; AUTHORITY SECTION:\n.\t86394\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 512\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n\n;; AUTHORITY SECTION:\n.\t86394\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NOERROR, id: 39610\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n","has_internal_ips":false,"status_code":"NOERROR","timestamp":"2022-07-12T01:09:16.127430216Z"}
2022-07-12T01:09:16.376000+00:00 2022/07/12/[LATEST]5c08bbcc MIKE: dial tcp, integration-test-fake-servers:80, false, false
2022-07-12T01:09:16.377000+00:00 2022/07/12/[LATEST]5c08bbcc MIKE: dataMarshaled1: {"host":"integration-test-fake-servers","resolver":["1.0.0.1:53","8.8.4.4:53","1.1.1.1:53","8.8.8.8:53","127.0.0.11:53"],"soa":["a.root-servers.net","nstld.verisign-grs.com"],"raw":";; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 1232\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86400\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 512\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86399\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 1232\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86400\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 512\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86399\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 1232\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t A\n\n;; AUTHORITY SECTION:\n.\t86400\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 1232\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n\n;; AUTHORITY SECTION:\n.\t86400\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 512\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n\n;; AUTHORITY SECTION:\n.\t86394\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NXDOMAIN, id: 39610\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 512\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n\n;; AUTHORITY SECTION:\n.\t86394\tIN\tSOA\ta.root-servers.net. nstld.verisign-grs.com. 2022071102 1800 900 604800 86400\n;; opcode: QUERY, status: NOERROR, id: 39610\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0\n\n;; QUESTION SECTION:\n;integration-test-fake-servers.\tIN\t AAAA\n","has_internal_ips":false,"status_code":"NOERROR","timestamp":"2022-07-12T01:09:16.127430216Z"}
2022-07-12T01:09:16.379000+00:00 2022/07/12/[LATEST]5c08bbcc MIKE: got err: GET http://integration-test-fake-servers/bitbucket/rest/applinks/1.0/manifest giving up after 1 attempts: Get "http://integration-test-fake-servers/bitbucket/rest/applinks/1.0/manifest": no address found for host

[mikerott]

At this point I'm wondering if the hm *hybrid.HybridMap in fastdialer/dialer.go.Dialer is thread (errr "go routine") safe, or if the cache needs more protection against bad entries.

I've added some print statements, like:

	// check if data is in cache
	data, err := d.GetDNSData(hostname)
	if err != nil {
		// otherwise attempt to retrieve it
		data, err = d.dnsclient.Resolve(hostname)
	} else if data != nil {
		fmt.Printf("MIKE %s: it was in the cache as %v or %v\n", mikePrefix, data.A, data.AAAA)
	}

And the output occasionally shows:

2022-07-12T16:38:38.631000+00:00 2022/07/12/[LATEST]db5e9a81 MIKE reg: it was in the cache as [] or []

... which implies the cache has been populated with an entry that lacks an Answer (A or AAAA record), despite being resolvable.

Which of course results in returning NoAddressFoundError despite the hostname having just been resolved in another go routine handling the https case.

FWIW, my testing is inside a lambda, inside a localstack container, inside docker-compose. I've opened a proposal PR: projectdiscovery/fastdialer#51