Randomize the order of DNS servers to prevent reaching the limitations of public resolvers

[serge-r]

Is there an existing issue that is already proposing this?

• I have searched the existing issues

Application

Outline Client

What are you trying to do? What is your use case?

Hello.

I understand that it might seem odd, but I am conducting research on mobile app issues related to Outline VPN, Apple, and Cloudflare's DNS over TLS.

You can find many details along with logs here: https://community.cloudflare.com/t/1-1-1-1-1-0-0-1-dns-over-tls-limitations/643561
I am still awaiting a response from the Cloudflare Community, but it seems like it might take some time.

For those who are unable to follow the link I provided, here is a TL;DR:

When Outline is connected, MacOS (and iOS too) starts using DNS over TLS with Cloudflare servers, because 1.1.1.1 is listed first.

My theory is that Cloudflare has some per-IP limits for DNS DoT queries, and when these limits are reached, I receive a REFUSED response for any domain.

The problem with MacOS X and iOS (which use the same system resolver service, mDNSResponder) when DoT is enabled, is that this negative response is cached for a period, causing all apps that use the system resolver to fail. As a result, retries and other DNS servers configured in the system do not work. This is a significant problem for mobile users, as mobile apps can generate tons of DNS traffic, and iOS users of our mobile app who use Outline are experiencing issues with the application.

I have an Apple developer certificate, and I was able to build and test the Outline Client with different DNS servers. The problems appear only with 1.1.1.1. It seems that changing the order of servers could help us resolve many issues for mobile users.

Im not sure about Windows and Android users, because I don't have devices for testing, but I think would be better to use same behavior on all platforms.

Is your feature request related to a problem? Please describe it.

Problem not with the Outline but with Cloudflare, details are here: https://community.cloudflare.com/t/1-1-1-1-1-0-0-1-dns-over-tls-limitations/643561

Describe the solution you'd like.

Would be better to have random order of DNS servers here

Describe alternatives you've considered

An alternative way is to move 1.1.1.1 from the first position to another, as I did not reproduce the issue with other public resolvers. But Im not sure about others limitations.

[cornzzy]

Could also set from dynamic access links

[serge-r]

@cornzzy yeah it's a variant too, but anyway I need to redefine default behavior because a lot of clients are using defaults and 1.1.1.1 only as a first resolver

[RomanIzvozchikov]

My colleagues and I have the same problem. It is really annoying bug. Investigation results are the same as @serge-r described.

We faced this problem on all our Mac computers. The problem occurs on any application that uses mDNSResponder (standard way in MacOS to resolve DNS name).

Interesting fact: we faced this problem on Safari browser (it uses mDNSResponder), but Chrome works better, because it does not rely on mDNSResponder and uses its own DNS resolving mechanism.

So here we have three problems:

1. Cloudflare has some not documented policy that sometimes refuses DNS requests.
2. mDNSResponder written by Apple does not work as expected and does not use other provided DNS servers as a failover mechanism in case of refusing DNS request by Cloudflare.
3. We cannot change DNS server to another one that will work better than hardcoded one in Outline VPN app.

Probably the best way to solve this problem is to add to Outline client app the possibility to set DNS servers by user.

UPD: I made additional investigation regarding refused DNS requests. I tried to use 1.1.1.1 DNS server without enabled VPN connection (using an IP address of my internet provider) and I didn't faced the refused DNS requests problem.
After that I enabled my Outline VPN connection and I faced the problem soon.
My Outline VPN server is deployed in AWS.
So from my perspective Cloudflare recognises AWS IP addresses ranges as suspicious and applies more stict policy.

@daniellacosse probably it is not a good choice to hardcode DNS server address that applies more restrictive policy to AWS IP address ranges (the most popular public cloud service).

UPD2: I found a workaround of this problem. I overrode DNS settings set by Outline VPN on my Mac. It is described in the article how to do this.

My version of a script described in the article.

#!/bin/bash
 
sudo scutil << EOF
get State:/Network/Service/033FC8AB-2300-484E-800A-0112945C3D63/DNS
d.add ServerAddresses * 8.8.8.8
set State:/Network/Service/033FC8AB-2300-484E-800A-0112945C3D63/DNS
exit
EOF

Then I created an alias to run this script easier.
Not really convenient, but acceptable as a workaround.

Hope DNS select feature will be released soon!

UPD3: I faced this problem on my iPhone. In some random time some random pages cannot be opened. After turning off VPN it worked. I don't have any proofs, because I don't know how to troubleshoot iPhone but I am sure that it is the same problem, because iOS devices also use mDNSResolver.

@serge-r thank you for your work!