serve-expired does not adhere to secure-by-default principle

[pimlie]

Describe the bug

If you just set serve-expired: yes in your config, then unbound could serve stale dns entries for ever as the default value for serve-expired-ttl is 0. This is not a secure configuration.

To reproduce
n.a.

Expected behavior

unbound should also follow suggestions from RFC-8767 and not just recommendation and therefore set a non-zero default value for serve-expired-ttl. The decision to serve expired data forever should be an explicit decision by the user as they put themselves at risk by doing so. See below for why.

System:
n.a.

Additional information

1. Security issue

As background and to explain why this should probably be considered a security issue, I was bit by this last week when I was performing an online payment when my browser stopped the request as I was about to be redirected to an unsecure page/domain.
After investigating I noticed that the SSL certificate of the page I was on did not belong to the PSP that was handling the payment but some random other third party. After contacting the PSP they mentioned that they had reason to believe this was caused by DNS caching. Although I didn't believe them at first, after diving into the unbound config more I'm quite sure they were right (sry PSP).
Looking at the unbound logs, I had last visited that PSP checkout page something like a month ago and I guess that when I first requested the PSP's checkout page last week that unbound served the cached DNS response from last month. But it seems that in between last month and last week, the PSP scaled down/changed their capacity causing the resources they used last month to not be used anymore. And their cloud provider had allocated those resources to another third party.

TLDR: this is a security issue as due to this default unbound configuration I almost submitted personal / payment information to some random third party as unbound served me a month old DNS record. Luckily my browser stopped me from doing that.

2. RFC

Also, I wonder whether unbound does really adhere to RFC-8767. Unbound says about enabling serve-expired that it attempts to serve old responses ... without waiting for the actual resolution to finish. But section 7 of the RFC says that a good-faith effort has been recently made to refresh the stale data before it is delivered to any client.
I might misunderstand the RFC here, but it seems the RFC says an attempt to refresh has to be made while unbound immediately sends stale data without trying to refresh? I guess that unbound could better adhere to the RFC by also setting a non-zero value for serve-expired-client-timeout by default.

3. Mitigation

Instead or in addition to setting serve-expired-ttl, maybe a setting like serve-expired-ttl-factor could be added. This might be outside the scope of RFC-8767, but the party who knows best how long DNS data should be cached is the owner of the DNS recordset who already sets a TTL themselves. And setting a fixed duration for serve-expired-ttl seems wrong as it doesn't take the intentions of the original DNS owner into account. A setting like serve-expired-ttl-factor: 10 could be added to work next to serve-expired-ttl, where serve-expired-ttl is the maximum value and serve-expired-ttl-factor means that unbound only serves stale cache data for the configured factor value times the TTL that the owner of the DNS record configured.

F.e. setting a config like:

serve-expired-ttl: 86400
serve-expired-ttl-factor: 10

could mean that when requesting a DNS record with a configured TTL of 60 seconds would then only be cached for 10 * 60 = 600 seconds while a DNS record with a TTL of 10.000 would be cached for 86400 seconds (and thus not 10 * 10.000 = 100.000)

[gthess]

Hi Pim,
Thanks for bringing this up!

serve-expired itself is not a default setting. It was even there before the RFC and Unbound would reply straight from cache before trying to update the record. This was/is desired in certain environments where the ecosystem is somewhat controlled.

When implementing RFC 8767 we purposely left the serve-expired-client-timeout to 0 to not break the default behavior of the serve-expired logic. Now that people know of the existence of the new configuration options for quite some time I believe it is the right time to use the recommended 1800 value by default.

Serving expired answers is not ideal because the upstream explicitly communicates the TTL of the original record. Serving expired answers is mostly for controlled environments or big installations where the expired answers are used to push the qps up.

For regular resolving I would advise to turn it off and instead use prefetch to try and keep fresh records in the cache.

[pimlie]

Hey Yorgos, yeah this was definitely my own mistake for just using 'some' docker container and trusting that container was using a sane / secure configuration without taking a good look at it's configuration myself. So the only one to blame for that is me 😿
That said, it would probably also helped if NLnet Labs would publish an official docker container with recommended config. Would be awesome if y'all could consider that ;)

One other thought I had about serve-expired-client-timeout, it would maybe be nice if unbound could track & report the average resolve time for upstream requests. Cause just setting serve-expired-client-timeout to some value feels quite arbitrary. Even the recommended value of 1.8s in the RFC seems somewhat arbitrary, for a proper functioning cache that 1.8s seems much more like an upper limit then a meaningful value for serve-expired-client-timeout as in a normal functioning environment I would expect the average resolve time to be much lower then 1.8s.
So if unbound could report like that upstream DNS requests would on average resolve in M time with N sigma then serve-expired-client-timeout could probably be tweaked to an meaningful value for each environment it runs in 🙏

[AlexanderBand]

Hey Yorgos, yeah this was definitely my own mistake for just using 'some' docker container and trusting that container was using a sane / secure configuration

On the Docker topic, I hear good stories about the one maintained by @madnuttah.
https://github.com/madnuttah/unbound-docker

[pimlie]

On the Docker topic, I hear good stories about the one maintained by @madnuttah. https://github.com/madnuttah/unbound-docker

An (semi) official recommendation would be welcome too, will look at that docker image. Thanks!

I'm currently using https://hub.docker.com/r/klutchell/unbound (see also mentioned issue above) because his build/config seemed ok as well and that image has 10M+ pulls on docker hub as opposed to 50K+ for the image by @madnuttah. I remember looking at https://hub.docker.com/r/mvance/unbound too, but iirc correctly then klutchell's image is very small / distribution less while mvance's isn't.

[gthess]

One other thought I had about serve-expired-client-timeout, it would maybe be nice if unbound could track & report the average resolve time for upstream requests. Cause just setting serve-expired-client-timeout to some value feels quite arbitrary. Even the recommended value of 1.8s in the RFC seems somewhat arbitrary, for a proper functioning cache that 1.8s seems much more like an upper limit then a meaningful value for serve-expired-client-timeout as in a normal functioning environment I would expect the average resolve time to be much lower then 1.8s. So if unbound could report like that upstream DNS requests would on average resolve in M time with N sigma then serve-expired-client-timeout could probably be tweaked to an meaningful value for each environment it runs in 🙏

There are such numbers! Look for recursion.time and histogram in the unbound-control manpage or online for the latest version.

But I think you are holding it wrong :)
The value of 1.8s was selected to be less than the usual amount of time a client will usually wait for a reply before retrying (2 seconds). Not for the average recursion time. So by the time they are almost ready to give up, Unbound will serve them an expired entry instead. This is ofcourse data based on observation and indeed in controlled environments where you expect an upstream answer in X seconds (think datacenters), you can configure serve-expired-client-timeout to match your expectation, because anything slower means trouble somewhere.

[Dynamic5912]

If you're going into the config to set serve-expired to yes - it doesn't take much to change the serve-expired-ttl at the same time whilst in the config?

I don't see this as a "real" issue, as such.

[pimlie]

@AlexanderBand FWIW, madnuttah also sets serve-expired: yes without setting serve-expired-ttl in it's example config which madnuttah says he uses himself. The default config in the docker image does indeed seem to use serve-expired: no.

@gthess Thanks for the tip about unbound-control, will take a look :)

@Dynamic5912 That's not the point, you make an assumption that people already know that they have to configure serve-expired-ttl because otherwise they put themselves at risk. If you read this thread then you'll see that at least 2 docker maintainers and their thousands of users were more then likely not aware of that. Also please read up on the secure-by-design/default principle published by CISA and partners :)

[Dynamic5912]

If people are configuring Unbound they should have the knowledge and know-how to understand what they're configuring and enabling.

[madnuttah]

FWIW, madnuttah also sets serve-expired: yes without setting serve-expired-ttl in it's example config which madnuttah says he uses himself. The default config in the docker image does indeed seem to use serve-expired: no.

Thank you for your heads-up! I'll fix that asap. As you've mentioned, the default config doesn't serve expired entries.

[Dynamic5912]

Hi Pim,

Thanks for bringing this up!

serve-expired itself is not a default setting. It was even there before the RFC and Unbound would reply straight from cache before trying to update the record. This was/is desired in certain environments where the ecosystem is somewhat controlled.

When implementing RFC 8767 we purposely left the serve-expired-client-timeout to 0 to not break the default behavior of the serve-expired logic. Now that people know of the existence of the new configuration options for quite some time I believe it is the right time to use the recommended 1800 value by default.

Serving expired answers is not ideal because the upstream explicitly communicates the TTL of the original record. Serving expired answers is mostly for controlled environments or big installations where the expired answers are used to push the qps up.

For regular resolving I would advise to turn it off and instead use prefetch to try and keep fresh records in the cache.

Wouldn't having the cache TTL as 0 and using Prefetch negate any issues though?

Prefetch would (should) update those records in the cache as they are technically expired?

[gthess]

Prefetch would (should) update those records in the cache as they are technically expired?

I am not sure I follow but prefetch and serve-expire trigger the same fetching logic. Prefetch does it when a currently non-expired reply is in the last 10% of the original TTL, serve-expired does it when the record is expired. Both try to update the cache. So if you only have prefetch enabled, then an expired record will not be used and instead resolved normally.

[Dynamic5912]

Is serve-expired-ttl still staying with a default value of "0" with this new change?

[gthess]

Good question.
Any value you are going to choose there is already violating the lifetime of a record so there is technically no difference between infinity and some value. But since the RFC suggests "1 to 3 days" maybe now it's a good time to update that value, only for being close to the RFC suggestion. I'll see if we miss something expectation-wise if we change the default, otherwise the change will land on the same version as the serve-expired-client-timeout default change.

[Dynamic5912]

Good question.

Any value you are going to choose there is already violating the lifetime of a record so there is technically no difference between infinity and some value. But since the RFC suggests "1 to 3 days" maybe now it's a good time to update that value, only for being close to the RFC suggestion. I'll see if we miss something expectation-wise if we change the default, otherwise the change will land on the same version as the serve-expired-client-timeout default change.

Most people set a time of 1 day if you look at various forums/posts/blogs about setting up Unbound.

So I think 1 day would be a good baseline default setting.

[pimlie]

So I think 1 day would be a good baseline default setting.

Agreed, secure-by-default also means to stay on the safe side (of recommendations) which in this case is the lower value. That said, I still think that adding both an upper bound as taking the actual TTL value of the DNS record into account could be worthwhile (ie the above suggested serve-expired-ttl-factor setting).

[pimlie]

Also as mentioned above, the serve-expired-ttl-factor setting could be a way to tune the system according to the preferences by the domain holder. In my case with the PSP, they use a TTL of ~60s because they use a fully automatic scaling cloud solution and only block previously used resources for a couple of hours before releasing them. So even setting serve-expired-ttl to 1 day could result in an outdated DNS record being served.

I just don't think that many people are already really aware of RFC-8767 and it's implications for scalable cloud solutions. But those same people are probably aware of general potential DNS issues, so they likely already have set a reasonable TTL (but again without realizing that RFC-8767 recommends a static value to cache for 1-3 days). Which means that any scalable cloud hosted application that wants to provide ultimate safety to their clients (which is basically a GDPR requirement) either needs to buy their own ip ranges or they need to block cloud resources for 3 days after scaling down (which seems unreasonable).

To be fair, this also seems like a short coming / oversight in RFC-8767. So maybe this should be discussed with the authors of RFC-8767? Wdyt @gthess & @AlexanderBand ?

[gthess]

I understand you concern but to be fair, using serve-expired is altogether unreasonable. The upstream sends a TTL with the answer and clearly clarifies when the record should not be used anymore. The upstream cannot account for serve-stale being used because there lies madness. And if a resolver party points you to stale data that is the resolver party's fault/optimization.

Both the static value and a dynamic one can lead to failures as any assumptions on the original TTL are probably wrong because the only thing that TTL conveys is the expiry time. For your factor proposal I am seeing it from a different angle. I would use a factor of the original TTL like 80% for example. Then it would mean that a cached record can be served for 180% of the original TTL, the last 80% being expired. I can't find a good value to suggest for either your proposal or mine because nothing makes sense to me :)

The serve stale RFC made an effort to standardize the operational practice of serving expired records; these installations have the traffic to keep expired records fairly up-to-date.

I am not sure if you are involved with the IETF, but if not it is easy to start a discussion, the IETF is a pretty open space. You can contact the authors directly or create a new topic on the dnsop mailing list.

[Dynamic5912]

Many TTL's have expirations of 1 hour or less, some are even a matter of minutes or seconds.

I highly doubt DNS records are changing that frequently and it just causes unnecessary traffic on connections.

Serve-Expired is fine when used within the RFC guidelines, and an expiration of 1 day leaves little room for error/compromise compared to an indefinite "0" as currently used by default.

[pimlie]

I understand you concern but to be fair, using serve-expired is altogether unreasonable

Not sure I read this correctly, but also given what you said before do you/NLnet (strongly?) recommend to disable serve-expired unless in very specific scenario's like f.e. when you run into performance issues and/or flaky upstream behaviour? If so, maybe a clear notification/warning about this could be added to the docs.

Both the static value and a dynamic one can lead to failures as any assumptions on the original TTL are probably wrong because the only thing that TTL conveys is the expiry time. For your factor proposal I am seeing it from a different angle. I would use a factor of the original TTL like 80% for example. Then it would mean that a cached record can be served for 180% of the original TTL, the last 80% being expired. I can't find a good value to suggest for either your proposal or mine because nothing makes sense to me :)

Although we might word things differently, it seems we've been almost in perfect sync about the issues. F.e. using 80% instead of a factor of 1.8 seems more of an implementation detail to me 😉 The main reason I was thinking in factors was because my starting point was a 1 minute TTL, and I think that caching a 1 minute TTL for 5 minutes (factor 5) should probably be ok (but not for a TTL of 1 day)..

I am not sure if you are involved with the IETF, but if not it is easy to start a discussion, the IETF is a pretty open space. You can contact the authors directly or create a new topic on the dnsop mailing list.

No, I'm not involved with IETF. So if you think some cache timer that takes the original TTL into account is a good idea, then I have no issue if you / NLnet take ownership as y'all are probably much more into the nitty-gritty details that comes with dns RFC's then I am 🙏