Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Netty's asynchronous DNS resolver #1498

Closed
yueki1993 opened this issue Nov 6, 2020 · 2 comments
Closed

Use Netty's asynchronous DNS resolver #1498

yueki1993 opened this issue Nov 6, 2020 · 2 comments
Labels
type: feature A new feature
Milestone

Comments

@yueki1993
Copy link
Contributor

yueki1993 commented Nov 6, 2020

Bug Report

According to the motivation of #498, netty's non-blocking DNS resolver should be used.
However netty's blocking DNS resolver seems to be used now.

netty's release note suggests if we want to use async DNS resolver we need to pass resolver to Bootstrap; they say By default it will use the JDK name resolution, just as before, which is blocking.
The modifications of commit 3c8d021 and cf71131 is to simply delegate netty's DNS resolution, which seems insufficient to use async DNS resolver.

Maybe due to this behaviour I observe lettuce' epoll event loop is blocked during topology refresh (with dynamicRefreshSources=false) when DNS lookup is heavily loaded.

thread dump:

"lettuce-epollEventLoop-5-1" #25 daemon prio=5 os_prio=0 tid=xxxxxx nid=0x2d64 runnable [xxxxxxx]
   java.lang.Thread.State: RUNNABLE
	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
	at java.net.InetAddress.getByName(InetAddress.java:1077)
	at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156)
	at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153)
	at java.security.AccessController.doPrivileged(Native Method)
	at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153)
	at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41)
	at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61)
	at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53)
	at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55)
	at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31)
	at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106)
	at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:206)
	at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:46)
	at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:180)
	at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:166)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:984)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:504)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:417)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:474)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Current Behavior

netty's blocking DNS resolver(DefaultNameResolver) seems to be used.

Input Code

Expected behavior/code

netty's non-blocking DNS resolver(DnsNameResolver) should be used.

Environment

  • Lettuce version(s): 6.1.0.BUILD-SNAPSHOT (commit: fe70d44)
  • Redis version: 6.0.5

Possible Solution

yueki1993@be7a99a

Additional context

To check this behaviour debugging a test with intellij would be the simplest.
Debug run AtLeastOnceTest#connectionIsConnectedAfterConnect with breakpoints at:

@yueki1993 yueki1993 added the type: bug A general bug label Nov 6, 2020
@yueki1993
Copy link
Contributor Author

yueki1993 commented Nov 12, 2020

Hi @mp911de,
If my understanding is correct and you agree with replacing blocking dns resolver with the non-blocking one, can I submit a PR?
Or is it better to wait for evaluating this report for a while?
I am really sorry if this report is nonsense. If so I will close this issue.
(I understand you are really busy and I am willing to do anything I can if you need.)

@mp911de
Copy link
Collaborator

mp911de commented Nov 12, 2020

Thanks for bringing this issue up. We're happy to receive a pull request to use netty's asynchronous DNS resolver.

@mp911de mp911de added status: help-wanted An issue that a contributor can help us with type: feature A new feature and removed type: bug A general bug labels Nov 12, 2020
yueki1993 added a commit to yueki1993/lettuce-core that referenced this issue Nov 13, 2020
We now use netty's non-blocking DNS resolver upon connect.
mp911de pushed a commit that referenced this issue Dec 2, 2020
We now use netty's AddressResolverGroup and configure DnsResolverGroup if netty-dns-resolver is on the classpath.

Original pull request: #1517.
mp911de added a commit that referenced this issue Dec 2, 2020
Move Transports to resource package. Move Bootstrap initialization from Transports to ConnectionBuilder.

Rebase and adopt code to refactorings.

Reorder methods and fields.

Fix KeepAliveOptions.interval(…) builder method.

Original pull request: #1517.
@mp911de mp911de removed the status: help-wanted An issue that a contributor can help us with label Dec 2, 2020
@mp911de mp911de added this to the 6.1 M1 milestone Dec 2, 2020
@mp911de mp911de closed this as completed Dec 2, 2020
@mp911de mp911de changed the title netty's asynchronous DNS resolver seems not to be used Use Netty's asynchronous DNS resolver Mar 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: feature A new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants