Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release requests in cors handler #32364

Merged
merged 2 commits into from
Jul 26, 2018
Merged

Release requests in cors handler #32364

merged 2 commits into from
Jul 26, 2018

Conversation

Tim-Brooks
Copy link
Contributor

There are two scenarios where a http request could terminate in the cors
handler. If that occurs, the requests need to be released. This commit
releases those requests.

@Tim-Brooks
Release requests in cors handler
3ad1d68 
There are two scenarios where a http request could terminate in the cors
handler. If that occurs, the requests need to be released. This commit
releases those requests.
@Tim-Brooks Tim-Brooks added >bug review :Distributed Coordination/Network Http and internode communication implementations v7.0.0 v6.5.0 labels Jul 25, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra

@original-brownbear
Copy link
Member

original-brownbear commented Jul 25, 2018 •
edited
Loading

@tbrooks8 just ran this with leak detection set to paranoid and it still fails unfortunately:

WARNING: 14 leak records were discarded because the leak record count is limited to 4. Use system property io.netty.leakDetection.maxRecords to increase the limit.
Recent access records: 4
#4:
	io.netty.buffer.AdvancedLeakAwareByteBuf.writeShort(AdvancedLeakAwareByteBuf.java:546)
	io.netty.buffer.ByteBufUtil.writeShortBE(ByteBufUtil.java:406)
	io.netty.handler.codec.http.HttpResponseEncoder.encodeInitialLine(HttpResponseEncoder.java:39)
	io.netty.handler.codec.http.HttpResponseEncoder.encodeInitialLine(HttpResponseEncoder.java:27)
	io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:94)
	io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
	io.netty.handler.codec.MessageToMessageCodec.write(MessageToMessageCodec.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	org.elasticsearch.http.nio.cors.NioCorsHandler.write(NioCorsHandler.java:96)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	org.elasticsearch.http.nio.NioHttpPipeliningHandler.write(NioHttpPipeliningHandler.java:71)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:801)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
	io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
	io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1036)
	io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:305)
	org.elasticsearch.http.nio.NettyAdaptor.write(NettyAdaptor.java:113)
	org.elasticsearch.http.nio.HttpReadWriteHandler.writeToBytes(HttpReadWriteHandler.java:99)
	org.elasticsearch.http.nio.HttpReadWriteHandlerTests.executeCorsRequest(HttpReadWriteHandlerTests.java:302)
	org.elasticsearch.http.nio.HttpReadWriteHandlerTests.testCorsAllowOriginWithSameHost(HttpReadWriteHandlerTests.java:250)
	java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.base/java.lang.reflect.Method.invoke(Method.java:564)
	com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	java.base/java.lang.Thread.run(Thread.java:844)
#3:
	io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:606)
	io.netty.handler.codec.http.HttpResponseStatus.encode(HttpResponseStatus.java:684)
	io.netty.handler.codec.http.HttpResponseEncoder.encodeInitialLine(HttpResponseEncoder.java:38)
	io.netty.handler.codec.http.HttpResponseEncoder.encodeInitialLine(HttpResponseEncoder.java:27)
	io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:94)
	io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
	io.netty.handler.codec.MessageToMessageCodec.write(MessageToMessageCodec.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	org.elasticsearch.http.nio.cors.NioCorsHandler.write(NioCorsHandler.java:96)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	org.elasticsearch.http.nio.NioHttpPipeliningHandler.write(NioHttpPipeliningHandler.java:71)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:801)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
	io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
	io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1036)
	io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:305)
	org.elasticsearch.http.nio.NettyAdaptor.write(NettyAdaptor.java:113)
	org.elasticsearch.http.nio.HttpReadWriteHandler.writeToBytes(HttpReadWriteHandler.java:99)
	org.elasticsearch.http.nio.HttpReadWriteHandlerTests.executeCorsRequest(HttpReadWriteHandlerTests.java:302)
	org.elasticsearch.http.nio.HttpReadWriteHandlerTests.testCorsAllowOriginWithSameHost(HttpReadWriteHandlerTests.java:250)
	java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.base/java.lang.reflect.Method.invoke(Method.java:564)
	com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	java.base/java.lang.Thread.run(Thread.java:844)
#2:
	io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:540)
	io.netty.handler.codec.http.HttpResponseEncoder.encodeInitialLine(HttpResponseEncoder.java:37)
	io.netty.handler.codec.http.HttpResponseEncoder.encodeInitialLine(HttpResponseEncoder.java:27)
	io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:94)
	io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
	io.netty.handler.codec.MessageToMessageCodec.write(MessageToMessageCodec.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	org.elasticsearch.http.nio.cors.NioCorsHandler.write(NioCorsHandler.java:96)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	org.elasticsearch.http.nio.NioHttpPipeliningHandler.write(NioHttpPipeliningHandler.java:71)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:801)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
	io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
	io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1036)
	io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:305)
	org.elasticsearch.http.nio.NettyAdaptor.write(NettyAdaptor.java:113)
	org.elasticsearch.http.nio.HttpReadWriteHandler.writeToBytes(HttpReadWriteHandler.java:99)
	org.elasticsearch.http.nio.HttpReadWriteHandlerTests.executeCorsRequest(HttpReadWriteHandlerTests.java:302)
	org.elasticsearch.http.nio.HttpReadWriteHandlerTests.testCorsAllowOriginWithSameHost(HttpReadWriteHandlerTests.java:250)
	java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.base/java.lang.reflect.Method.invoke(Method.java:564)
	com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	java.base/java.lang.Thread.run(Thread.java:844)
#1:
	io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:606)
	io.netty.handler.codec.http.HttpVersion.encode(HttpVersion.java:269)
	io.netty.handler.codec.http.HttpResponseEncoder.encodeInitialLine(HttpResponseEncoder.java:36)
	io.netty.handler.codec.http.HttpResponseEncoder.encodeInitialLine(HttpResponseEncoder.java:27)
	io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:94)
	io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
	io.netty.handler.codec.MessageToMessageCodec.write(MessageToMessageCodec.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	org.elasticsearch.http.nio.cors.NioCorsHandler.write(NioCorsHandler.java:96)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	org.elasticsearch.http.nio.NioHttpPipeliningHandler.write(NioHttpPipeliningHandler.java:71)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:801)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
	io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
	io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1036)
	io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:305)
	org.elasticsearch.http.nio.NettyAdaptor.write(NettyAdaptor.java:113)
	org.elasticsearch.http.nio.HttpReadWriteHandler.writeToBytes(HttpReadWriteHandler.java:99)
	org.elasticsearch.http.nio.HttpReadWriteHandlerTests.executeCorsRequest(HttpReadWriteHandlerTests.java:302)
	org.elasticsearch.http.nio.HttpReadWriteHandlerTests.testCorsAllowOriginWithSameHost(HttpReadWriteHandlerTests.java:250)
	java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.base/java.lang.reflect.Method.invoke(Method.java:564)
	com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	java.base/java.lang.Thread.run(Thread.java:844)
Created at:
	io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:314)
	io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:162)
	io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:153)
	io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:111)
	io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:92)
	io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
	io.netty.handler.codec.MessageToMessageCodec.write(MessageToMessageCodec.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	org.elasticsearch.http.nio.cors.NioCorsHandler.write(NioCorsHandler.java:96)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	org.elasticsearch.http.nio.NioHttpPipeliningHandler.write(NioHttpPipeliningHandler.java:71)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:801)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
	io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
	io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1036)
	io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:305)
	org.elasticsearch.http.nio.NettyAdaptor.write(NettyAdaptor.java:113)
	org.elasticsearch.http.nio.HttpReadWriteHandler.writeToBytes(HttpReadWriteHandler.java:99)
	org.elasticsearch.http.nio.HttpReadWriteHandlerTests.executeCorsRequest(HttpReadWriteHandlerTests.java:302)
	org.elasticsearch.http.nio.HttpReadWriteHandlerTests.testCorsAllowOriginWithSameHost(HttpReadWriteHandlerTests.java:250)
	java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.base/java.lang.reflect.Method.invoke(Method.java:564)
	com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	java.base/java.lang.Thread.run(Thread.java:844)

haven't looked at the code yet though, just a heads up.

@Tim-Brooks
Copy link
Contributor Author

@original-brownbear - Yeah. That one is unrelated to this PR. That one is not actually an issue. Just a test thing. Can you record that in the issue where you "introduce" paranoid level?

@original-brownbear
Copy link
Member

@tbrooks8 sure thanks added it to #32354. Fix here makes sense to me now that I read the code, but someone else should probably take a second look!

@original-brownbear
Copy link
Member

@tbrooks8 good news :) Just tested this change and it seems to fix the last remaining leaks in the netty http tests when I merge it together with my work from #32377.

I can confirm (did like 50 runs of the goal and it failed 100% of the time without your fix :)) that this fixes:

   > Throwable #1: java.lang.AssertionError: 
   > Expected: an empty collection
   >      but: <[LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
   > Recent access records: 4
   > #4:
   >    Hint: 'NioCorsHandler#0' will handle the message from this point.
   >    io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.touch(AdvancedLeakAwareCompositeByteBuf.java:36)
   >    io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpMessage.touch(HttpObjectAggregator.java:374)
   >    io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:454)
   >    io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:404)
   >    io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
   >    io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
   >    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
   >    io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:318)
   >    org.elasticsearch.http.nio.NettyAdaptor.read(NettyAdaptor.java:104)
   >    org.elasticsearch.http.nio.HttpReadWriteHandler.consumeReads(HttpReadWriteHandler.java:81)
   >    org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213)
   >    org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:54)
   >    org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119)
   >    org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:355)
   >    org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:216)
   >    org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144)
   >    org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:109)
   >    java.base/java.lang.Thread.run(Thread.java:844)
   > #3:
   >    io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpMessage.release(HttpObjectAggregator.java:386)
   >    io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:84)
   >    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:90)
   >    io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
   >    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
   >    io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:318)
   >    org.elasticsearch.http.nio.NettyAdaptor.read(NettyAdaptor.java:104)
   >    org.elasticsearch.http.nio.HttpReadWriteHandler.consumeReads(HttpReadWriteHandler.java:81)
   >    org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213)
   >    org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:54)
   >    org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119)
   >    org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:355)
   >    org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:216)
   >    org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144)
   >    org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:109)
   >    java.base/java.lang.Thread.run(Thread.java:844)
   > #2:
   >    io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.retain(AdvancedLeakAwareCompositeByteBuf.java:36)
   >    io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpMessage.retain(HttpObjectAggregator.java:362)
   >    io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.retain(HttpObjectAggregator.java:442)
   >    io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.retain(HttpObjectAggregator.java:404)
   >    io.netty.util.ReferenceCountUtil.retain(ReferenceCountUtil.java:36)
   >    io.netty.handler.codec.http.HttpContentEncoder.decode(HttpContentEncoder.java:88)
   >    io.netty.handler.codec.http.HttpContentEncoder.decode(HttpContentEncoder.java:51)
   >    io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)
   >    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
   >    io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
   >    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
   >    io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:318)
   >    org.elasticsearch.http.nio.NettyAdaptor.read(NettyAdaptor.java:104)
   >    org.elasticsearch.http.nio.HttpReadWriteHandler.consumeReads(HttpReadWriteHandler.java:81)
   >    org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213)
   >    org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:54)
   >    org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119)
   >    org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:355)
   >    org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:216)
   >    org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144)
   >    org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:109)
   >    java.base/java.lang.Thread.run(Thread.java:844)
   > #1:
   >    Hint: 'HttpContentCompressor#0' will handle the message from this point.
   >    io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.touch(AdvancedLeakAwareCompositeByteBuf.java:36)
   >    io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpMessage.touch(HttpObjectAggregator.java:374)
   >    io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:454)
   >    io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:404)
   >    io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
   >    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
   >    io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:318)
   >    org.elasticsearch.http.nio.NettyAdaptor.read(NettyAdaptor.java:104)
   >    org.elasticsearch.http.nio.HttpReadWriteHandler.consumeReads(HttpReadWriteHandler.java:81)
   >    org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213)
   >    org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:54)
   >    org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119)
   >    org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:355)
   >    org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:216)
   >    org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144)
   >    org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:109)
   >    java.base/java.lang.Thread.run(Thread.java:844)
   > Created at:
   >    io.netty.buffer.AbstractByteBufAllocator.compositeHeapBuffer(AbstractByteBufAllocator.java:207)
   >    io.netty.buffer.AbstractByteBufAllocator.compositeBuffer(AbstractByteBufAllocator.java:197)
   >    io.netty.handler.codec.MessageAggregator.decode(MessageAggregator.java:255)
   >    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
   >    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
   >    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
   >    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
   >    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
   >    io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:318)
   >    org.elasticsearch.http.nio.NettyAdaptor.read(NettyAdaptor.java:104)
   >    org.elasticsearch.http.nio.HttpReadWriteHandler.consumeReads(HttpReadWriteHandler.java:81)
   >    org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213)
   >    org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:54)
   >    org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119)
   >    org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:355)
   >    org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:216)
   >    org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144)
   >    org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:109)
   >    java.base/java.lang.Thread.run(Thread.java:844)]>
   >    at __randomizedtesting.SeedInfo.seed([6593DDE58A3BFD1B:847320810D495419]:0)
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.test.ESTestCase.checkStaticState(ESTestCase.java:473)
   >    at org.elasticsearch.test.ESTestCase.after(ESTestCase.java:342)
   >    at jdk.internal.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   >    at java.base/java.lang.Thread.run(Thread.java:844)
  1> [2018-07-25T16:38:11,164][INFO ][o.e.h.DeprecationHttpIT  ] [testDeprecationHeadersDoNotGetStuck]: before test
  1> [2018-07-25T16:38:11,165][INFO ][o.e.h.DeprecationHttpIT  ] [DeprecationHttpIT#testDeprecationHeadersDoNotGetStuck]: setting up test
  1> [2018-07-25T16:38:11,216][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_sm0] adding template [random_index_template] for index patterns [*]
  1> [2018-07-25T16:38:11,233][INFO ][o.e.h.DeprecationHttpIT  ]

s1monw
s1monw approved these changes Jul 26, 2018
View reviewed changes
Copy link
Contributor

@s1monw s1monw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any way to catch this in our test suite?

@original-brownbear
Copy link
Member

@s1monw we do since I added #31861 the other day, that's why this stuff started failing tests.
This should also be the last known leak so we could merge #32354 (that should go green once this one is in master) to get more deterministic behaviour here and catch essentially all leaks.

@Tim-Brooks Tim-Brooks merged commit 7a56df7 into elastic:master Jul 26, 2018
dnhatn added a commit that referenced this pull request Jul 27, 2018
@dnhatn
Merge branch 'master' into ccr
2f756b0 
* master:
  Remove reference to non-existent store type (#32418)
  [TEST] Mute failing FlushIT test
  Fix ordering of bootstrap checks in docs (#32417)
  [TEST] Mute failing InternalEngineTests#testSeqNoAndCheckpoints
  [TEST] Mute failing testConvertLongHexError
  bump lucene version after backport
  Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
  [Kerberos] Avoid vagrant update on precommit (#32416)
  TESTS: Move netty leak detection to paranoid level (#32354)
  [DOCS] Fixes formatting of scope object in job resource
  Copy missing segment attributes in getSegmentInfo (#32396)
  AbstractQueryTestCase should run without type less often (#28936)
  INGEST: Fix Deprecation Warning in Script Proc. (#32407)
  Switch x-pack/plugin to new style Requests (#32327)
  Docs: Correcting a typo in tophits (#32359)
  Build: Stop double generating buildSrc pom (#32408)
  TEST: Avoid triggering merges in FlushIT
  Fix missing JavaDoc for @throws in several places in KerberosTicketValidator.
  Switch x-pack full restart to new style Requests (#32294)
  Release requests in cors handler (#32364)
  Painless: Clean Up PainlessClass Variables (#32380)
  Docs: Fix callouts in put license HL REST docs (#32363)
  [ML] Consistent pattern for strict/lenient parser names (#32399)
  Update update-settings.asciidoc (#31378)
  Remove some dead code (#31993)
  Introduce index store plugins (#32375)
  Rank-Eval: Reduce scope of an unchecked supression
  Make sure _forcemerge respects `max_num_segments`. (#32291)
  TESTS: Fix Buf Leaks in HttpReadWriteHandlerTests (#32377)
  Only enforce password hashing check if FIPS enabled (#32383)
@Tim-Brooks Tim-Brooks deleted the cors_release branch December 18, 2019 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@s1monw s1monw s1monw approved these changes

@original-brownbear original-brownbear original-brownbear approved these changes

@jasontedor jasontedor Awaiting requested review from jasontedor

Assignees
No one assigned
Labels
>bug :Distributed Coordination/Network Http and internode communication implementations v6.5.0 v7.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Tim-Brooks @elasticmachine @original-brownbear @s1monw @colings86