Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Authorization Headers missing for some requests in elasticsearch communication #8015

Closed
jrunu opened this issue Apr 30, 2020 · 6 comments
Closed

Authorization Headers missing for some requests in elasticsearch communication #8015

jrunu opened this issue Apr 30, 2020 · 6 comments
Labels
bug elasticsearch triaged
Milestone
4.0.0

Comments

@jrunu
Copy link
Contributor

jrunu commented Apr 30, 2020

Graylog does not always provide the Authorization Header when communicating with elasicsearch, despite credentials being supplied in the connection string.

This leads to issues when the elasticsearch cluster requires authentication.

Current Behavior

This information was gathered by analysing packet dumps from the graylog to elasticsearch communication.

The following URIs where noted as most prominent/frequent examples of this behaviour:

GET /_cluster/health/graylog_deflector?timeout=60s
GET /_cluster/health/_all?local=true&timeout=60s
GET /
GET /_nodes/_all/http
GET /_cluster/health/graylog_deflector%2Cgl-events_deflector%2Cgl-system-events_deflector?timeout=60s

Notable is /_nodes/_all/http. It was observed in one instance that the first call was missing the header, the next and subsequent requests from the same connection had the header.

Here is a excerpt from one of the logs on startup:

java.lang.IllegalArgumentException: Cat response did not contain a JSON Array
	at io.searchbox.core.Cat.parseResponseBody(Cat.java:61) ~[graylog.jar:?]
	at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:71) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:44) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:16) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:212) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:88) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:47) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.Cluster.catNodes(Cluster.java:121) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.Cluster.getFileDescriptorStats(Cluster.java:126) ~[graylog.jar:?]
	at org.graylog2.periodical.IndexerClusterCheckerThread.doRun(IndexerClusterCheckerThread.java:58) ~[graylog.jar:?]
	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_252]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_252]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_252]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_252]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
2020-04-30T13:58:20.960Z ERROR [ConfigurationManagementPeriodical] Error while running migration <V20161130141500_DefaultStreamRecalcIndexRanges{2016-11-30T14:15:00Z}>
org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector

missing authentication token for REST request [/_all/_alias]
	at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:108) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:58) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:336) ~[graylog.jar:?]
	at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
	at org.graylog2.migrations.V20161130141500_DefaultStreamRecalcIndexRanges.upgrade(V20161130141500_DefaultStreamRecalcIndexRanges.java:108) ~[graylog.jar:?]
	at org.graylog2.periodical.ConfigurationManagementPeriodical.doRun(ConfigurationManagementPeriodical.java:43) [graylog.jar:?]
	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
2020-04-30T13:58:21.003Z ERROR [ConfigurationManagementPeriodical] Error while running migration <V20170607164210_MigrateReopenedIndicesToAliases{2017-06-07T16:42:10Z}>
org.graylog2.indexer.ElasticsearchException: Unable to retrieve Elasticsearch version

missing authentication token for REST request [/]
	at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:108) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:58) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.Node.getVersion(Node.java:39) ~[graylog.jar:?]
	at org.graylog2.migrations.V20170607164210_MigrateReopenedIndicesToAliases.getReopenedIndices(V20170607164210_MigrateReopenedIndicesToAliases.java:85) ~[graylog.jar:?]
	at org.graylog2.migrations.V20170607164210_MigrateReopenedIndicesToAliases.getReopenedIndices(V20170607164210_MigrateReopenedIndicesToAliases.java:136) ~[graylog.jar:?]
	at org.graylog2.migrations.V20170607164210_MigrateReopenedIndicesToAliases.lambda$upgrade$0(V20170607164210_MigrateReopenedIndicesToAliases.java:79) ~[graylog.jar:?]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:269) ~[?:1.8.0_252]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_252]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_252]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:1.8.0_252]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_252]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) ~[?:1.8.0_252]
	at org.graylog2.migrations.V20170607164210_MigrateReopenedIndicesToAliases.upgrade(V20170607164210_MigrateReopenedIndicesToAliases.java:81) ~[graylog.jar:?]
	at org.graylog2.periodical.ConfigurationManagementPeriodical.doRun(ConfigurationManagementPeriodical.java:43) [graylog.jar:?]
	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
2020-04-30T13:58:21.174Z INFO  [JerseyService] Enabling CORS for HTTP endpoint
2020-04-30T13:58:21.882Z ERROR [Cluster] Couldn't read cluster health for indices [graylog_deflector, gl-events_deflector, gl-system-events_deflector]

missing authentication token for REST request [/_cluster/health/graylog_deflector%2Cgl-events_deflector%2Cgl-system-events_deflector?timeout=60s] (n/a)

Context

This was discovered in a running setup that had credentials in place but authentication on the elasticsearch side not yet activated. Resuming the processing after activating it on the elasticsearch cluster lead to the Graylog behaving erratic.

Reconstructing this in a lab lead to the rediscovery of #6212. After using the mentioned ways to work around that, the behaviour initially observed was reproduceable.

Your Environment

  • Graylog Version: 3.2 and 3.0.2
  • Elasticsearch Version: 6.8.8
  • MongoDB Version: 4.0
  • Operating System: Ubuntu 20.04
@jrunu jrunu added the bug label Apr 30, 2020
@florianpopp florianpopp added this to the 4.0.0 milestone May 4, 2020
@AndreasUfert
Copy link

This bug affects us, too. Is there any way to work around this problem? Waiting for 4.0.0 is not an option: Our security staff expects us to switch on Elasticsearch authentication. However after switching on authentication Graylog is able to search the current index but fails to do so for older indices.

@dennisoelkers
Copy link
Member

I tried to reproduce this with 3.3.6 and ES 6.8.0 with authentication enabled, but it all looks good. I explicitly tested the related code parts of the listed stack traces and they all worked for me. Please retry this with Graylog 3.3.x. If you still see problems with authentication, please reopen this issue.

@BcTpe4HbIu
Copy link

Still affects 3.3.8. Even search is acting weird. If I just open /search - default search shows up normally. But after some time it all collapses to missing authentication token for REST request [/_msearch].
Also there is error messages from various tasks like:

2020-11-05T21:09:15.167Z ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set <website-prod/5d1e086e0fbead3e9b6fdacf>
org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias website-prod_deflector

missing authentication token for REST request [/_all/_alias]
        at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:110) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:60) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:65) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:336) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
        at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_262]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_262]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_262]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_262]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_262]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_262]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]

@BcTpe4HbIu
Copy link

Looks like somehow disabling elasticsearch_discovery_enabled and rising elasticsearch_max_total_connections and elasticsearch_max_total_connections_per_route to their new defaults solved this issue. Changed from 20 and 2 to 200 and 20.

@dennisoelkers
Copy link
Member

@BcTpe4HbIu: Using node discovery does not work in conjunction with auth. FWIW, that is also noted explicitly in the docs.

@AndreasUfert: Are you using node discovery as well? Disabling it and feeding the complete node list to GL would allow you to work around this issue. GL 4.0 will support node discovery + auth.

@BcTpe4HbIu
Copy link

@dennisoelkers yeah, sure. That's why I've disabled it. But errors are puzzling. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug elasticsearch triaged
Projects
None yet
Milestone
4.0.0
Development

No branches or pull requests
5 participants
@dennisoelkers @BcTpe4HbIu @florianpopp @AndreasUfert @jrunu