ES X-Pack Security: Bootstraping graylog as unprivileged user fails. Operation as unprivileged user is fine. #6212

jpengenheiro · 2019-07-29T15:31:58Z

Recently I've observed what I consider to be unexpected behaviour while bootstraping graylog as an unprivileged user.

Here is a link to the troubleshooting I already made:

Expected Behavior

Graylog should create the correct templates, indices and aliases with a user that can only access indices with a cretain prefix, this would allow for easy setup of multitenant ES clusters with several graylog clients.

Current Behavior

while bootstraping as an unprivileged user:

graylog fails to create the graylog-internal template
graylog creates an index suffixed with deflector ex: graylogdev_default_deflector.

while bootstraping as privileged elastic user:

graylog creates the graylog-internal template
graylog creates an index suffixed with a number ex: graylogdev_default_0.

after bootstraping I can switch to an unprivileged user:

index set creation results in the correct creation of a new template
a new index is correctly created, suffixed with a number ex: graylogdev_new_0.

Possible Solution

It would be nice to know exactly what are the strictly required privileges to have graylog using an ES cluster as an unprivileged user.

Steps to Reproduce (for bugs)

The troubleshooting I already made:

https://community.graylog.org/t/unusual-behaviour-while-using-elasticsearch-non-super-user/11355/1

Context

This is preventing the design of a multitenant ES cluster solution, where every user would not be able to access any index not explicitely allow by an administrator.

This was encountered when attempting to implement authentication and authorization for the ES cluster by creating a user with restricted index use permissions.

Environment

Graylog Version: 3.0.2
Elasticsearch Version: 6.8
MongoDB Version: 3.6.13
Operating System: CentOS 7

The text was updated successfully, but these errors were encountered:

jdekoning · 2019-09-16T08:55:15Z

@jpengenheiro I am very interested in how your configuration now looks like... Whenever I enable x-pack I always end up in your first scenario (fails to create templates and indices, which is actually auto created by ES). To reduce the problem as much as possible I have disabled almost all x-pack features. When I disable it entirely, there are no problems (xpack.security.enabled = "false").

I only have xpack.security.enabled = "true" enabled and pretty much disabled authentication as well: xpack.security.authc.anonymous.roles = "superuser".

Http ssl is still disabled xpack.security.http.ssl.enabled = "false"

At the time of writing I was using:
elasticsearch:6.8.2
graylog:3.1

And to avoid the Deflector exists as an index and is not an alias error i have added action.auto_create_index = ".watches,.triggered_watches,.watcher-history-*". Which makes the logs flood with messages like:
2019-09-16 08:37:34,010 WARN : org.graylog2.indexer.messages.Messages - Failed to index message: index= id=<3a767271-d85d-11e9-a848-0242ac110002> error=<{"type":"index_not_found_exception","reason":"no such index and [action.auto_create_index] ([.watches,.triggered_watches,.watcher-history-*]) doesn't match","index_uuid":"_na_","index":"graylog_deflector"}>

To me this issue seems similar, but I can also create a new one if needed.

jdekoning · 2019-09-16T16:28:46Z

After some poking around I got to the point that I would initialize the graylog-internal template myself. After creating graylog-internal and also creating a first graylog_0 index, everything seems to work fine. Using the free services of x-pack such as authentication and tls on both transport and http for elasticsearch is now not a problem anymore.

So the workaround I found to be working:

curl -X PUT --user <user>:<pass> https://elastic.node:9200/_template/graylog-internal -H 'Content-Type: application/json' -d' {"order":-1,"index_patterns":["graylog_*"],"settings":{"index":{"analysis":{"analyzer":{"analyzer_keyword":{"filter":"lowercase","tokenizer":"keyword"}}}}},"mappings":{"message":{"properties":{"message":{"type":"text","analyzer":"standard","fielddata":false},"full_message":{"type":"text","analyzer":"standard","fielddata":false},"timestamp":{"type":"date","format":"yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"yyyy-MM-dd HH:mm:ss.SSS"},"gl2_processing_timestamp":{"type":"date","format":"yyyy-MM-dd HH:mm:ss.SSS"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"streams":{"type":"keyword"}},"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"_source":{"enabled":true}}},"aliases":{"graylog_deflector": {}}} '

Followed by a initialization of the first alias:

curl -X PUT --user <user>:<pass> https://elastic.node:9200/graylog_0

jalogisch · 2019-10-04T19:59:37Z

according to this community posting it is working with Elasticsearch 6.7.x

https://community.graylog.org/t/kubernetes-install-of-graylog-not-able-to-create-indexes-in-elasticsearch/12247/2

williamsjasonj · 2019-10-04T21:59:12Z

This workaround is great! Thanks for sharing until a fix is out.

jpengenheiro · 2019-10-15T10:22:19Z

@jdekoning I have since moved on to other projects and so I probably won't be pursuing this issue anymore, still if you require any information from me please feel free to ask

Your troubleshooting appears to confirm what I already suspected, the problem arises when graylog attempts to create the templates and first index, I remember I could not see any templates being created if I bootstrapped the cluster with an unpriviliged user.

Yours is a better workaround since it does not require knowledge of the elasticsearch's super user's credentials.

mpiscaer · 2019-12-20T11:19:25Z

@jdekoning Thank you for the workaround.

jrunu · 2020-04-30T13:29:41Z

At least part of this issue can be attribute to the behaviour change I described in issue elasticsearch/56027. The gist of it is that when x-pack security is enabled /_all/_alias returns a 404, while returning a 200 with an empty JSON object with x-pack disabled. This throws a wrench into the lookup for the deflectors.

Sidenote: /_alias, which is functionally the same I believe, returns a 200 in both cases.

frantz45 · 2020-06-10T15:46:14Z

Same issue here.
In addition to the @jdekoning 's workaround I had to recalculate index ranges in order to be able to search logs

dbu1986 · 2020-10-23T11:31:45Z

Same here and therefore adding my observations:

We're currently setting up Graylog 3.3.8 with ES 6.8.12. We also need to make use of Elasticsearch Authentication with different users for Graylog and other applications writing directly to Elasticsearch.

According to @jdekoning's solution, I experienced 2 possible workarounds (in our environment) to proceed without errors:

A) Bootstrap Graylog with elastic Superuser in server.conf, then switch to graylog user and restart Graylog.

PRO: Easy to handle.
CON: Knowledge of the elastic Superuser Credentials is mandatory.

B) Manually create 5 objects via Elasticsearch API, then initially starting Graylog.

PRO: No need to use the elastic Superuser
CON: More complex than A), and you need to get the index-template Configuration first, which could differ by Graylog releases.

For the second workaround, I initially tried to only set up the index template and the graylog_0 index (as @jdekoning did) but I still got errors in the graylog-server.log. So these are the 5 Requests I had to send in this order:

curl -X PUT -u 'graylog:[PASSWORD]' http://[ES-HOST]:9200/_template/graylog-internal -H 'Content-Type: application/json' -d '[CONTENT]'
curl -X PUT -u 'graylog:[PASSWORD]' http://[ES-HOST]:9200/graylog_0
curl -X PUT -u 'graylog:[PASSWORD]' http://[ES-HOST]:9200/gl-system-events__0
curl -X PUT -u 'graylog:[PASSWORD]' http://[ES-HOST]:9200/gl-events__0
curl -X POST -u 'graylog:[PASSWORD]' -H 'Content-Type: application/json' "http://[ES-HOST]:9200/_aliases" -d '{ "actions" : [ { "add" : { "index" : "graylog_0", "alias" : "graylog_deflector" } } ] }'

Fun Fact: There was no need to add the aliases gl-events_deflector and gl-system-events_deflector as Graylog just created them without any trouble:

2020-10-23 12:33:24,633 INFO : org.graylog2.indexer.MongoIndexSet - Did not find a deflector alias. Setting one up now.
2020-10-23 12:33:24,638 INFO : org.graylog2.indexer.MongoIndexSet - There is no index target to point to. Creating one now.
2020-10-23 12:33:24,646 INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <none> to <gl-events_0>.
2020-10-23 12:33:24,646 INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <gl-events_0>.
2020-10-23 12:33:24,689 INFO : org.graylog2.indexer.indices.Indices - Successfully created index template gl-events-template
2020-10-23 12:33:24,896 INFO : org.graylog2.indexer.MongoIndexSet - Waiting for allocation of index <gl-events_0>.
2020-10-23 12:33:24,908 INFO : org.graylog2.indexer.MongoIndexSet - Index <gl-events_0> has been successfully allocated.
2020-10-23 12:33:24,909 INFO : org.graylog2.indexer.MongoIndexSet - Pointing index alias <gl-events_deflector> to new index <gl-events_0>.
2020-10-23 12:33:24,952 INFO : org.graylog2.indexer.MongoIndexSet - Successfully pointed index alias <gl-events_deflector> to index <gl-events_0>.
2020-10-23 12:33:24,994 INFO : org.graylog2.indexer.MongoIndexSet - Did not find a deflector alias. Setting one up now.
2020-10-23 12:33:24,997 INFO : org.graylog2.indexer.MongoIndexSet - There is no index target to point to. Creating one now.
2020-10-23 12:33:25,004 INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <none> to <gl-system-events_0>.
2020-10-23 12:33:25,004 INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <gl-system-events_0>.
2020-10-23 12:33:25,026 INFO : org.graylog2.indexer.indices.Indices - Successfully created index template gl-system-events-template
2020-10-23 12:33:25,234 INFO : org.graylog2.indexer.MongoIndexSet - Waiting for allocation of index <gl-system-events_0>.
2020-10-23 12:33:25,241 INFO : org.graylog2.indexer.MongoIndexSet - Index <gl-system-events_0> has been successfully allocated.
2020-10-23 12:33:25,242 INFO : org.graylog2.indexer.MongoIndexSet - Pointing index alias <gl-system-events_deflector> to new index <gl-system-events_0>.
2020-10-23 12:33:25,288 INFO : org.graylog2.indexer.MongoIndexSet - Successfully pointed index alias <gl-system-events_deflector> to index <gl-system-events_0>.

The Elasticsearch user graylog is receiving these permissions by role:

curl -X GET -u 'elastic:[PASSWORD]' -H 'Content-Type: application/json' "http://[ES-HOST]:9200/_xpack/security/role/allow_graylog_indices?pretty"
{
  "allow_graylog_indices" : {
    "cluster" : [
      "monitor",
      "manage_index_templates"
    ],
    "indices" : [
      {
        "names" : [
          "graylog*",
          "gl*"
        ],
        "privileges" : [
          "all"
        ],
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : { },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}

My xpack settings in elasticsearch.yml:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: certs/[ES-HOST].key
xpack.security.transport.ssl.certificate: certs/[ES-HOST].crt
xpack.security.transport.ssl.certificate_authorities: [ "certs/ca.crt" ]
xpack.security.authc.realms.native1.type: native
xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.emit_request_body: true
xpack.security.audit.logfile.events.include: access_denied, access_granted, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, run_as_granted, authentication_success, realm_authentication_failed, connection_granted

My Graylog server.conf:

[]# grep -Ev '#|^\s*$' conf/server.conf
is_master = true
node_id_file = /opt/graylog/conf/node-id
password_secret = [PASSWORD_SECRET]
root_password_sha2 = bb87ef5dea2fdf3816c33e3bf58e5a384b07f7b9c8294565c7bacef6d2152210
root_timezone = Europe/Berlin
bin_dir = bin
data_dir = /opt/graylog/data/graylog
plugin_dir = plugin
http_bind_address = [IP]:9000
http_external_uri = https://[EXT-IP]/
elasticsearch_hosts = http://graylog:[PASSWORD]@[ES-HOST]:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 3
elasticsearch_replicas = 1
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /opt/graylog/data/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://[GL-NODE1],[GL-NODE2],[GL-NODE3]/graylog?replicaSet=Graylog-RS01
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

jalogisch changed the title ~~Bootstraping graylog as unprivileged user fails. Operation as unprivileged user is fine.~~ ES X-Pack Security: Bootstraping graylog as unprivileged user fails. Operation as unprivileged user is fine. Jul 29, 2019

jalogisch added elasticsearch bug labels Jul 29, 2019

florianpopp added documentation triaged and removed bug labels Aug 5, 2019

jalogisch mentioned this issue Sep 24, 2019

Cannot create deflector alias in an empty secured Elasticsearch cluster #6485

Open

jrunu mentioned this issue Apr 30, 2020

Authorization Headers missing for some requests in elasticsearch communication #8015

Closed

jalogisch mentioned this issue Feb 11, 2021

Document the Access Roles Graylog need in Elasticsearch x-pack Security Graylog2/documentation#1062

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ES X-Pack Security: Bootstraping graylog as unprivileged user fails. Operation as unprivileged user is fine. #6212

ES X-Pack Security: Bootstraping graylog as unprivileged user fails. Operation as unprivileged user is fine. #6212

jpengenheiro commented Jul 29, 2019 •

edited

Loading

jdekoning commented Sep 16, 2019

jdekoning commented Sep 16, 2019

jalogisch commented Oct 4, 2019

williamsjasonj commented Oct 4, 2019

jpengenheiro commented Oct 15, 2019

mpiscaer commented Dec 20, 2019

jrunu commented Apr 30, 2020 •

edited

Loading

frantz45 commented Jun 10, 2020 •

edited

Loading

dbu1986 commented Oct 23, 2020 •

edited

Loading

ES X-Pack Security: Bootstraping graylog as unprivileged user fails. Operation as unprivileged user is fine. #6212

ES X-Pack Security: Bootstraping graylog as unprivileged user fails. Operation as unprivileged user is fine. #6212

Comments

jpengenheiro commented Jul 29, 2019 • edited Loading

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Context

Environment

jdekoning commented Sep 16, 2019

jdekoning commented Sep 16, 2019

So the workaround I found to be working:

Followed by a initialization of the first alias:

jalogisch commented Oct 4, 2019

williamsjasonj commented Oct 4, 2019

jpengenheiro commented Oct 15, 2019

mpiscaer commented Dec 20, 2019

jrunu commented Apr 30, 2020 • edited Loading

frantz45 commented Jun 10, 2020 • edited Loading

dbu1986 commented Oct 23, 2020 • edited Loading

jpengenheiro commented Jul 29, 2019 •

edited

Loading

jrunu commented Apr 30, 2020 •

edited

Loading

frantz45 commented Jun 10, 2020 •

edited

Loading

dbu1986 commented Oct 23, 2020 •

edited

Loading