Elastic Agent: expecting Dict and received *transpiler.Key for '0'

[mtojek]

Stack version: 7.13.0-SNAPSHOT

While enabling the fleet server in elastic/elastic-package#279 , I encountered the following issue:

2021-03-09T18:37:09.755Z	ERROR	application/fleet_gateway.go:185	failed to dispatch actions, error: fail to generate program configuration: expecting Dict and received *transpiler.Key for '0'
2021-03-09T18:37:09.755Z	WARN	status/reporter.go:233	Elastic Agent status changed to: 'degraded'
2021-03-09T18:37:09.755Z	INFO	status/reporter.go:233	Elastic Agent status changed to: 'online'

Seems that the agent can't pick up any policy?

cc @blakerouse @michalpristas

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[mtojek]

I didn't see this one for a longer time, not sure if the issue still exists.

[ruflin]

@mtojek Did also not see this recently, I suggest to close and reopen if needed?

[mtojek]

SGTM

[EricDavisX]

I am seeing the exact same error in Ansible automation to deploy our 'demo' test server with the 8.0 snapshot and have confirmed it is using the latest snapshots which are built recently enough, near daily. I"ll add it to my urgent tracking list. I submit it is possibly an intermittent timing bug where the automation is finding a problem as it triggers steps faster than a human could.

[mtojek]

Hint: you can double check if your configuration is correct. I remember we faced this problem while we're hitting Kibana endpoint with enabled fleet-server support.

[ruflin]

@EricDavisX Any chance you could share the exact config you are using?

[ph]

I don't think it's critical bfor 7.13, Agree on this?

[ruflin]

@ph As far as I can see we don't know yet and is what worries me?

[ph]

fair, point, if we have the configuration this should be simple to debug.

[EricDavisX]

It's from the 'demo' test server we have available. It is a self-managed 8.0 snapshot instance, where the 'Default Fleet Server policy' is created successfully, which consists of:

id: 6db12c50-a806-11eb-b5a1-a54876e2d326
revision: 2
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://bastion.endpoint.elastic.dev:9201'
output_permissions:
  default:
    _fallback:
      cluster:
        - monitor
      indices:
        - names:
            - logs-*
            - metrics-*
            - traces-*
            - .logs-endpoint.diagnostic.collection-*
            - synthetics-*
          privileges:
            - auto_configure
            - create_doc
agent:
  monitoring:
    enabled: true
    use_output: default
    logs: true
    metrics: true
inputs:
  - id: 7a700b2f-8815-4ca6-9ba6-eb6e8f4c7c64
    name: fleet_server-1
    revision: 1
    type: fleet-server
    use_output: default
    meta:
      package:
        name: fleet_server
        version: 0.2.2
    data_stream:
      namespace: default
    server:
      port: 8220
      host: 0.0.0.0
fleet:
  hosts:
    - 'https://ingest-linux-1.endpoint.elastic.dev:8220'
    - 'https://ingest-win-1.endpoint.elastic.dev:8220'

anything wrong with that ^ ? It looks ok to me, except maybe for the 0.0.0.0 for server: hosts: value. I am not yet knowledgeable enough in the architecture to know how that is used.

[michalpristas]

looks ok, when i paste it into a file elastic-agent.yml it works also inspect works, can you also share content of fleet yml if present (PM or redacted here)

[ruflin]

@michalpristas I see our test suite supports test data: https://github.com/elastic/beats/tree/master/x-pack/elastic-agent/pkg/agent/program/testdata Could we add these examples there too?

@mtojek @EricDavisX Do you remember if it was a temporary issue that resolved itself or was presistent?

[michalpristas]

the fact it's coming from fleet_gateway tells us that not configuration from elastic-agent.yml is broken but the one which is being retrieved from server. i will work with eric to obtain that one

[michalpristas]

the issue here comes from a spec file

- select_into:
      selectors: [ fleet.server.policy.id ]
      path: inputs.0.policy

  - select_into:
      selectors:
        - fleet.server.host
        - fleet.server.port
        - fleet.server.ssl
      path: inputs.0.server

where in inputs .0.server is not resolved correctly, we do not support indexing here. will provide a fix

[EricDavisX]

I'm not seeing this anymore in my 8.0 automation 'demo' setup, so I'm closing it - nice work Michal, thank you.