Adding apache test package #111

ycombinator · 2020-09-18T18:24:21Z

This PR adds an apache test package.

Included in this test package are files needed for system testing this package. Consequently, we should see CI running system tests for the apache package (and hopefully passing).

Once this PR is merged the files related to system testing the apache test package in this PR should be incorporated into elastic/integrations#263.

elasticmachine · 2020-09-18T18:37:27Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #111 updated]
Start Time: 2020-10-01T09:50:35.086+0000
Duration: 13 min 5 sec

ycombinator · 2020-09-18T18:37:54Z

There is a problem with the system test runner code. It is replacing the wrong path in the test policy:

That path should be the in the agent container (/tmp/service_logs/...) instead.

mtojek · 2020-09-21T07:12:20Z

That path should be the in the agent container (/tmp/service_logs/...) instead.

I'm going to investigate it today.

mtojek · 2020-09-28T09:09:08Z

@ycombinator I think you can move forward with this PR as #112 is merged.

mtojek · 2020-09-29T11:16:48Z

If you prefer to merge it partially (2/3 datasets), I'm fine with it. Feel free to convert it a standard PR once is ready for review.

ycombinator · 2020-09-29T16:59:16Z

If you prefer to merge it partially (2/3 datasets), I'm fine with it.

Nah, I think I'm pretty close to getting the metrics dataset working too. If I can't get that by tomorrow, I'll just go with the two I have and put it into review.

ycombinator · 2020-09-29T20:08:30Z

Hmmm, after I rebased on the latest master I'm no longer able to get the system tests to pass for the apache/access and apache/error datasets. The test runner never finds any hits in the data streams.

I checked the Agent container and the Apache access and error logs are making it to the expected folder and have data in them:

$ docker exec -it elastic-package-stack_elastic-agent_1 sh -c 'ls -l /tmp/service_logs'
total 12
-rw-r--r-- 1 elastic-agent elastic-agent 737 Sep 29 20:04 access.log
-rw-r--r-- 1 elastic-agent elastic-agent 444 Sep 29 20:04 error.log
-rw-r--r-- 1 elastic-agent elastic-agent   2 Sep 29 20:04 httpd.pid

However, if I check the action_store.yml file, it is empty:

$ docker exec -it elastic-package-stack_elastic-agent_1 sh -c 'ls -l data/elastic-agent-*/action_store.yml'
-rw------- 1 elastic-agent elastic-agent 0 Sep 29 19:58 data/elastic-agent-30ff4d/action_store.yml

If I check the policy in Kibana, it appears to be correct:

id: eb844d60-028e-11eb-bb5f-bddaad03feef
revision: 2
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://elasticsearch:9200'
agent:
  monitoring:
    enabled: false
    logs: false
    metrics: false
inputs:
  - id: ed0e48c0-028e-11eb-bb5f-bddaad03feef
    name: apache-access
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: apache
        version: 0.1.4
    data_stream:
      namespace: ep
    streams:
      - id: logfile-apache.access
        data_stream:
          dataset: apache.access
          type: logs
        paths:
          - /tmp/service_logs/access.log*
        exclude_files:
          - .gz$
        processors:
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.5.0
fleet:
  kibana:
    hosts:
      - 'http://kibana:5601'

@mtojek do you spot anything wrong in the above? Can you try to reproduce on your end with this PR? Thanks. I'll keep investigating on my end too, of course.

ycombinator · 2020-09-30T00:32:35Z

Not sure if this is related to the problem mentioned in the previous comment, but I inspected the Agent logs. I see messages like these repeating frequently:

{"log.level":"error","@timestamp":"2020-09-30T00:19:05.937Z","log.origin":{"file.name":"operation/operation_retryable.go","file.line":85},"message":"operation operation-verify failed","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-30T00:19:05.937Z","log.origin":{"file.name":"application/action_dispatcher.go","file.line":93},"message":"Failed to dispatch action 'action_id: 942d0b07-3bdb-4dff-9ee6-33cb98a8af68, type: CONFIG_CHANGE', error: operator: failed to execute step sc-run, error: operation 'operation-verify' failed to verify filebeat.8.0.0-SNAPSHOT: 3 errors occurred:\n\t* fetching asc file from '/usr/share/elastic-agent/data/elastic-agent-30ff4d/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /usr/share/elastic-agent/data/elastic-agent-30ff4d/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory\n\t* check detached signature: openpgp: invalid signature: hash tag doesn't match\n\t* fetching asc file from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc: call to 'https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc' returned unsuccessful status code: 404: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/artifact/download/http/verifier.go[205]: unknown error\n\n: operation 'operation-verify' failed to verify filebeat.8.0.0-SNAPSHOT: 3 errors occurred:\n\t* fetching asc file from '/usr/share/elastic-agent/data/elastic-agent-30ff4d/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /usr/share/elastic-agent/data/elastic-agent-30ff4d/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory\n\t* check detached signature: openpgp: invalid signature: hash tag doesn't match\n\t* fetching asc file from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc: call to 'https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc' returned unsuccessful status code: 404: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/artifact/download/http/verifier.go[205]: unknown error\n\n","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-09-30T00:19:05.937Z","log.origin":{"file.name":"application/fleet_gateway.go","file.line":159},"message":"failed to dispatch actions, error: operator: failed to execute step sc-run, error: operation 'operation-verify' failed to verify filebeat.8.0.0-SNAPSHOT: 3 errors occurred:\n\t* fetching asc file from '/usr/share/elastic-agent/data/elastic-agent-30ff4d/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /usr/share/elastic-agent/data/elastic-agent-30ff4d/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory\n\t* check detached signature: openpgp: invalid signature: hash tag doesn't match\n\t* fetching asc file from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc: call to 'https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc' returned unsuccessful status code: 404: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/artifact/download/http/verifier.go[205]: unknown error\n\n: operation 'operation-verify' failed to verify filebeat.8.0.0-SNAPSHOT: 3 errors occurred:\n\t* fetching asc file from '/usr/share/elastic-agent/data/elastic-agent-30ff4d/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /usr/share/elastic-agent/data/elastic-agent-30ff4d/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory\n\t* check detached signature: openpgp: invalid signature: hash tag doesn't match\n\t* fetching asc file from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc: call to 'https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc' returned unsuccessful status code: 404: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/artifact/download/http/verifier.go[205]: unknown error\n\n","ecs.version":"1.5.0"}

Sure enough, if you try to visit https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc, it 404s. I think the domain in that URL needs to be snapshots.elastic.co?

ycombinator · 2020-09-30T00:39:37Z

FWIW, the same error as shown in the previous comment is seen with the 7.10.0-SNAPSHOT image as well:

2020-09-30T00:38:49.031Z        ERROR   application/fleet_gateway.go:159        failed to dispatch actions, error: operator: failed to execute step sc-run, error: operation 'operation-verify' failed to verify filebeat.7.10.0-SNAPSHOT: 3 errors occurred:
        * fetching asc file from '/usr/share/elastic-agent/data/elastic-agent-6aa14f/downloads/filebeat-7.10.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /usr/share/elastic-agent/data/elastic-agent-6aa14f/downloads/filebeat-7.10.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory
        * check detached signature: openpgp: invalid signature: hash tag doesn't match
        * fetching asc file from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.0-SNAPSHOT-linux-x86_64.tar.gz.asc: call to 'https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.0-SNAPSHOT-linux-x86_64.tar.gz.asc' returned unsuccessful status code: 404: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/artifact/download/http/verifier.go[205]: unknown error

: operation 'operation-verify' failed to verify filebeat.7.10.0-SNAPSHOT: 3 errors occurred:
        * fetching asc file from '/usr/share/elastic-agent/data/elastic-agent-6aa14f/downloads/filebeat-7.10.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /usr/share/elastic-agent/data/elastic-agent-6aa14f/downloads/filebeat-7.10.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory
        * check detached signature: openpgp: invalid signature: hash tag doesn't match
        * fetching asc file from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.0-SNAPSHOT-linux-x86_64.tar.gz.asc: call to 'https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.0-SNAPSHOT-linux-x86_64.tar.gz.asc' returned unsuccessful status code: 404: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/artifact/download/http/verifier.go[205]: unknown error

ycombinator · 2020-09-30T11:04:10Z

Related issue: elastic/beats#21320.

mtojek

LGTM!

mtojek · 2020-10-01T10:15:14Z

internal/testrunner/runners/system/runner.go

@@ -238,10 +239,22 @@ func (r *runner) run() error {
 }

 func (r *runner) tearDown() {
+	if logger.IsDebugMode() {
+		// Sleep to give some time for humans to scrutinize the current


I disabled it on purpose in one of recent PR not to slow down testing. Especially if we add next tests in the future. If you prefer we can leave it here or add a command argument not to wipe out the cluster state.

Yeah, it is only adding the sleep in debug mode. I wonder if we can add a check here to only do this if the test fails? If the test passes it's unlikely that we want to inspect the state of the system, right?

[EDIT] Let me know what you think of this idea. If you like it I'll make an issue/PR for it.

mtojek · 2020-10-01T10:15:41Z

internal/compose/compose.go

@@ -43,6 +43,32 @@ type portMapping struct {
 // UnmarshalYAML unmarshals a Docker Compose port mapping in YAML to
 // a portMapping.
 func (p *portMapping) UnmarshalYAML(node *yaml.Node) error {
+	// Depending on how the port mapping is specified in the Docker Compose


I didn't know there are kinds of responses present. Good to know.

mtojek · 2020-10-01T11:07:59Z

Let met merge this PR as it's already decent improvement.

mtojek mentioned this pull request Sep 21, 2020

Fix: SERVICE_LOGS points to agent path #112

Merged

ycombinator force-pushed the add-test-pkg-apache branch from 745a3d0 to 0bb06db Compare September 28, 2020 22:38

ycombinator mentioned this pull request Sep 29, 2020

[System test runner] --datasets flag is not working #118

Closed

ycombinator force-pushed the add-test-pkg-apache branch from 0bb06db to 76f437f Compare October 1, 2020 08:52

ycombinator added 12 commits October 1, 2020 05:03

Adding apache test package

a622698

Fixing filenames

be931bc

Adding system test config for apache/error dataset

f8ce07b

Only sleep before destructing in debug mode

99ae711

WIP: testing a fake host variable name

209eee1

Fleshing out config

78ff878

Updating parsing logic

e234ab2

Alternative parsing

21e1f90

Fixing syntax for array item lookup

1b102b8

Adding delete test policy handler

9c18c60

Adding Port to service context

14b4404

Using Port from service context

26a9e7e

ycombinator force-pushed the add-test-pkg-apache branch from 02ddb48 to 26a9e7e Compare October 1, 2020 09:04

ycombinator added 3 commits October 1, 2020 05:13

Consistent terminology

11c5b6d

Fixing some naming

c93a651

Fixing more naming

a557a18

Reducing teardown sleep duration

1f7883f

ycombinator marked this pull request as ready for review October 1, 2020 09:44

ycombinator requested a review from mtojek October 1, 2020 09:44

Adding explanatory comment

f98c359

mtojek approved these changes Oct 1, 2020

View reviewed changes

mtojek merged commit 0bb695c into elastic:master Oct 1, 2020

ycombinator deleted the add-test-pkg-apache branch October 5, 2020 12:50

This was referenced Oct 5, 2020

Adding system test files for Apache package elastic/integrations#263

Merged

Remove setting that's no longer needed #127

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding apache test package #111

Adding apache test package #111

ycombinator commented Sep 18, 2020 •

edited

Loading

elasticmachine commented Sep 18, 2020 •

edited

Loading

Build stats

ycombinator commented Sep 18, 2020

mtojek commented Sep 21, 2020

mtojek commented Sep 28, 2020

mtojek commented Sep 29, 2020

ycombinator commented Sep 29, 2020

ycombinator commented Sep 29, 2020

ycombinator commented Sep 30, 2020 •

edited

Loading

ycombinator commented Sep 30, 2020

ycombinator commented Sep 30, 2020

mtojek left a comment

mtojek Oct 1, 2020

ycombinator Oct 5, 2020 •

edited

Loading

mtojek Oct 1, 2020

mtojek commented Oct 1, 2020

Adding apache test package #111

Adding apache test package #111

Conversation

ycombinator commented Sep 18, 2020 • edited Loading

elasticmachine commented Sep 18, 2020 • edited Loading

💚 Build Succeeded

Build stats

ycombinator commented Sep 18, 2020

mtojek commented Sep 21, 2020

mtojek commented Sep 28, 2020

mtojek commented Sep 29, 2020

ycombinator commented Sep 29, 2020

ycombinator commented Sep 29, 2020

ycombinator commented Sep 30, 2020 • edited Loading

ycombinator commented Sep 30, 2020

ycombinator commented Sep 30, 2020

mtojek left a comment

Choose a reason for hiding this comment

mtojek Oct 1, 2020

Choose a reason for hiding this comment

ycombinator Oct 5, 2020 • edited Loading

Choose a reason for hiding this comment

mtojek Oct 1, 2020

Choose a reason for hiding this comment

mtojek commented Oct 1, 2020

ycombinator commented Sep 18, 2020 •

edited

Loading

elasticmachine commented Sep 18, 2020 •

edited

Loading

ycombinator commented Sep 30, 2020 •

edited

Loading

ycombinator Oct 5, 2020 •

edited

Loading