Allow to define custom scripts in independent Elastic Agents #1822

mrodm · 2024-05-07T17:05:22Z

Part of #787

This PR allows to define custom scripts for independent Elastic Agents. This custom scripts can be used to install required software to run the required tests successfully. For instance some libraries as it happens with oracle package.

Example of the new keys added into the configuration file:

agent:
  custom_script: |
    mkdir -p /my/path
    wget ...
  pre_start_script: |
    export PATH=${PATH}:/my/path

Now the files required to spin a new independent Elastic Agent using containers has been migrated to go-resources. Files generated can be checked within the profile folder:

 $ ls -l ~/.elastic-package/profiles/default/deployer/docker-agent-oracle-memory-61511/
total 12
-rw-r--r-- 1 user user 1046 may  8 16:15 docker-agent-base.yml
-rw-r--r-- 1 user user  203 may  8 16:15 Dockerfile
-rwxr-xr-x 1 user user  764 may  8 16:15 script.sh

If custom scripts are required, elastic-package builds a new Docker image adding the suffix -custom to the tag. Example:

 $ docker images |grep custom
docker.elastic.co/elastic-agent/elastic-agent-complete   8.13.2-custom     ee55b4ece29b   22 hours ago    3.95GB

Testing

Added a new test package test/packages/parallel/oracle based on the one from test/packages/with-custom-agent/oracle but with some changes:

Removed custom "agent" deploy (deleted folder).
Added just the oracle service in _dev/deploy/docker.
Added the required agent settings in the configuration file for system tests

This package does not work when tested with the Elastic stack running from the stack since it is missing some required libraries/dependencies the container.

Process to test this locally:

elastic-package stack up -v -d
export ELASTIC_PACKAGE_TEST_ENABLE_INDEPENDENT_AGENT=true

# test new package with independent Elastic Agents
pushd test/packages/parallel/oracle
elastic-package test system -v
popd

# test old package - it should keep using previous Custom Agents with servicedeployer
pushd test/packages/with-custom-agent/oracle
elastic-package test system -v
popd

elastic-package stack down -v

Update agent code to use go-resources to deploy the required files.

mrodm · 2024-05-08T11:00:04Z

.buildkite/pipeline.trigger.integration.tests.sh

+    if [[ "$independent_agent" == "false" && "$package_name" == "oracle" ]]; then
+        echoerr "Package \"${package_name}\" skipped: not supported with Elastic Agent running in the stack (missing required software)."
+        continue
+    fi


Required exception, since CI tests this package with both:

Generic Elastic stack started with elastic-package stack up

Independent Elastic Agents (setting env. variable)

Elastic Agent from the stack does not have the required libraries for this package.

mrodm · 2024-05-08T11:01:35Z

internal/agentdeployer/info.go

@@ -13,6 +13,27 @@ const (
 	elasticAgentTagsEnv = "ELASTIC_AGENT_TAGS"
 )

+type AgentSettings struct {


Moved to an independent struct in order to be used as embbeded struct here and in the test config. That makes possible to copy all fields at one in https://github.com/elastic/elastic-package/pull/1822/files#diff-de89bc59d9fa58f6220a67bda14afddc8f04ef1b816e81c2ca851711625f5834R339

mrodm · 2024-05-08T14:18:43Z

test/packages/parallel/oracle/data_stream/memory/_dev/test/system/test-memory-config.yml

+  pre_start_script: |
+    export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/oracle/instantclient_21_4"
+    export PATH="${PATH}:/opt/oracle/instantclient_21_4"
+    cd /opt/oracle/instantclient_21_4


Tested locally without this pre-start script and it also worked. However, I'd prefer to keep it since it was set in the Dockerfile too here:

elastic-package/test/packages/with-custom-agent/oracle/data_stream/memory/_dev/deploy/agent/Dockerfile

Lines 13 to 16 in e53417a

RUN export LD_LIBRARY_PATH=/opt/oracle/instantclient_21_4:$LD_LIBRARY_PATH && export PATH=/opt/oracle/instantclient_21_7:$PATH

ENV LD_LIBRARY_PATH "${LD_LIBRARY_PATH}:/opt/oracle/instantclient_21_4"

ENV PATH "${PATH}:/opt/oracle/instantclient_21_4"

WORKDIR /opt/oracle/instantclient_21_4

Maybe we can use a simpler package for testing. It could even maybe enough to have a script that writes a file, and a config that reads this file with a log input.

Added a new test package under test/packages/other with a minimal configuration to use these scripts.

About the test package oracle , do you think it should be kept under parallel as an example for the migration ?

mrodm · 2024-05-08T15:07:20Z

test/packages/parallel/oracle/data_stream/memory/_dev/test/system/test-memory-config.yml

+vars:
+  hosts:
+    - "oracle://sys:Oradoc_db1@elastic-package-service-oracle-1:1521/ORCLCDB.localdomain?sysdba=1"
+agent:


According to the tests in this PR, to update the oracle package in the integrations repository it should be needed:

add this key agent to each of the test system configuration files

update the hosts value to set as host elastic-package-service-oracle-1.

remove _dev/deploy/agent folders

replace agent folders with _dev/deploy/docker folder (something similar to this PR) with just the service definition (oracle).

mrodm · 2024-05-08T15:07:47Z

test/packages/parallel/oracle/data_stream/memory/_dev/test/system/test-memory-config.yml

@@ -0,0 +1,16 @@
+vars:
+  hosts:
+    - "oracle://sys:Oradoc_db1@elastic-package-service-oracle-1:1521/ORCLCDB.localdomain?sysdba=1"


This is required to be changed since the container name changes from oracle to elastic-package-service-oracle-1.

- - "oracle://sys:Oradoc_db1@oracle:1521/ORCLCDB.localdomain?sysdba=1" + - "oracle://sys:Oradoc_db1@elastic-package-service-oracle-1:1521/ORCLCDB.localdomain?sysdba=1"

jsoriano · 2024-05-09T16:10:31Z

internal/agentdeployer/_static/docker-agent-base.yml.tmpl

 services:
  elastic-agent:
    hostname: ${AGENT_HOSTNAME}
+    {{ if ne $custom_script "" }}
+    image: "${ELASTIC_AGENT_IMAGE_REF}-custom"


Should we use some other suffix to avoid conflicts? The image tag will be used if it exists, and it could be an unexpected one. Maybe we can use a checksum of the scripts.

Suggested change

image: "${ELASTIC_AGENT_IMAGE_REF}-custom"

image: "${ELASTIC_AGENT_IMAGE_REF}-{{ $some_id }}"

Added as a suffix the MD5 sum of the Dockerfile created.
image: "elastic-package-test-elastic-agent-complete:{{ $stack_version }}-{{ $dockerfile_hash }}"

The result is images like this:

elastic-package-test-elastic-agent-complete:8.13.2-861d45f9a2042f22c52105b4b3d1708a

jsoriano · 2024-05-09T16:13:21Z

test/packages/parallel/oracle/data_stream/memory/_dev/test/system/test-memory-config.yml

+    - "oracle://sys:Oradoc_db1@elastic-package-service-oracle-1:1521/ORCLCDB.localdomain?sysdba=1"
+agent:
+  runtime: docker
+  custom_script: |


Should we make this an object?

Suggested change

custom_script: |

custom_script:

content: |

This could be useful to support referencing scripts in the future, or even other interpreters:

custom_script: language: bash source: ./custom.sh

Added for custom_script (now it's named provisioning_script) and also for pre_start_script.

For pre_start_script , it is just allowed language sh, since it is injected directly those commands into the start script (entrypoint.sh).

internal/agentdeployer/_static/dockerfile.tmpl

test/packages/parallel/oracle/data_stream/memory/_dev/test/system/test-memory-config.yml

jsoriano · 2024-05-09T16:32:43Z

test/packages/parallel/oracle/data_stream/memory/_dev/test/system/test-memory-config.yml

+  pre_start_script: |
+    export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/oracle/instantclient_21_4"
+    export PATH="${PATH}:/opt/oracle/instantclient_21_4"
+    cd /opt/oracle/instantclient_21_4


Maybe we can use a simpler package for testing. It could even maybe enough to have a script that writes a file, and a config that reads this file with a log input.

jsoriano · 2024-05-09T16:35:05Z

internal/agentdeployer/agent.go

+		"kibana_host":                "https://kibana:5601",
+		"fleet_url":                  "https://fleet-server:8220",


Umm, would that work with the serverless provider?

I took as a basis the docker-custom base template, and according to that I'd say custom agents would not work with the serverless provider.

elastic-package/internal/servicedeployer/_static/docker-custom-agent-base.yml

Lines 11 to 15 in 901661e

environment:

- FLEET_ENROLL=1

- FLEET_URL=https://fleet-server:8220

- KIBANA_HOST=https://kibana:5601

- FLEET_TOKEN_POLICY_NAME=${FLEET_TOKEN_POLICY_NAME}

I'll try to give it a chance to make it work.

Well, we can leave this for a different change.

Yes, it would be better. It would require a few more changes. Afterwards, I'll create a PR to add Serverless support for independent Elastic Agents.

mrodm · 2024-05-10T14:15:50Z

/test

elasticmachine · 2024-05-10T14:55:49Z

💚 Build Succeeded

Buildkite Build
Commit: b54981d

History

💔 Build #3067 failed a9169f4
💔 Build #3066 failed 2cbfa97
💔 Build #3065 failed 78af72c
💔 Build #3064 failed 689c5ea
💔 Build #3063 failed 55edfba

cc @mrodm

mrodm · 2024-05-13T09:32:03Z

test/packages/parallel/oracle/manifest.yml

@@ -0,0 +1,40 @@
+format_version: 1.0.0
+name: oracle


Maybe it's interesting to keep this for reference until oracle package is updated in integrations. WDYT ?

mrodm · 2024-05-13T09:33:11Z

.../parallel/custom_entrypoint/data_stream/agent_logs/elasticsearch/ingest_pipeline/default.yml

+- json:
+    field: message
+    target_field: target
+- set:
+    field: error.message
+    value: not present target
+    if: ctx?.target == null
+on_failure:
+- set:
+    field: error.message
+    value: '{{ _ingest.on_failure_message }}'


If message is not a JSON (written by the entrypoint script) , system tests are going to fail.

jsoriano

👍

mrodm added 4 commits May 7, 2024 12:06

PoC add custom scripts

4e55e1d

Add custom scripts

63094cd

Update agent code to use go-resources to deploy the required files.

Fix check about custom scripts in template

790e91b

Add test package

95a0390

mrodm self-assigned this May 7, 2024

mrodm added 7 commits May 7, 2024 19:10

Re-order imports

b0763e5

Add exception oracle in parallel - requires installing software

fb4b510

Write error to stderr

d7bd970

Add comments

d65ae48

Revert changes in mode permissions

26d3667

Copy all agent settings at once - embedded struct

4e1ccb0

Update oracle test config

0a60daa

mrodm commented May 8, 2024

View reviewed changes

mrodm marked this pull request as ready for review May 8, 2024 14:41

mrodm requested a review from a team May 8, 2024 14:41

mrodm commented May 8, 2024

View reviewed changes

mrodm mentioned this pull request May 9, 2024

Add ability to deploy custom elastic-agents on different OS or runtimes #787

Open

Add runtime

9a15f05

jsoriano reviewed May 9, 2024

View reviewed changes

mrodm added 8 commits May 9, 2024 20:19

Update Dockerfile template

e99cc3b

Convert scripts keys to objects (language and contents)

2849bab

Use path in /tmp in custom entrypoint

c55e84d

Run directly the commands

8043dfa

Add new test package

55edfba

Add md5 hash as docker tag suffix

048774c

Create function for dockerfile resources

660bb66

Add agent name to entrypoint filename

b6a417a

mrodm added 8 commits May 10, 2024 12:03

Add build dependencies for test package

0de02d9

Set default programming language sh

d595933

Update test package

689c5ea

Update pre_start script

78af72c

Add into suffix tag also ep as elastic-package reference

2cbfa97

Change docker image name if Dockerfile is required

a9169f4

Update format

3c01a23

Move package custom_entrypoint and add exception

b54981d

mrodm mentioned this pull request May 10, 2024

Add support for system tests using independent Elastic agents in Elastic Serverless #1828

Merged

1 task

mrodm commented May 13, 2024

View reviewed changes

mrodm requested a review from jsoriano May 13, 2024 09:33

jsoriano approved these changes May 13, 2024

View reviewed changes

mrodm merged commit 5c414a6 into elastic:main May 13, 2024
3 checks passed

mrodm deleted the add-custom-script-agents branch May 13, 2024 17:45

mrodm mentioned this pull request May 15, 2024

Remove unused extra docker-compose files in default agent deployer #1834

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow to define custom scripts in independent Elastic Agents #1822

Allow to define custom scripts in independent Elastic Agents #1822

mrodm commented May 7, 2024 •

edited

Loading

mrodm May 8, 2024

mrodm May 8, 2024

mrodm May 8, 2024

jsoriano May 9, 2024

mrodm May 10, 2024

mrodm May 8, 2024 •

edited

Loading

mrodm May 8, 2024 •

edited

Loading

jsoriano May 9, 2024

mrodm May 10, 2024 •

edited

Loading

jsoriano May 9, 2024

mrodm May 10, 2024

jsoriano May 9, 2024

jsoriano May 9, 2024

mrodm May 9, 2024

jsoriano May 9, 2024

mrodm May 10, 2024

mrodm commented May 10, 2024

elasticmachine commented May 10, 2024

mrodm May 13, 2024

mrodm May 13, 2024

jsoriano left a comment

	RUN export LD_LIBRARY_PATH=/opt/oracle/instantclient_21_4:$LD_LIBRARY_PATH && export PATH=/opt/oracle/instantclient_21_7:$PATH
	ENV LD_LIBRARY_PATH "${LD_LIBRARY_PATH}:/opt/oracle/instantclient_21_4"
	ENV PATH "${PATH}:/opt/oracle/instantclient_21_4"
	WORKDIR /opt/oracle/instantclient_21_4

	image: "${ELASTIC_AGENT_IMAGE_REF}-custom"
	image: "${ELASTIC_AGENT_IMAGE_REF}-{{ $some_id }}"

		"kibana_host": "https://kibana:5601",
		"fleet_url": "https://fleet-server:8220",

	environment:
	- FLEET_ENROLL=1
	- FLEET_URL=https://fleet-server:8220
	- KIBANA_HOST=https://kibana:5601
	- FLEET_TOKEN_POLICY_NAME=${FLEET_TOKEN_POLICY_NAME}

Allow to define custom scripts in independent Elastic Agents #1822

Allow to define custom scripts in independent Elastic Agents #1822

Conversation

mrodm commented May 7, 2024 • edited Loading

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrodm May 8, 2024 • edited Loading

Choose a reason for hiding this comment

mrodm May 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrodm May 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrodm commented May 10, 2024

elasticmachine commented May 10, 2024

💚 Build Succeeded

History

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

mrodm commented May 7, 2024 •

edited

Loading

mrodm May 8, 2024 •

edited

Loading

mrodm May 8, 2024 •

edited

Loading

mrodm May 10, 2024 •

edited

Loading