From 23a2151528c7f0ff5ae079a92ecde7f01cb7a2b9 Mon Sep 17 00:00:00 2001 From: David Lougheed Date: Wed, 8 Jan 2025 16:32:06 -0500 Subject: [PATCH 1/7] docs: start working on service adding docs --- docs/adding-services.md | 59 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 docs/adding-services.md diff --git a/docs/adding-services.md b/docs/adding-services.md new file mode 100644 index 00000000..a3536add --- /dev/null +++ b/docs/adding-services.md @@ -0,0 +1,59 @@ +# Adding services to Bento + +There are two types of services in Bento: + +* Bento services, which have been developed by the Bento team specifically for the platform, and +* other services, which support the Bento services or provide additional platform features. + + +## Aspects to consider when adding any service to Bento + +* Service environment variables, used for configuring the image and some aspects of the service itself, should be added + to `etc/bento.env`. These variables typically include: + * Image + * Image version (tag) + * Container name template + * Service Docker network (**Note:** we typically give each service its own network, and add services to multiple + networks only as needed) + * Debugger ports +* Configuration environment variables TODO +* Docker container configuration via a Compose file in `lib//docker-compose..yaml` +* *As needed:* a gateway NGINX config in `lib/gateway/` + +* TODO + +Non-Bento services **MUST NOT** be put into `etc/bento_services.json`; this file is for Bento services only (see below). + +### Required `bentoctl` changes + +TODO + + +## Additional considerations when adding new Bento services + +### `/service-info` and service record in `bento_services.json` + +Bento services **MUST** implement the GA4GH [Service Info](https://www.ga4gh.org/product/service-info/) API. +They must also be registered in the `etc/bento_services.json` file, which allows them to be loaded into the +[Bento Service Registry](https://github.com/bento-platform/bento_service_registry). + +Each entry of this file follows the format: + +```js +{ + // ... + "": { + "service_kind": "", + "url_template": "{BENTO_PUBLIC_URL}/api/{service_kind}", + "repository": "git@github.com:bento-platform/<...>" + }, + // ... +} +``` + +In this format: +* `` is the key of the service in its `docker-compose.<...>.yaml` file +* `` is a special Bento-unique identifier for the service, allowing front ends to look up services. +* The `url_template` key is a template for the base URL used to access the service's API. +* The `repository` key is an SSH Git repository URL for the service code, so it can be cloned into the `repos` folder + for development. From 0177f2989b72bf9b3408e5d76fae4b9f6dd7d174 Mon Sep 17 00:00:00 2001 From: David Lougheed Date: Wed, 8 Jan 2025 16:56:55 -0500 Subject: [PATCH 2/7] docs: adding services continued --- docs/adding-services.md | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/docs/adding-services.md b/docs/adding-services.md index a3536add..27bc1a55 100644 --- a/docs/adding-services.md +++ b/docs/adding-services.md @@ -8,6 +8,8 @@ There are two types of services in Bento: ## Aspects to consider when adding any service to Bento +### Environment variables + * Service environment variables, used for configuring the image and some aspects of the service itself, should be added to `etc/bento.env`. These variables typically include: * Image @@ -17,17 +19,38 @@ There are two types of services in Bento: networks only as needed) * Debugger ports * Configuration environment variables TODO -* Docker container configuration via a Compose file in `lib//docker-compose..yaml` -* *As needed:* a gateway NGINX config in `lib/gateway/` -* TODO +### Container setup -Non-Bento services **MUST NOT** be put into `etc/bento_services.json`; this file is for Bento services only (see below). +The service's Docker container must be set up via a Compose file in `lib//docker-compose..yaml`. +This must then be included in the main `docker-compose.yaml` file, in the `include` block. + +The service's network (and potentially feature flag, if applicable), as well as container name and port environment +variables must be added to the gateway compose file (`lib/gateway/docker-compose.gateway.yaml`) if the service is to be +externally accessible. + +### Gateway configuration + +*As needed,* a gateway NGINX config must be placed into `lib/gateway/`. ### Required `bentoctl` changes TODO +* If new certificates are needed, add new entries to the `init_self_signed_certs` function (for development purposes) + in `py_bentoctl/other_helpers.py`. + +### Documentation changes + +* Make sure to add a note about how to set up the service for the first time to the + [Installation guide](./installation.md), as well as the migration guide for the version the service is introduced in. +* If additional deployment steps are needed (i.e., new certificates), add a note to the + [Deployment guide](./deployment.md). + +### Additional notes + +Non-Bento services **MUST NOT** be put into `etc/bento_services.json`; this file is for Bento services only (see below). + ## Additional considerations when adding new Bento services From cd169563eeedd1e2b1e0bec4ecdd8385e4adbb7c Mon Sep 17 00:00:00 2001 From: David Lougheed Date: Thu, 9 Jan 2025 10:59:49 -0500 Subject: [PATCH 3/7] docs: adding services - bentoctl notes --- docs/adding-services.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/docs/adding-services.md b/docs/adding-services.md index 27bc1a55..94c33e4e 100644 --- a/docs/adding-services.md +++ b/docs/adding-services.md @@ -35,10 +35,20 @@ externally accessible. ### Required `bentoctl` changes -TODO - -* If new certificates are needed, add new entries to the `init_self_signed_certs` function (for development purposes) - in `py_bentoctl/other_helpers.py`. +Inside the `py_bentoctl` Python module: + +* If the service is locked behind a feature flag, add the feature (as an `BentoOptionalFeature` instance) to + `config.py`, modeling it after other definitions. +* Add the service image environment variables to the `service_image_vars` variable in `services.py`. +* If the service is not a Bento service (or does not have the `bento` user in the Docker image), add the service to the + `BENTO_USER_EXCLUDED_SERVICES` variable. +* In `other_helpers.py`: + * If the service has a data directory that needs to be initialized, add an entry to the `data_dir_vars` variable + in the `init_dirs(...)` function containing the name of the environment variable which points to the data volume + directory. + * Add any entry with the name of the environment variable storing the name of the Docker network to the `networks` + variable in the `init_docker(...)` function. + * If new certificates are needed, add new entries to the `init_self_signed_certs` function (for development purposes). ### Documentation changes From ec66ded4d4e8bf88d53c6ea8c128f9de38ccb2dd Mon Sep 17 00:00:00 2001 From: David Lougheed Date: Thu, 9 Jan 2025 11:02:36 -0500 Subject: [PATCH 4/7] fix: add minio to services which dont have bento user --- py_bentoctl/services.py | 1 + 1 file changed, 1 insertion(+) diff --git a/py_bentoctl/services.py b/py_bentoctl/services.py index b721d499..14af30e9 100644 --- a/py_bentoctl/services.py +++ b/py_bentoctl/services.py @@ -33,6 +33,7 @@ "authz-db", "gateway", "katsu-db", + "minio", "redis", "reference-db", ) From 678d14fcb2e34c98a68f2c6ccebb74e14bcd972c Mon Sep 17 00:00:00 2001 From: David Lougheed Date: Thu, 9 Jan 2025 11:02:50 -0500 Subject: [PATCH 5/7] docs: v18 migrate line wrap --- docs/migrating_to_18.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/migrating_to_18.md b/docs/migrating_to_18.md index 9f2bf7d2..3d67b504 100644 --- a/docs/migrating_to_18.md +++ b/docs/migrating_to_18.md @@ -21,7 +21,8 @@ TODO ## 3. Enabling MinIO -To enable the deployment of a MinIO server for S3 storage, refer to the documentation on [configuring MinIO for Bento](/docs/minio.md). +To enable the deployment of a MinIO server for S3 storage, refer to the documentation on +[configuring MinIO for Bento](/docs/minio.md). ## TODO. Restart services From 21157b8e27922c99c3a0a16b0c8cfd2e24291848 Mon Sep 17 00:00:00 2001 From: David Lougheed Date: Thu, 9 Jan 2025 11:33:36 -0500 Subject: [PATCH 6/7] docs: 1st draft of adding services --- docs/adding-services.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/adding-services.md b/docs/adding-services.md index 94c33e4e..cf28ddbb 100644 --- a/docs/adding-services.md +++ b/docs/adding-services.md @@ -18,7 +18,11 @@ There are two types of services in Bento: * Service Docker network (**Note:** we typically give each service its own network, and add services to multiple networks only as needed) * Debugger ports -* Configuration environment variables TODO +* Configuration environment variables, for setting up feature flags and passwords, should be added to + `etc/default_config.env` and the example files `etc/bento_deploy.env` and `etc/bento_dev.env`. + * `etc/default_config.env` contains feature flags and "empty definitions" for passwords/secrets. + * `etc/bento_deploy.env` is an example / template setup (to be copied to `local.env`) for a production deployment. + * `etc/bento_dev.env` is an example / template setup (to be copied to `local.env`) for a development setup. ### Container setup @@ -64,6 +68,13 @@ Non-Bento services **MUST NOT** be put into `etc/bento_services.json`; this file ## Additional considerations when adding new Bento services +### User and base image + +It is expected that Bento services will use one of the +[Bento base images](https://github.com/bento-platform/bento_base_images). + +These images provide a `bento` user, whose UID is set to the host user's UID. + ### `/service-info` and service record in `bento_services.json` Bento services **MUST** implement the GA4GH [Service Info](https://www.ga4gh.org/product/service-info/) API. From 6d719ec21c62989cf921d3d54ba8fddde8dfd223 Mon Sep 17 00:00:00 2001 From: David Lougheed Date: Thu, 9 Jan 2025 11:35:45 -0500 Subject: [PATCH 7/7] docs: links to adding services doc --- README.md | 1 + docs/development.md | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/README.md b/README.md index f2b4aa5e..cf426551 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,7 @@ that make up the Bento platform. * [`bentoctl`: the Bento deployment command line management tool](./docs/bentoctl.md) * [Installation](./docs/installation.md) * [Development](./docs/development.md) + * [Adding services to Bento](./docs/adding-services.md) * [Troubleshooting guide](./docs/troubleshooting.md) * [Deployment](./docs/deployment.md) * [Monitoring](./docs/monitoring.md) diff --git a/docs/development.md b/docs/development.md index c357bb1c..ddc476d0 100644 --- a/docs/development.md +++ b/docs/development.md @@ -185,6 +185,11 @@ If subsequent modifications are made to the package's code, you will need to cre and install it again in the app with `npm install`. +## Adding services + +See [`adding-services.md`](./adding-services.md) for some considerations when adding new services to Bento. + + ## Using Adminer An [Adminer](https://www.adminer.org/) container is deployed in dev and local mode, it can be used to inspect the