Merge pull request #300 from bento-platform/docs/services

docs: note on adding services to Bento

README.md

@@ -34,6 +34,7 @@ that make up the Bento platform.
 * [`bentoctl`: the Bento deployment command line management tool](./docs/bentoctl.md)
 * [Installation](./docs/installation.md)
 * [Development](./docs/development.md)
+  * [Adding services to Bento](./docs/adding-services.md)
 * [Troubleshooting guide](./docs/troubleshooting.md)
 * [Deployment](./docs/deployment.md)
 * [Monitoring](./docs/monitoring.md)

docs/adding-services.md

@@ -0,0 +1,103 @@
+# Adding services to Bento
+
+There are two types of services in Bento:
+
+* Bento services, which have been developed by the Bento team specifically for the platform, and
+* other services, which support the Bento services or provide additional platform features.
+
+
+## Aspects to consider when adding any service to Bento
+
+### Environment variables
+
+* Service environment variables, used for configuring the image and some aspects of the service itself, should be added 
+  to `etc/bento.env`. These variables typically include:
+  * Image
+  * Image version (tag)
+  * Container name template
+  * Service Docker network (**Note:** we typically give each service its own network, and add services to multiple 
+    networks only as needed)
+  * Debugger ports
+* Configuration environment variables, for setting up feature flags and passwords, should be added to 
+  `etc/default_config.env` and the example files `etc/bento_deploy.env` and `etc/bento_dev.env`.
+  * `etc/default_config.env` contains feature flags and "empty definitions" for passwords/secrets. 
+  * `etc/bento_deploy.env` is an example / template setup (to be copied to `local.env`) for a production deployment.
+  * `etc/bento_dev.env` is an example / template setup (to be copied to `local.env`) for a development setup.
+
+### Container setup
+
+The service's Docker container must be set up via a Compose file in `lib/<service>/docker-compose.<service>.yaml`.
+This must then be included in the main `docker-compose.yaml` file, in the `include` block.
+
+The service's network (and potentially feature flag, if applicable), as well as container name and port environment 
+variables must be added to the gateway compose file (`lib/gateway/docker-compose.gateway.yaml`) if the service is to be 
+externally accessible.
+
+### Gateway configuration
+
+*As needed,* a gateway NGINX config must be placed into `lib/gateway/<public_services|services>`.
+
+### Required `bentoctl` changes
+
+Inside the `py_bentoctl` Python module:
+
+* If the service is locked behind a feature flag, add the feature (as an `BentoOptionalFeature` instance) to 
+  `config.py`, modeling it after other definitions.
+* Add the service image environment variables to the `service_image_vars` variable in `services.py`. 
+* If the service is not a Bento service (or does not have the `bento` user in the Docker image), add the service to the
+  `BENTO_USER_EXCLUDED_SERVICES` variable.
+* In `other_helpers.py`:
+  * If the service has a data directory that needs to be initialized, add an entry to the `data_dir_vars` variable 
+    in the `init_dirs(...)` function containing the name of the environment variable which points to the data volume 
+    directory.
+  * Add any entry with the name of the environment variable storing the name of the Docker network to the `networks` 
+    variable in the `init_docker(...)` function.
+  * If new certificates are needed, add new entries to the `init_self_signed_certs` function (for development purposes).
+
+### Documentation changes
+
+* Make sure to add a note about how to set up the service for the first time to the 
+  [Installation guide](./installation.md), as well as the migration guide for the version the service is introduced in.
+* If additional deployment steps are needed (i.e., new certificates), add a note to the 
+  [Deployment guide](./deployment.md).
+
+### Additional notes
+
+Non-Bento services **MUST NOT** be put into `etc/bento_services.json`; this file is for Bento services only (see below).
+
+
+## Additional considerations when adding new Bento services
+
+### User and base image
+
+It is expected that Bento services will use one of the 
+[Bento base images](https://github.com/bento-platform/bento_base_images).
+
+These images provide a `bento` user, whose UID is set to the host user's UID.
+
+### `/service-info` and service record in `bento_services.json`
+
+Bento services **MUST** implement the GA4GH [Service Info](https://www.ga4gh.org/product/service-info/) API.
+They must also be registered in the `etc/bento_services.json` file, which allows them to be loaded into the 
+[Bento Service Registry](https://github.com/bento-platform/bento_service_registry).
+
+Each entry of this file follows the format:
+
+```js
+{
+  // ...
+   "<compose ID>": {
+     "service_kind": "<service kind>",
+     "url_template": "{BENTO_PUBLIC_URL}/api/{service_kind}",
+     "repository": "[email protected]:bento-platform/<...>"
+   },
+  // ...
+}
+```
+
+In this format:
+* `<compose ID>` is the key of the service in its `docker-compose.<...>.yaml` file
+* `<service kind>` is a special Bento-unique identifier for the service, allowing front ends to look up services.
+* The `url_template` key is a template for the base URL used to access the service's API.
+* The `repository` key is an SSH Git repository URL for the service code, so it can be cloned into the `repos` folder
+  for development.

docs/development.md

@@ -185,6 +185,11 @@ If subsequent modifications are made to the package's code, you will need to cre
 and install it again in the app with `npm install`.
 
 
+## Adding services
+
+See [`adding-services.md`](./adding-services.md) for some considerations when adding new services to Bento.
+
+
 ## Using Adminer
 
 An [Adminer](https://www.adminer.org/) container is deployed in dev and local mode, it can be used to inspect the

docs/migrating_to_18.md

@@ -21,7 +21,8 @@ TODO
 
 ## 3. Enabling MinIO
 
-To enable the deployment of a MinIO server for S3 storage, refer to the documentation on [configuring MinIO for Bento](/docs/minio.md).
+To enable the deployment of a MinIO server for S3 storage, refer to the documentation on 
+[configuring MinIO for Bento](/docs/minio.md).
 
 
 ## TODO. Restart services

py_bentoctl/services.py

@@ -33,6 +33,7 @@
     "authz-db",
     "gateway",
     "katsu-db",
+    "minio",
     "redis",
     "reference-db",
 )