Merge pull request #296 from bento-platform/releases/v18

Release v18

README.md

@@ -34,11 +34,13 @@ that make up the Bento platform.
 * [`bentoctl`: the Bento deployment command line management tool](./docs/bentoctl.md)
 * [Installation](./docs/installation.md)
 * [Development](./docs/development.md)
+  * [Adding services to Bento](./docs/adding-services.md)
 * [Troubleshooting guide](./docs/troubleshooting.md)
 * [Deployment](./docs/deployment.md)
 * [Monitoring](./docs/monitoring.md)
 * [Public discovery configuration](./docs/public_discovery.md)
 * [Using a reverse proxy in front of Bento](./docs/reverse-proxy.md)
+* [MinIO object storage](./docs/minio.md)
 
 ### Data ingestion and usage
 
@@ -48,6 +50,7 @@ that make up the Bento platform.
 
 ### Migration documents
 
+* [v17.1 to v18](./docs/migrating_to_18.md)
 * [v17 to v17.1](./docs/migrating_to_17_1.md)
 * [v16 to v17](./docs/migrating_to_17.md)
 * [v15.2 to v16](./docs/migrating_to_16.md)

docker-compose.dev.yaml

@@ -40,11 +40,13 @@ services:
           - ${BENTOV2_DOMAIN}
           - ${BENTOV2_PORTAL_DOMAIN}
           - ${BENTOV2_AUTH_DOMAIN}
+          - ${BENTO_MINIO_DOMAIN}
       drs-net:
         aliases:
           - ${BENTOV2_DOMAIN}
           - ${BENTOV2_PORTAL_DOMAIN}
           - ${BENTOV2_AUTH_DOMAIN}
+          - ${BENTO_MINIO_DOMAIN}
       event-relay-net:
         aliases:
           - ${BENTOV2_DOMAIN}

docker-compose.local.yaml

@@ -19,6 +19,7 @@ services:
       - ${PWD}/lib/public/en_about.html:/bento-public/src/public/en_about.html
       - ${PWD}/lib/public/fr_about.html:/bento-public/src/public/fr_about.html
       - ${PWD}/lib/public/branding.png:/bento-public/src/public/assets/branding.png
+      - ${PWD}/lib/public/branding.lightbg.png:/bento-public/src/public/assets/branding.lightbg.png
       - ${PWD}/lib/public/translations/en.json:/bento-public/src/public/locales/en/translation_en.json
       - ${PWD}/lib/public/translations/fr.json:/bento-public/src/public/locales/fr/translation_fr.json
     environment:

docker-compose.yaml

@@ -14,6 +14,7 @@ include:
       - lib/gohan/docker-compose.gohan.yaml  # Optional feature; controlled by a compose profile
       - lib/katsu/docker-compose.katsu.yaml
       - lib/logs/docker-compose.logs.yaml
+      - lib/minio/docker-compose.minio.yaml
       - lib/notification/docker-compose.notification.yaml
       - lib/public/docker-compose.public.yaml  # Optional feature; controlled by a compose profile
       - lib/redis/docker-compose.redis.yaml

docs/adding-services.md

@@ -0,0 +1,155 @@
+# Adding services to Bento
+
+There are two types of services in Bento:
+
+* Bento services, which have been developed by the Bento team specifically for the platform, and
+* other services, which support the Bento services or provide additional platform features.
+
+
+## Aspects to consider when adding any service to Bento
+
+### Environment variables
+
+* Service environment variables, used for configuring the image and some aspects of the service itself, should be added 
+  to `etc/bento.env`. These variables typically include:
+  * Image
+  * Image version (tag)
+  * Container name template
+  * Service Docker network (**Note:** we typically give each service its own network, and add services to multiple 
+    networks only as needed)
+  * Debugger ports
+* Configuration environment variables, for setting up feature flags and passwords, should be added to 
+  `etc/default_config.env` and the example files `etc/bento_deploy.env` and `etc/bento_dev.env`.
+  * `etc/default_config.env` contains feature flags and "empty definitions" for passwords/secrets. 
+  * `etc/bento_deploy.env` is an example / template setup (to be copied to `local.env`) for a production deployment.
+  * `etc/bento_dev.env` is an example / template setup (to be copied to `local.env`) for a development setup.
+
+### Container setup
+
+The service's Docker container must be set up via a Compose file in `lib/<service>/docker-compose.<service>.yaml`.
+This must then be included in the main `docker-compose.yaml` file, in the `include` block.
+
+The service's network (and potentially feature flag, if applicable), as well as container name and port environment 
+variables must be added to the gateway compose file (`lib/gateway/docker-compose.gateway.yaml`) if the service is to be 
+externally accessible.
+
+### Gateway configuration
+
+*As needed,* a gateway NGINX config must be placed into `lib/gateway/<public_services|services>`.
+
+### Required `bentoctl` changes
+
+Inside the `py_bentoctl` Python module:
+
+* If the service is locked behind a feature flag, add the feature (as an `BentoOptionalFeature` instance) to 
+  `config.py`, modeling it after other definitions.
+* Add the service image environment variables to the `service_image_vars` variable in `services.py`. 
+* If the service is not a Bento service (or does not have the `bento` user in the Docker image), add the service to the
+  `BENTO_USER_EXCLUDED_SERVICES` variable.
+* In `other_helpers.py`:
+  * If the service has a data directory that needs to be initialized, add an entry to the `data_dir_vars` variable 
+    in the `init_dirs(...)` function containing the name of the environment variable which points to the data volume 
+    directory.
+  * Add any entry with the name of the environment variable storing the name of the Docker network to the `networks` 
+    variable in the `init_docker(...)` function.
+  * If new certificates are needed, add new entries to the `init_self_signed_certs` function (for development purposes).
+
+### Documentation changes
+
+* Make sure to add a note about how to set up the service for the first time to the 
+  [Installation guide](./installation.md), as well as the migration guide for the version the service is introduced in.
+* If additional deployment steps are needed (i.e., new certificates), add a note to the 
+  [Deployment guide](./deployment.md).
+
+### Additional notes
+
+Non-Bento services **MUST NOT** be put into `etc/bento_services.json`; this file is for Bento services only (see below).
+
+
+## Additional considerations when adding new Bento services
+
+### User and base image
+
+It is expected that Bento services will use one of the 
+[Bento base images](https://github.com/bento-platform/bento_base_images).
+
+These images provide a `bento` user, whose UID is set to the host user's UID.
+
+### `/service-info` and service record in `bento_services.json`
+
+Bento services **MUST** implement the GA4GH [Service Info](https://www.ga4gh.org/product/service-info/) API.
+They must also be registered in the `etc/bento_services.json` file, which allows them to be loaded into the 
+[Bento Service Registry](https://github.com/bento-platform/bento_service_registry).
+
+Each entry of this file follows the format:
+
+```js
+{
+  // ...
+   "<compose ID>": {
+     "service_kind": "<service kind>",
+     "url_template": "{BENTO_PUBLIC_URL}/api/{service_kind}",
+     "repository": "[email protected]:bento-platform/<...>"
+   },
+  // ...
+}
+```
+
+In this format:
+* `<compose ID>` is the key of the service in its `docker-compose.<...>.yaml` file
+* `<service kind>` is a special Bento-unique identifier for the service, allowing front ends to look up services.
+* The `url_template` key is a template for the base URL used to access the service's API.
+* The `repository` key is an SSH Git repository URL for the service code, so it can be cloned into the `repos` folder
+  for development.
+
+
+### Making service-to-service requests go through the gateway (dev)
+
+Bento relies on three mechanisms to resolve hostnames to IP addresses:
+- DNS records (production only)
+  - `/etc/hosts` entries when in local dev
+  - For requests originating outside of the Docker networks (e.g. web browsers)
+- Container names (production & dev)
+  - When two containers are on the same Docker network and need to talk to each other directly
+  - Docker resolves a container's name to its IP on a Docker network
+  - e.g. Katsu can talk directly to DRS with `http://${BENTOV2_DRS_CONTAINER_NAME}:${BENTOV2_DRS_INTERNAL_PORT}`
+- Docker network aliases (dev only)
+  - When two services need to communicate with each other via the gateway only.
+  - In production, this is taken care of by DNS records
+
+When developing locally, some services may need to be interacted with strictly through the gateway.
+This is the case for Keycloak (auth) and Minio, as both services require a subdomain and HTTPS.
+
+As such, drop-box cannot use the Docker resolver in order to connect to Minio.
+
+Since we are in local, there is no DNS record to resolve Minio's domain, 
+and the host's `/etc/hosts` entries will not be of help from the container's perspective.
+
+For these situations, we rely on [Docker network aliases.](https://docs.docker.com/reference/compose-file/services/#aliases)
+
+Taking the Minio example, we need:
+  - Drop-Box to interact with Minio via the gateway
+  - DRS to interact with Minio via the gateway
+
+Enabling this is done by adding `${BENTO_MINIO_DOMAIN}` to the respective service networks aliases.
+
+This snippet comes from [docker-compose.dev.yaml](../docker-compose.dev.yaml):
+```yaml
+services:
+  gateway:
+    networks:
+      drop-box-net:
+        aliases:
+          - ${BENTOV2_DOMAIN}
+          - ${BENTOV2_PORTAL_DOMAIN}
+          - ${BENTOV2_AUTH_DOMAIN}
+          - ${BENTO_MINIO_DOMAIN}
+      drs-net:
+        aliases:
+          - ${BENTOV2_DOMAIN}
+          - ${BENTOV2_PORTAL_DOMAIN}
+          - ${BENTOV2_AUTH_DOMAIN}
+          - ${BENTO_MINIO_DOMAIN}
+```
+
+Doing so, we make sure that `${BENTO_MINIO_DOMAIN}` is resolved to the gateway for drop-box and DRS.

docs/deployment.md

@@ -9,6 +9,7 @@ BENTOV2_DOMAIN=bento.example.com
 BENTOV2_PORTAL_DOMAIN=portal.${BENTOV2_DOMAIN}
 BENTOV2_AUTH_DOMAIN=auth.${BENTOV2_DOMAIN}
 BENTOV2_CBIOPORTAL_DOMAIN=cbioportal.${BENTOV2_DOMAIN}
+BENTO_MINIO_DOMAIN=minio.${BENTOV2_DOMAIN}
 ```
 
 For a real deployment, make sure that your `local.env` file uses valid domain names for which SSL certificates 

docs/development.md

@@ -185,6 +185,11 @@ If subsequent modifications are made to the package's code, you will need to cre
 and install it again in the app with `npm install`.
 
 
+## Adding services
+
+See [`adding-services.md`](./adding-services.md) for some considerations when adding new services to Bento.
+
+
 ## Using Adminer
 
 An [Adminer](https://www.adminer.org/) container is deployed in dev and local mode, it can be used to inspect the

docs/installation.md

@@ -325,10 +325,18 @@ value in the `BENTOV2_AUTH_CLIENT_ID` environment variable. On local instances,
 this is set to `local_bentov2` by default.
 
 
-## 7. *Production only:* set up translations for Bento-Public
+## 7. *Production only:* set up translations and branding
 
-Now that Bento Public has been initialized by either `./bentoctl.bash init-all` or `./bentoctl.bash init-web public`,
-adjust the default translation set as necessary:
+Now that Bento Public and Web have been initialized by either `./bentoctl.bash init-all` or 
+`./bentoctl.bash init-web <public|web>`, translation files and branding (logos) can be configured as necessary.
+
+**For branding (logos)**, copy files to the following paths:
+
+* A logo which works on dark backgrounds should be placed at `lib/public/branding.png` and `lib/web/branding.png`.
+* A logo which works on light backgrounds should be placed at `lib/public/branding.lightbg.png`. This is primarily 
+  useful for Beacon Network. 
+
+**For translations** (which apply only to Bento Public), adjust the default translation set as necessary:
 
 ```js
 // lib/public/translations/<en|fr>.json

docs/migrating_to_18.md

@@ -0,0 +1,58 @@
+# Migrating to Bento v18
+
+* Bento v18 implements some new features for branding, which may require changes as described below.
+* It also adds MinIO as a backend for future S3-compatible object storage. 
+  * This can be enabled now, but will not be used until a future version. 
+  * For instances hosted on the Secure Data for Health (SD4H) infrastructure, the SD4H object store should be used for 
+    production instances when S3-compatible services become ready in a future version.
+
+
+## 1. Stop and update services, and initialize new networks
+
+```bash
+./bentoctl.bash stop
+./bentoctl.bash pull
+# set up new Docker networks for MinIO (needed even if MinIO is not set up)
+./bentoctl.bash init-docker
+```
+
+
+## 2. Set up light and dark-background branding for Bento Public
+
+* Make sure `lib/public/branding.png` and `lib/web/branding.png` are images which work on dark backgrounds.
+* **If you have a light-background logo to add:** put this file at `lib/public/branding.lightbg.png`.
+* **If you do not have a light-background logo:** run `./bentoctl.bash init-web public` to copy the Bento logo to the 
+  above location, or copy `branding.png` to `branding.lightbg.png`
+
+
+## 3. If desired, disable Bento Public sign in / portal links
+
+Version 18 includes two new Bento Public environment variables for customizing an instance:
+
+* `BENTO_PUBLIC_SHOW_PORTAL_LINK`: Whether to show a link to the private data portal in the header
+* `BENTO_PUBLIC_SHOW_SIGN_IN`: Whether to show a "Sign In" button in the header
+* `BENTO_PUBLIC_FORCE_CATALOGUE`: Whether to force the data catalogue to display, even with only a single project.
+
+The first two are set to `true` by default, and the last is set to `false`. If desired, they can be toggled to 
+non-default settings by modifying `local.env`, for example:
+
+```bash
+# ...
+BENTO_PUBLIC_SHOW_PORTAL_LINK='false'
+BENTO_PUBLIC_SHOW_SIGN_IN='false'
+BENTO_PUBLIC_FORCE_CATALOGUE='true'
+# ...
+```
+
+
+## 4. Enabling MinIO
+
+To enable the deployment of a MinIO server for S3 storage, refer to the documentation on 
+[configuring MinIO for Bento](/docs/minio.md).
+
+
+## 5. Restart services
+
+```bash
+./bentoctl.bash start
+```