Skip to content

Commit

Permalink
Updating sower job sa so we can add annotations without enabled "aws"… (
Browse files Browse the repository at this point in the history
#242)

* Updating sower job sa so we can add annotations without enabled "aws" in the global configmap.

Also, adding the ability to pass in "env" values to specific jobs/deployment.

Updating and adding new configurations for revproxy nginx.

* version bump
  • Loading branch information
EliseCastle23 authored Feb 17, 2025
1 parent 77de91a commit 282b4f7
Show file tree
Hide file tree
Showing 23 changed files with 149 additions and 58 deletions.
2 changes: 1 addition & 1 deletion helm/fence/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.29
version: 0.1.30

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
Expand Down
5 changes: 3 additions & 2 deletions helm/fence/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# fence

![Version: 0.1.29](https://img.shields.io/badge/Version-0.1.29-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)
![Version: 0.1.30](https://img.shields.io/badge/Version-0.1.30-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)

A Helm chart for gen3 Fence

Expand Down Expand Up @@ -184,9 +184,10 @@ A Helm chart for gen3 Fence
| serviceAccount.create | bool | `true` | Specifies whether a service account should be created. |
| serviceAccount.name | string | `"fence-sa"` | The name of the service account |
| tolerations | list | `[]` | Tolerations for the pods |
| usersync | map | `{"addDbgap":false,"custom_image":null,"onlyDbgap":false,"schedule":"*/30 * * * *","slack_send_dbgap":false,"slack_webhook":"None","syncFromDbgap":false,"userYamlS3Path":"s3://cdis-gen3-users/helm-test/user.yaml","usersync":false}` | Configuration options for usersync cronjob. |
| usersync | map | `{"addDbgap":false,"custom_image":null,"env":null,"onlyDbgap":false,"schedule":"*/30 * * * *","slack_send_dbgap":false,"slack_webhook":"None","syncFromDbgap":false,"userYamlS3Path":"s3://cdis-gen3-users/helm-test/user.yaml","usersync":false}` | Configuration options for usersync cronjob. |
| usersync.addDbgap | bool | `false` | Force attempting a dbgap sync if "true", falls back on user.yaml |
| usersync.custom_image | string | `nil` | To set a custom image for pulling the user.yaml file from S3. Default is the Gen3 Awshelper image. |
| usersync.env | list | `nil` | Environment variables to pass to the job. |
| usersync.onlyDbgap | bool | `false` | Forces ONLY a dbgap sync if "true", IGNORING user.yaml |
| usersync.schedule | string | `"*/30 * * * *"` | The cron schedule expression to use in the usersync cronjob. Runs every 30 minutes by default. |
| usersync.slack_send_dbgap | bool | `false` | Will echo what files we are seeing on dbgap ftp to Slack. |
Expand Down
42 changes: 25 additions & 17 deletions helm/fence/templates/usersync-cron.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -71,15 +71,18 @@ spec:
image: {{ .Values.usersync.custom_image | default "quay.io/cdis/awshelper:master" }}
imagePullPolicy: Always
env:
- name: gen3Env
valueFrom:
- name: gen3Env
valueFrom:
configMapKeyRef:
name: manifest-global
key: hostname
- name: userYamlS3Path
value: {{ .Values.usersync.userYamlS3Path | quote }}
- name: slackWebHook
value: {{ .Values.usersync.slack_webhook | quote }}
- name: userYamlS3Path
value: {{ .Values.usersync.userYamlS3Path | quote }}
- name: slackWebHook
value: {{ .Values.usersync.slack_webhook | quote }}
{{- with .Values.usersync.env }}
{{- toYaml . | nindent 16 }}
{{- end }}
volumeMounts:
- name: user-yaml
mountPath: /var/www/fence
Expand Down Expand Up @@ -131,17 +134,22 @@ spec:
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: Always
env:
- name: SYNC_FROM_DBGAP
value: {{ .Values.usersync.syncFromDbgap | quote }}
- name: ADD_DBGAP
value: {{ .Values.usersync.addDbgap | quote }}
- name: ONLY_DBGAP
value: {{ .Values.usersync.onlyDbgap | quote }}
- name: SLACK_SEND_DBGAP
value: {{ .Values.usersync.slack_send_dbgap | quote }}
- name: slackWebHook
value: {{ .Values.usersync.slack_webhook | quote }}
{{- toYaml .Values.env | nindent 12 }}
- name: SYNC_FROM_DBGAP
value: {{ .Values.usersync.syncFromDbgap | quote }}
- name: ADD_DBGAP
value: {{ .Values.usersync.addDbgap | quote }}
- name: ONLY_DBGAP
value: {{ .Values.usersync.onlyDbgap | quote }}
- name: SLACK_SEND_DBGAP
value: {{ .Values.usersync.slack_send_dbgap | quote }}
- name: slackWebHook
value: {{ .Values.usersync.slack_webhook | default "" | quote }}
{{- with .Values.env }}
{{- toYaml . | nindent 14 }}
{{- end }}
{{- with .Values.usersync.env }}
{{- toYaml . | nindent 14 }}
{{- end }}
volumeMounts:
- name: shared-data
mountPath: /mnt/shared
Expand Down
2 changes: 2 additions & 0 deletions helm/fence/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,8 @@ usersync:
slack_webhook: None
# -- (bool) Will echo what files we are seeing on dbgap ftp to Slack.
slack_send_dbgap: false
# -- (list) Environment variables to pass to the job.
env:

# -- (map) Secret information for Usersync and External Secrets.
secrets:
Expand Down
12 changes: 6 additions & 6 deletions helm/gen3/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ dependencies:
repository: "file://../frontend-framework"
condition: frontend-framework.enabled
- name: fence
version: 0.1.29
version: 0.1.30
repository: "file://../fence"
condition: fence.enabled
- name: guppy
Expand All @@ -60,7 +60,7 @@ dependencies:
repository: "file://../metadata"
condition: metadata.enabled
- name: peregrine
version: 0.1.17
version: 0.1.18
repository: "file://../peregrine"
condition: peregrine.enabled
- name: portal
Expand All @@ -72,19 +72,19 @@ dependencies:
repository: "file://../requestor"
condition: requestor.enabled
- name: revproxy
version: 0.1.20
version: 0.1.21
repository: "file://../revproxy"
condition: revproxy.enabled
- name: sheepdog
version: 0.1.20
repository: "file://../sheepdog"
condition: sheepdog.enabled
- name: ssjdispatcher
version: 0.1.17
version: 0.1.18
repository: "file://../ssjdispatcher"
condition: ssjdispatcher.enabled
- name: sower
version: 0.1.16
version: 0.1.17
condition: sower.enabled
repository: "file://../sower"
- name: wts
Expand Down Expand Up @@ -128,7 +128,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.59
version: 0.1.60

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
Expand Down
12 changes: 6 additions & 6 deletions helm/gen3/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# gen3

![Version: 0.1.58](https://img.shields.io/badge/Version-0.1.58-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)
![Version: 0.1.60](https://img.shields.io/badge/Version-0.1.60-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)

Helm chart to deploy Gen3 Data Commons

Expand All @@ -25,7 +25,7 @@ Helm chart to deploy Gen3 Data Commons
| file://../aws-es-proxy | aws-es-proxy | 0.1.13 |
| file://../common | common | 0.1.16 |
| file://../etl | etl | 0.1.10 |
| file://../fence | fence | 0.1.29 |
| file://../fence | fence | 0.1.30 |
| file://../frontend-framework | frontend-framework | 0.1.6 |
| file://../gen3-network-policies | gen3-network-policies | 0.1.2 |
| file://../guppy | guppy | 0.1.17 |
Expand All @@ -34,13 +34,13 @@ Helm chart to deploy Gen3 Data Commons
| file://../manifestservice | manifestservice | 0.1.18 |
| file://../metadata | metadata | 0.1.17 |
| file://../neuvector | neuvector | 0.1.2 |
| file://../peregrine | peregrine | 0.1.17 |
| file://../peregrine | peregrine | 0.1.18 |
| file://../portal | portal | 0.1.26 |
| file://../requestor | requestor | 0.1.16 |
| file://../revproxy | revproxy | 0.1.19 |
| file://../revproxy | revproxy | 0.1.21 |
| file://../sheepdog | sheepdog | 0.1.20 |
| file://../sower | sower | 0.1.16 |
| file://../ssjdispatcher | ssjdispatcher | 0.1.16 |
| file://../sower | sower | 0.1.17 |
| file://../ssjdispatcher | ssjdispatcher | 0.1.18 |
| file://../wts | wts | 0.1.18 |
| https://charts.bitnami.com/bitnami | postgresql | 11.9.13 |
| https://helm.elastic.co | elasticsearch | 7.10.2 |
Expand Down
2 changes: 1 addition & 1 deletion helm/peregrine/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.17
version: 0.1.18

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
Expand Down
2 changes: 1 addition & 1 deletion helm/peregrine/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# peregrine

![Version: 0.1.17](https://img.shields.io/badge/Version-0.1.17-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)
![Version: 0.1.18](https://img.shields.io/badge/Version-0.1.18-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)

A Helm chart for gen3 Peregrine service

Expand Down
5 changes: 4 additions & 1 deletion helm/peregrine/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,9 @@ spec:
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
env:
{{- with .Values.env }}
{{- toYaml . | nindent 12 }}
{{- end }}
- name: FENCE_DB_USER
valueFrom:
secretKeyRef:
Expand Down Expand Up @@ -149,7 +152,7 @@ spec:
- name: GEN3_SIDECAR
value: "False"
- name: CONF_HOSTNAME
value: {{ .Values.global.hostname }}
value: {{ .Values.global.hostname | quote }}
{{- with .Values.volumeMounts }}
volumeMounts:
{{- toYaml . | nindent 10 }}
Expand Down
2 changes: 1 addition & 1 deletion helm/revproxy/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.20
version: 0.1.21

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
Expand Down
5 changes: 3 additions & 2 deletions helm/revproxy/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# revproxy

![Version: 0.1.19](https://img.shields.io/badge/Version-0.1.19-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)
![Version: 0.1.21](https://img.shields.io/badge/Version-0.1.21-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)

A Helm chart for gen3 revproxy

Expand All @@ -23,10 +23,11 @@ A Helm chart for gen3 revproxy
| commonLabels | map | `nil` | Will completely override the commonLabels defined in the common chart's _label_setup.tpl |
| criticalService | string | `"true"` | Valid options are "true" or "false". If invalid option is set- the value will default to "false". |
| fullnameOverride | string | `""` | Override the full name of the deployment. |
| global.aws | map | `{"awsAccessKeyId":null,"awsSecretAccessKey":null,"enabled":false,"wafv2":{"enabled":false,"wafAclArn":null}}` | AWS configuration |
| global.aws | map | `{"awsAccessKeyId":null,"awsSecretAccessKey":null,"enabled":false,"scheme":"internet-facing","wafv2":{"enabled":false,"wafAclArn":null}}` | AWS configuration |
| global.aws.awsAccessKeyId | string | `nil` | Credentials for AWS stuff. |
| global.aws.awsSecretAccessKey | string | `nil` | Credentials for AWS stuff. |
| global.aws.enabled | bool | `false` | Set to true if deploying to AWS. Controls ingress annotations. |
| global.aws.scheme | string | `"internet-facing"` | internal or internet-facing |
| global.aws.wafv2 | map | `{"enabled":false,"wafAclArn":null}` | WAF configuration |
| global.aws.wafv2.enabled | bool | `false` | Set to true if using AWS WAFv2 |
| global.aws.wafv2.wafAclArn | string | `nil` | ARN for the WAFv2 ACL. |
Expand Down
19 changes: 19 additions & 0 deletions helm/revproxy/gen3.nginx.conf/argo-server.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
location /argo/ {
error_page 403 @errorworkspace;
set $authz_resource "/argo";
set $authz_method "access";
set $authz_service "argo";
# be careful - sub-request runs in same context as this request
auth_request /gen3-authz;

set $proxy_service "argo";
set $upstream SERVICE_URL;

rewrite ^/argo/(.*) /$1 break;

proxy_set_header Connection '';
proxy_http_version 1.1;
chunked_transfer_encoding off;

proxy_pass $upstream;
}
12 changes: 12 additions & 0 deletions helm/revproxy/gen3.nginx.conf/gen3-discovery-ai-service.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
location /ai {
if ($csrf_check !~ ^ok-\S.+$) {
return 403 "failed csrf check";
}

set $proxy_service "gen3-discovery-ai-service";
set $upstream http://gen3-discovery-ai-service$des_domain;
rewrite ^/ai/(.*) /$1 break;
proxy_pass $upstream;
proxy_redirect http://$host/ https://$host/ai/;
client_max_body_size 0;
}
12 changes: 12 additions & 0 deletions helm/revproxy/gen3.nginx.conf/gen3-user-data-library-service.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
location /library {
if ($csrf_check !~ ^ok-\S.+$) {
return 403 "failed csrf check";
}

set $proxy_service "gen3-user-data-library-service";
set $upstream http://gen3-user-data-library-service$des_domain;
rewrite ^/library/(.*) /$1 break;
proxy_pass $upstream;
proxy_redirect http://$host/ https://$host/library/;
client_max_body_size 0;
}
20 changes: 20 additions & 0 deletions helm/revproxy/gen3.nginx.conf/gen3-workflow-service.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
location /ga4gh/tes/v1/ {
if ($csrf_check !~ ^ok-\S.+$) {
return 403 "failed csrf check";
}

set $proxy_service "gen3-workflow";
set $upstream http://gen3-workflow-service.$namespace.svc.cluster.local;
proxy_pass $upstream;
}

location /workflows/ {
if ($csrf_check !~ ^ok-\S.+$) {
return 403 "failed csrf check";
}

set $proxy_service "gen3-workflow";
set $upstream http://gen3-workflow-service.$namespace.svc.cluster.local;
rewrite ^/workflows/(.*) /$1 break;
proxy_pass $upstream;
}
28 changes: 16 additions & 12 deletions helm/revproxy/gen3.nginx.conf/guppy-service.conf
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
location /guppy/ {
proxy_connect_timeout 600s;
proxy_send_timeout 600s;
proxy_read_timeout 600s;
send_timeout 600s;
location /guppy/ {
if ($csrf_check !~ ^ok-\S.+$) {
return 403 "failed csrf check, make sure data-portal version >= 2023.12 or >= 5.19.0";
}

proxy_connect_timeout 600s;
proxy_send_timeout 600s;
proxy_read_timeout 600s;
send_timeout 600s;

set $proxy_service "guppy";
# upstream is written to logs
set $upstream http://guppy-service.$namespace.svc.cluster.local;
rewrite ^/guppy/(.*) /$1 break;
proxy_pass $upstream;
client_max_body_size 0;
}
set $proxy_service "guppy";
# upstream is written to logs
set $upstream http://guppy-service.$namespace.svc.cluster.local;
rewrite ^/guppy/(.*) /$1 break;
proxy_pass $upstream;
client_max_body_size 0;
}
2 changes: 1 addition & 1 deletion helm/sower/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.16
version: 0.1.17

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
Expand Down
3 changes: 2 additions & 1 deletion helm/sower/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# sower

![Version: 0.1.16](https://img.shields.io/badge/Version-0.1.16-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)
![Version: 0.1.17](https://img.shields.io/badge/Version-0.1.17-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)

A Helm chart for gen3 sower

Expand Down Expand Up @@ -31,6 +31,7 @@ A Helm chart for gen3 sower
| awsStsRegionalEndpoints | string | `"regional"` | AWS STS to issue temporary credentials to users and roles that make an AWS STS request. Values regional or global. |
| commonLabels | map | `nil` | Will completely override the commonLabels defined in the common chart's _label_setup.tpl |
| criticalService | string | `"false"` | Valid options are "true" or "false". If invalid option is set- the value will default to "false". |
| env | list | `nil` | Environment variables to pass to the container |
| externalSecrets | map | `{"createK8sPelicanServiceSecret":false,"createK8sSowerJobsSecret":false,"pelicanserviceG3auto":null,"sowerjobsG3auto":null}` | External Secrets settings. |
| externalSecrets.createK8sPelicanServiceSecret | string | `false` | Will create the Helm "pelicanservice-g3auto" secret even if Secrets Manager is enabled. This is helpful if you are wanting to use External Secrets for some, but not all secrets. |
| externalSecrets.createK8sSowerJobsSecret | string | `false` | Will create the Helm "sower-jobs-g3auto" secret even if Secrets Manager is enabled. This is helpful if you are wanting to use External Secrets for some, but not all secrets. |
Expand Down
3 changes: 3 additions & 0 deletions helm/sower/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ spec:
volumeMounts:
{{- toYaml .Values.volumeMounts | nindent 12 }}
env:
{{- with .Values.env }}
{{- toYaml . | nindent 12 }}
{{- end }}
- name: POD_NAMESPACE
valueFrom:
fieldRef:
Expand Down
Loading

0 comments on commit 282b4f7

Please sign in to comment.