Waiting for review: Chore(QA): DCF PROD Release 2020 Q1 1585800777 #1330

themarcelor · 2020-04-02T04:19:05Z

copying files from nci-crdc-staging.datacommons.io to nci-crdc.datacommons.io.

This PR has been created with:

python gen3release-sdk/env_cli.py copy 
-s ~/workspace/cdis-manifest/nci-crdc-staging.datacommons.io 
-e ~/workspace/cdis-manifest/nci-crdc.datacommons.io 
-pr "JUST A TEST, PLEASE IGNORE: Chore(QA): DCF PROD Release 2020 Q1 1585800777"

…mmons.io

PlanXCyborg · 2020-04-02T04:26:11Z

nci-crdc.datacommons.io/manifest.json

Deployment changes

fence
- In fence-config, the region field of S3_BUCKETS is now required: if not specified, Fence will print a warning and try to call GetBucketLocation. Fence will error at startup if that doesn't work. (Fix/make region required for s3 bucket config fence#749)
peregrine
- ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) peregrine#142)
- Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist peregrine#138)
- Requires Arborist to be deployed (Feat/arborist peregrine#138)
- look for INDEX_CLIENT instead of SIGNPOST in app.config (Remove signpost peregrine#135)
- get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (Remove signpost peregrine#135)
- Now uWSGI worker process will kill itself (harakiri) and restart depending on reverse proxy timeout (PXP-2441: graceful timeout in peregrine peregrine#112)
- a new environment variable PUBLIC_DATASETS will be required to set for anonymous access to datasets endpoint (feat(datasets): add datasets endpoint peregrine#111)
- Now the traversal cache is no longer loaded at startup, so it should be faster to start (PXD-2355: lazy load traversal cache peregrine#104)
portal
- New optional title parameter in Terra Export Warning configuration: terraExportWarning: { subjectThreshold: int, title: string(optional), message: string(optional) } (uc-cdis/portal#656)
- Added terraExportWarning: { subjectThreshold: <number>, message: <string, optional> } to portal config. (uc-cdis/portal#642)
- New optional portal config flag: dataExplorerConfig.terraTemplate. The value of this parameter is placed in the Terra export URL as the value of the query parameter 'template'. (uc-cdis/portal#645)
- InCommon login needs fence 4.8.0 (uc-cdis/portal#595)
- Need Guppy 0.3.7+ to handle large file size (uc-cdis/portal#603)
- if deployed with "useArboristUI": true, needs Arborist's GET /auth/mapping endpoint to be exposed via revproxy. (useArboristUI false by default) (uc-cdis/portal#592)
- New optional portal config field called adminPreAppliedFilters. Goes in data/config/*.json. (uc-cdis/portal#576)
- For a "cert" if the required document is a quiz, then hasCorrectAnswers should be true, otherwise false. (uc-cdis/portal#571)
- Windmill lists unmapped files by checking Indexd. Before centralized auth's deployment, a file was determined to be unmapped by checking if the acl field was null. After centralized auth, we use the authz field instead. The new manifest configuration setting use_indexd_authz in the global block specifies which of the two fields should be used. This field defaults to false. (uc-cdis/portal#570)
- Deployed in https://qa-canine.planx-pla.net with fence version < 3, index authz field set to false (uc-cdis/portal#570)
- (by default). (uc-cdis/portal#570)
- Deployed in https://qa-brain.planx-pla.net with fence version > 3, index authz field set to true. (uc-cdis/portal#570)
- New portal configuration and images for AnVIL (uc-cdis/portal#569)
- Requires a url to be specified in dataExplorerConfig called terraExportURL (uc-cdis/portal#568)
- This should be bvdp-saturn-${env}.appspot.com/#import-data, where ${env} is dev, staging, or prod (uc-cdis/portal#568)
- Needs a version of cloud-automation that exposes arborist's auth/proxy endpoint (uc-cdis/portal#565)
- fixed DAT requires Guppy >= 0.3.1 (uc-cdis/portal#563)
- DAT requires Guppy >= 0.3.0 (uc-cdis/portal#561)
- This change requires sower version >= 0.3.0. (uc-cdis/portal#562)
- This PR requires feat/support is not op for filter guppy#45 (0.2.1+) if want to use Guppy for NIAID (uc-cdis/portal#558)
- require fileExplorerConfig block with buttons (uc-cdis/portal#554)
- New manifest.json variable: logout_inactive_users. Defaults to true if not provided. (uc-cdis/portal#526)
- Feat/logout user opt out cloud-automation#818 (uc-cdis/portal#526)
- enable source map tool so that developers could have better debugging under dev mode (uc-cdis/portal#514)
- Fix warning from fsevents packages during docker image building (uc-cdis/portal#488)
- Add flag to npm ci in dockerfile to support npm prepared files from ui-component library (uc-cdis/portal#488)
- Workspace button in DCF Sandbox (uc-cdis/portal#466)
sheepdog
- Fix Travis coverage reports (combine the results of all 3 tests into one coverage report). (PXP-3408 Update Sheepdog to Python3 sheepdog#314)
- ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) sheepdog#304)
- Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist2 sheepdog#298)
- Requires Arborist to be deployed (Feat/arborist2 sheepdog#298)
- look for INDEX_CLIENT instead of SIGNPOST in app.config (remove signpost from sheepdog deps sheepdog#291)
- get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (remove signpost from sheepdog deps sheepdog#291)
- Sheepdog will now create (or replace) the missing indexes during migration (PXD-2916: implement uniqueKeys sheepdog#259)
- New unique indexes have been added to all nodes PXD-2916: add unique indexes for uniqueKeys gen3datamodel#12 (PXD-2916: implement uniqueKeys sheepdog#259)
- Now uWSGI worker process will kill itself (harakiri) and restart depending on reverse proxy timeout (PXP-2441: use harakiri timeout sheepdog#261)

itsJiaqi · 2020-04-07T14:49:34Z

nci-crdc.datacommons.io/manifests/fence/fence-config-public.yaml

@@ -1,137 +1,14 @@
 S3_BUCKETS:
-  tcga-protected-dcf-databucket-gen3:


I'm not sure why this has been deleted?

This PR was created by a new Gen3-QA autoamtion (from our gen3release-sdk) so it blindly copies everything from preprod to prod.
That seems to be one exception I will need to accommodate in that PR-creation logic.

Thanks for checking!

PlanXCyborg · 2020-04-07T15:51:42Z

nci-crdc.datacommons.io/manifest.json

Deployment changes

fence
- In fence-config, the region field of S3_BUCKETS is now required: if not specified, Fence will print a warning and try to call GetBucketLocation. Fence will error at startup if that doesn't work. (Fix/make region required for s3 bucket config fence#749)
peregrine
- ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) peregrine#142)
- Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist peregrine#138)
- Requires Arborist to be deployed (Feat/arborist peregrine#138)
- look for INDEX_CLIENT instead of SIGNPOST in app.config (Remove signpost peregrine#135)
- get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (Remove signpost peregrine#135)
- Now uWSGI worker process will kill itself (harakiri) and restart depending on reverse proxy timeout (PXP-2441: graceful timeout in peregrine peregrine#112)
- a new environment variable PUBLIC_DATASETS will be required to set for anonymous access to datasets endpoint (feat(datasets): add datasets endpoint peregrine#111)
- Now the traversal cache is no longer loaded at startup, so it should be faster to start (PXD-2355: lazy load traversal cache peregrine#104)
portal
- New optional title parameter in Terra Export Warning configuration: terraExportWarning: { subjectThreshold: int, title: string(optional), message: string(optional) } (uc-cdis/portal#656)
- Added terraExportWarning: { subjectThreshold: <number>, message: <string, optional> } to portal config. (uc-cdis/portal#642)
- New optional portal config flag: dataExplorerConfig.terraTemplate. The value of this parameter is placed in the Terra export URL as the value of the query parameter 'template'. (uc-cdis/portal#645)
- InCommon login needs fence 4.8.0 (uc-cdis/portal#595)
- Need Guppy 0.3.7+ to handle large file size (uc-cdis/portal#603)
- if deployed with "useArboristUI": true, needs Arborist's GET /auth/mapping endpoint to be exposed via revproxy. (useArboristUI false by default) (uc-cdis/portal#592)
- New optional portal config field called adminPreAppliedFilters. Goes in data/config/*.json. (uc-cdis/portal#576)
- For a "cert" if the required document is a quiz, then hasCorrectAnswers should be true, otherwise false. (uc-cdis/portal#571)
- Windmill lists unmapped files by checking Indexd. Before centralized auth's deployment, a file was determined to be unmapped by checking if the acl field was null. After centralized auth, we use the authz field instead. The new manifest configuration setting use_indexd_authz in the global block specifies which of the two fields should be used. This field defaults to false. (uc-cdis/portal#570)
- Deployed in https://qa-canine.planx-pla.net with fence version < 3, index authz field set to false (uc-cdis/portal#570)
- (by default). (uc-cdis/portal#570)
- Deployed in https://qa-brain.planx-pla.net with fence version > 3, index authz field set to true. (uc-cdis/portal#570)
- New portal configuration and images for AnVIL (uc-cdis/portal#569)
- Requires a url to be specified in dataExplorerConfig called terraExportURL (uc-cdis/portal#568)
- This should be bvdp-saturn-${env}.appspot.com/#import-data, where ${env} is dev, staging, or prod (uc-cdis/portal#568)
- Needs a version of cloud-automation that exposes arborist's auth/proxy endpoint (uc-cdis/portal#565)
- fixed DAT requires Guppy >= 0.3.1 (uc-cdis/portal#563)
- DAT requires Guppy >= 0.3.0 (uc-cdis/portal#561)
- This change requires sower version >= 0.3.0. (uc-cdis/portal#562)
- This PR requires feat/support is not op for filter guppy#45 (0.2.1+) if want to use Guppy for NIAID (uc-cdis/portal#558)
- require fileExplorerConfig block with buttons (uc-cdis/portal#554)
- New manifest.json variable: logout_inactive_users. Defaults to true if not provided. (uc-cdis/portal#526)
- Feat/logout user opt out cloud-automation#818 (uc-cdis/portal#526)
- enable source map tool so that developers could have better debugging under dev mode (uc-cdis/portal#514)
- Fix warning from fsevents packages during docker image building (uc-cdis/portal#488)
- Add flag to npm ci in dockerfile to support npm prepared files from ui-component library (uc-cdis/portal#488)
- Workspace button in DCF Sandbox (uc-cdis/portal#466)
sheepdog
- Fix Travis coverage reports (combine the results of all 3 tests into one coverage report). (PXP-3408 Update Sheepdog to Python3 sheepdog#314)
- ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) sheepdog#304)
- Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist2 sheepdog#298)
- Requires Arborist to be deployed (Feat/arborist2 sheepdog#298)
- look for INDEX_CLIENT instead of SIGNPOST in app.config (remove signpost from sheepdog deps sheepdog#291)
- get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (remove signpost from sheepdog deps sheepdog#291)
- Sheepdog will now create (or replace) the missing indexes during migration (PXD-2916: implement uniqueKeys sheepdog#259)
- New unique indexes have been added to all nodes PXD-2916: add unique indexes for uniqueKeys gen3datamodel#12 (PXD-2916: implement uniqueKeys sheepdog#259)
- Now uWSGI worker process will kill itself (harakiri) and restart depending on reverse proxy timeout (PXP-2441: use harakiri timeout sheepdog#261)

PlanXCyborg · 2020-04-07T15:55:46Z

nci-crdc.datacommons.io/manifest.json

Deployment changes

fence
- In fence-config, the region field of S3_BUCKETS is now required: if not specified, Fence will print a warning and try to call GetBucketLocation. Fence will error at startup if that doesn't work. (Fix/make region required for s3 bucket config fence#749)
peregrine
- ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) peregrine#142)
- Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist peregrine#138)
- Requires Arborist to be deployed (Feat/arborist peregrine#138)
- look for INDEX_CLIENT instead of SIGNPOST in app.config (Remove signpost peregrine#135)
- get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (Remove signpost peregrine#135)
- Now uWSGI worker process will kill itself (harakiri) and restart depending on reverse proxy timeout (PXP-2441: graceful timeout in peregrine peregrine#112)
- a new environment variable PUBLIC_DATASETS will be required to set for anonymous access to datasets endpoint (feat(datasets): add datasets endpoint peregrine#111)
- Now the traversal cache is no longer loaded at startup, so it should be faster to start (PXD-2355: lazy load traversal cache peregrine#104)
portal
- New optional title parameter in Terra Export Warning configuration: terraExportWarning: { subjectThreshold: int, title: string(optional), message: string(optional) } (uc-cdis/portal#656)
- Added terraExportWarning: { subjectThreshold: <number>, message: <string, optional> } to portal config. (uc-cdis/portal#642)
- New optional portal config flag: dataExplorerConfig.terraTemplate. The value of this parameter is placed in the Terra export URL as the value of the query parameter 'template'. (uc-cdis/portal#645)
- InCommon login needs fence 4.8.0 (uc-cdis/portal#595)
- Need Guppy 0.3.7+ to handle large file size (uc-cdis/portal#603)
- if deployed with "useArboristUI": true, needs Arborist's GET /auth/mapping endpoint to be exposed via revproxy. (useArboristUI false by default) (uc-cdis/portal#592)
- New optional portal config field called adminPreAppliedFilters. Goes in data/config/*.json. (uc-cdis/portal#576)
- For a "cert" if the required document is a quiz, then hasCorrectAnswers should be true, otherwise false. (uc-cdis/portal#571)
- Windmill lists unmapped files by checking Indexd. Before centralized auth's deployment, a file was determined to be unmapped by checking if the acl field was null. After centralized auth, we use the authz field instead. The new manifest configuration setting use_indexd_authz in the global block specifies which of the two fields should be used. This field defaults to false. (uc-cdis/portal#570)
- Deployed in https://qa-canine.planx-pla.net with fence version < 3, index authz field set to false (uc-cdis/portal#570)
- (by default). (uc-cdis/portal#570)
- Deployed in https://qa-brain.planx-pla.net with fence version > 3, index authz field set to true. (uc-cdis/portal#570)
- New portal configuration and images for AnVIL (uc-cdis/portal#569)
- Requires a url to be specified in dataExplorerConfig called terraExportURL (uc-cdis/portal#568)
- This should be bvdp-saturn-${env}.appspot.com/#import-data, where ${env} is dev, staging, or prod (uc-cdis/portal#568)
- Needs a version of cloud-automation that exposes arborist's auth/proxy endpoint (uc-cdis/portal#565)
- fixed DAT requires Guppy >= 0.3.1 (uc-cdis/portal#563)
- DAT requires Guppy >= 0.3.0 (uc-cdis/portal#561)
- This change requires sower version >= 0.3.0. (uc-cdis/portal#562)
- This PR requires feat/support is not op for filter guppy#45 (0.2.1+) if want to use Guppy for NIAID (uc-cdis/portal#558)
- require fileExplorerConfig block with buttons (uc-cdis/portal#554)
- New manifest.json variable: logout_inactive_users. Defaults to true if not provided. (uc-cdis/portal#526)
- Feat/logout user opt out cloud-automation#818 (uc-cdis/portal#526)
- enable source map tool so that developers could have better debugging under dev mode (uc-cdis/portal#514)
- Fix warning from fsevents packages during docker image building (uc-cdis/portal#488)
- Add flag to npm ci in dockerfile to support npm prepared files from ui-component library (uc-cdis/portal#488)
- Workspace button in DCF Sandbox (uc-cdis/portal#466)
sheepdog
- Fix Travis coverage reports (combine the results of all 3 tests into one coverage report). (PXP-3408 Update Sheepdog to Python3 sheepdog#314)
- ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) sheepdog#304)
- Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist2 sheepdog#298)
- Requires Arborist to be deployed (Feat/arborist2 sheepdog#298)
- look for INDEX_CLIENT instead of SIGNPOST in app.config (remove signpost from sheepdog deps sheepdog#291)
- get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (remove signpost from sheepdog deps sheepdog#291)
- Sheepdog will now create (or replace) the missing indexes during migration (PXD-2916: implement uniqueKeys sheepdog#259)
- New unique indexes have been added to all nodes PXD-2916: add unique indexes for uniqueKeys gen3datamodel#12 (PXD-2916: implement uniqueKeys sheepdog#259)
- Now uWSGI worker process will kill itself (harakiri) and restart depending on reverse proxy timeout (PXP-2441: use harakiri timeout sheepdog#261)

itsJiaqi · 2020-04-07T15:57:02Z

nci-crdc.datacommons.io/manifest.json

    "fluentd": "fluent/fluentd-kubernetes-daemonset:v1.2-debian-cloudwatch",
-    "jupyterhub": "quay.io/occ_data/jupyterhub:master"
+    "metadata": "quay.io/cdis/metadata-service:2020.03",


Oh I did not realize metadata service was part of the March Release, that's really cool, I'm okay with rolling it out.

That has been copied over from DCF staging.
Checking...

PlanXCyborg · 2020-04-07T16:33:26Z

nci-crdc.datacommons.io/manifest.json

⚠️ Services on branch

metadata

Deployment changes

fence
- In fence-config, the region field of S3_BUCKETS is now required: if not specified, Fence will print a warning and try to call GetBucketLocation. Fence will error at startup if that doesn't work. (Fix/make region required for s3 bucket config fence#749)
peregrine
- ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) peregrine#142)
- Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist peregrine#138)
- Requires Arborist to be deployed (Feat/arborist peregrine#138)
- look for INDEX_CLIENT instead of SIGNPOST in app.config (Remove signpost peregrine#135)
- get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (Remove signpost peregrine#135)
- Now uWSGI worker process will kill itself (harakiri) and restart depending on reverse proxy timeout (PXP-2441: graceful timeout in peregrine peregrine#112)
- a new environment variable PUBLIC_DATASETS will be required to set for anonymous access to datasets endpoint (feat(datasets): add datasets endpoint peregrine#111)
- Now the traversal cache is no longer loaded at startup, so it should be faster to start (PXD-2355: lazy load traversal cache peregrine#104)
portal
- New optional title parameter in Terra Export Warning configuration: terraExportWarning: { subjectThreshold: int, title: string(optional), message: string(optional) } (uc-cdis/portal#656)
- Added terraExportWarning: { subjectThreshold: <number>, message: <string, optional> } to portal config. (uc-cdis/portal#642)
- New optional portal config flag: dataExplorerConfig.terraTemplate. The value of this parameter is placed in the Terra export URL as the value of the query parameter 'template'. (uc-cdis/portal#645)
- InCommon login needs fence 4.8.0 (uc-cdis/portal#595)
- Need Guppy 0.3.7+ to handle large file size (uc-cdis/portal#603)
- if deployed with "useArboristUI": true, needs Arborist's GET /auth/mapping endpoint to be exposed via revproxy. (useArboristUI false by default) (uc-cdis/portal#592)
- New optional portal config field called adminPreAppliedFilters. Goes in data/config/*.json. (uc-cdis/portal#576)
- For a "cert" if the required document is a quiz, then hasCorrectAnswers should be true, otherwise false. (uc-cdis/portal#571)
- Windmill lists unmapped files by checking Indexd. Before centralized auth's deployment, a file was determined to be unmapped by checking if the acl field was null. After centralized auth, we use the authz field instead. The new manifest configuration setting use_indexd_authz in the global block specifies which of the two fields should be used. This field defaults to false. (uc-cdis/portal#570)
- Deployed in https://qa-canine.planx-pla.net with fence version < 3, index authz field set to false (uc-cdis/portal#570)
- (by default). (uc-cdis/portal#570)
- Deployed in https://qa-brain.planx-pla.net with fence version > 3, index authz field set to true. (uc-cdis/portal#570)
- New portal configuration and images for AnVIL (uc-cdis/portal#569)
- Requires a url to be specified in dataExplorerConfig called terraExportURL (uc-cdis/portal#568)
- This should be bvdp-saturn-${env}.appspot.com/#import-data, where ${env} is dev, staging, or prod (uc-cdis/portal#568)
- Needs a version of cloud-automation that exposes arborist's auth/proxy endpoint (uc-cdis/portal#565)
- fixed DAT requires Guppy >= 0.3.1 (uc-cdis/portal#563)
- DAT requires Guppy >= 0.3.0 (uc-cdis/portal#561)
- This change requires sower version >= 0.3.0. (uc-cdis/portal#562)
- This PR requires feat/support is not op for filter guppy#45 (0.2.1+) if want to use Guppy for NIAID (uc-cdis/portal#558)
- require fileExplorerConfig block with buttons (uc-cdis/portal#554)
- New manifest.json variable: logout_inactive_users. Defaults to true if not provided. (uc-cdis/portal#526)
- Feat/logout user opt out cloud-automation#818 (uc-cdis/portal#526)
- enable source map tool so that developers could have better debugging under dev mode (uc-cdis/portal#514)
- Fix warning from fsevents packages during docker image building (uc-cdis/portal#488)
- Add flag to npm ci in dockerfile to support npm prepared files from ui-component library (uc-cdis/portal#488)
- Workspace button in DCF Sandbox (uc-cdis/portal#466)
sheepdog
- Fix Travis coverage reports (combine the results of all 3 tests into one coverage report). (PXP-3408 Update Sheepdog to Python3 sheepdog#314)
- ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) sheepdog#304)
- Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist2 sheepdog#298)
- Requires Arborist to be deployed (Feat/arborist2 sheepdog#298)
- look for INDEX_CLIENT instead of SIGNPOST in app.config (remove signpost from sheepdog deps sheepdog#291)
- get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (remove signpost from sheepdog deps sheepdog#291)
- Sheepdog will now create (or replace) the missing indexes during migration (PXD-2916: implement uniqueKeys sheepdog#259)
- New unique indexes have been added to all nodes PXD-2916: add unique indexes for uniqueKeys gen3datamodel#12 (PXD-2916: implement uniqueKeys sheepdog#259)
- Now uWSGI worker process will kill itself (harakiri) and restart depending on reverse proxy timeout (PXP-2441: use harakiri timeout sheepdog#261)

emalinowski · 2020-04-07T13:10:11Z

nci-crdc.datacommons.io/manifests/fence/fence-config-public.yaml

-    cred: 'fence_bot'
-  tcga-public-dcf-databucket-gen3:
-    cred: 'fence_bot'
-  target-open:


Looks like a lot of these buckets are still being removed? Should keep these buckets here.

PlanXCyborg · 2020-04-07T16:49:46Z

nci-crdc.datacommons.io/manifest.json

Deployment changes

fence
- In fence-config, the region field of S3_BUCKETS is now required: if not specified, Fence will print a warning and try to call GetBucketLocation. Fence will error at startup if that doesn't work. (Fix/make region required for s3 bucket config fence#749)
peregrine
- ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) peregrine#142)
- Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist peregrine#138)
- Requires Arborist to be deployed (Feat/arborist peregrine#138)
- look for INDEX_CLIENT instead of SIGNPOST in app.config (Remove signpost peregrine#135)
- get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (Remove signpost peregrine#135)
- Now uWSGI worker process will kill itself (harakiri) and restart depending on reverse proxy timeout (PXP-2441: graceful timeout in peregrine peregrine#112)
- a new environment variable PUBLIC_DATASETS will be required to set for anonymous access to datasets endpoint (feat(datasets): add datasets endpoint peregrine#111)
- Now the traversal cache is no longer loaded at startup, so it should be faster to start (PXD-2355: lazy load traversal cache peregrine#104)
portal
- New optional title parameter in Terra Export Warning configuration: terraExportWarning: { subjectThreshold: int, title: string(optional), message: string(optional) } (uc-cdis/portal#656)
- Added terraExportWarning: { subjectThreshold: <number>, message: <string, optional> } to portal config. (uc-cdis/portal#642)
- New optional portal config flag: dataExplorerConfig.terraTemplate. The value of this parameter is placed in the Terra export URL as the value of the query parameter 'template'. (uc-cdis/portal#645)
- InCommon login needs fence 4.8.0 (uc-cdis/portal#595)
- Need Guppy 0.3.7+ to handle large file size (uc-cdis/portal#603)
- if deployed with "useArboristUI": true, needs Arborist's GET /auth/mapping endpoint to be exposed via revproxy. (useArboristUI false by default) (uc-cdis/portal#592)
- New optional portal config field called adminPreAppliedFilters. Goes in data/config/*.json. (uc-cdis/portal#576)
- For a "cert" if the required document is a quiz, then hasCorrectAnswers should be true, otherwise false. (uc-cdis/portal#571)
- Windmill lists unmapped files by checking Indexd. Before centralized auth's deployment, a file was determined to be unmapped by checking if the acl field was null. After centralized auth, we use the authz field instead. The new manifest configuration setting use_indexd_authz in the global block specifies which of the two fields should be used. This field defaults to false. (uc-cdis/portal#570)
- Deployed in https://qa-canine.planx-pla.net with fence version < 3, index authz field set to false (uc-cdis/portal#570)
- (by default). (uc-cdis/portal#570)
- Deployed in https://qa-brain.planx-pla.net with fence version > 3, index authz field set to true. (uc-cdis/portal#570)
- New portal configuration and images for AnVIL (uc-cdis/portal#569)
- Requires a url to be specified in dataExplorerConfig called terraExportURL (uc-cdis/portal#568)
- This should be bvdp-saturn-${env}.appspot.com/#import-data, where ${env} is dev, staging, or prod (uc-cdis/portal#568)
- Needs a version of cloud-automation that exposes arborist's auth/proxy endpoint (uc-cdis/portal#565)
- fixed DAT requires Guppy >= 0.3.1 (uc-cdis/portal#563)
- DAT requires Guppy >= 0.3.0 (uc-cdis/portal#561)
- This change requires sower version >= 0.3.0. (uc-cdis/portal#562)
- This PR requires feat/support is not op for filter guppy#45 (0.2.1+) if want to use Guppy for NIAID (uc-cdis/portal#558)
- require fileExplorerConfig block with buttons (uc-cdis/portal#554)
- New manifest.json variable: logout_inactive_users. Defaults to true if not provided. (uc-cdis/portal#526)
- Feat/logout user opt out cloud-automation#818 (uc-cdis/portal#526)
- enable source map tool so that developers could have better debugging under dev mode (uc-cdis/portal#514)
- Fix warning from fsevents packages during docker image building (uc-cdis/portal#488)
- Add flag to npm ci in dockerfile to support npm prepared files from ui-component library (uc-cdis/portal#488)
- Workspace button in DCF Sandbox (uc-cdis/portal#466)
sheepdog
- Fix Travis coverage reports (combine the results of all 3 tests into one coverage report). (PXP-3408 Update Sheepdog to Python3 sheepdog#314)
- ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) sheepdog#304)
- Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist2 sheepdog#298)
- Requires Arborist to be deployed (Feat/arborist2 sheepdog#298)
- look for INDEX_CLIENT instead of SIGNPOST in app.config (remove signpost from sheepdog deps sheepdog#291)
- get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (remove signpost from sheepdog deps sheepdog#291)
- Sheepdog will now create (or replace) the missing indexes during migration (PXD-2916: implement uniqueKeys sheepdog#259)
- New unique indexes have been added to all nodes PXD-2916: add unique indexes for uniqueKeys gen3datamodel#12 (PXD-2916: implement uniqueKeys sheepdog#259)
- Now uWSGI worker process will kill itself (harakiri) and restart depending on reverse proxy timeout (PXP-2441: use harakiri timeout sheepdog#261)

PlanXCyborg · 2020-04-07T18:37:18Z

nci-crdc.datacommons.io/manifest.json

Deployment changes

fence
- In fence-config, the region field of S3_BUCKETS is now required: if not specified, Fence will print a warning and try to call GetBucketLocation. Fence will error at startup if that doesn't work. (Fix/make region required for s3 bucket config fence#749)
peregrine
- ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) peregrine#142)
- Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist peregrine#138)
- Requires Arborist to be deployed (Feat/arborist peregrine#138)
- look for INDEX_CLIENT instead of SIGNPOST in app.config (Remove signpost peregrine#135)
- get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (Remove signpost peregrine#135)
- Now uWSGI worker process will kill itself (harakiri) and restart depending on reverse proxy timeout (PXP-2441: graceful timeout in peregrine peregrine#112)
- a new environment variable PUBLIC_DATASETS will be required to set for anonymous access to datasets endpoint (feat(datasets): add datasets endpoint peregrine#111)
- Now the traversal cache is no longer loaded at startup, so it should be faster to start (PXD-2355: lazy load traversal cache peregrine#104)
portal
- New optional title parameter in Terra Export Warning configuration: terraExportWarning: { subjectThreshold: int, title: string(optional), message: string(optional) } (uc-cdis/portal#656)
- Added terraExportWarning: { subjectThreshold: <number>, message: <string, optional> } to portal config. (uc-cdis/portal#642)
- New optional portal config flag: dataExplorerConfig.terraTemplate. The value of this parameter is placed in the Terra export URL as the value of the query parameter 'template'. (uc-cdis/portal#645)
- InCommon login needs fence 4.8.0 (uc-cdis/portal#595)
- Need Guppy 0.3.7+ to handle large file size (uc-cdis/portal#603)
- if deployed with "useArboristUI": true, needs Arborist's GET /auth/mapping endpoint to be exposed via revproxy. (useArboristUI false by default) (uc-cdis/portal#592)
- New optional portal config field called adminPreAppliedFilters. Goes in data/config/*.json. (uc-cdis/portal#576)
- For a "cert" if the required document is a quiz, then hasCorrectAnswers should be true, otherwise false. (uc-cdis/portal#571)
- Windmill lists unmapped files by checking Indexd. Before centralized auth's deployment, a file was determined to be unmapped by checking if the acl field was null. After centralized auth, we use the authz field instead. The new manifest configuration setting use_indexd_authz in the global block specifies which of the two fields should be used. This field defaults to false. (uc-cdis/portal#570)
- Deployed in https://qa-canine.planx-pla.net with fence version < 3, index authz field set to false (uc-cdis/portal#570)
- (by default). (uc-cdis/portal#570)
- Deployed in https://qa-brain.planx-pla.net with fence version > 3, index authz field set to true. (uc-cdis/portal#570)
- New portal configuration and images for AnVIL (uc-cdis/portal#569)
- Requires a url to be specified in dataExplorerConfig called terraExportURL (uc-cdis/portal#568)
- This should be bvdp-saturn-${env}.appspot.com/#import-data, where ${env} is dev, staging, or prod (uc-cdis/portal#568)
- Needs a version of cloud-automation that exposes arborist's auth/proxy endpoint (uc-cdis/portal#565)
- fixed DAT requires Guppy >= 0.3.1 (uc-cdis/portal#563)
- DAT requires Guppy >= 0.3.0 (uc-cdis/portal#561)
- This change requires sower version >= 0.3.0. (uc-cdis/portal#562)
- This PR requires feat/support is not op for filter guppy#45 (0.2.1+) if want to use Guppy for NIAID (uc-cdis/portal#558)
- require fileExplorerConfig block with buttons (uc-cdis/portal#554)
- New manifest.json variable: logout_inactive_users. Defaults to true if not provided. (uc-cdis/portal#526)
- Feat/logout user opt out cloud-automation#818 (uc-cdis/portal#526)
- enable source map tool so that developers could have better debugging under dev mode (uc-cdis/portal#514)
- Fix warning from fsevents packages during docker image building (uc-cdis/portal#488)
- Add flag to npm ci in dockerfile to support npm prepared files from ui-component library (uc-cdis/portal#488)
- Workspace button in DCF Sandbox (uc-cdis/portal#466)
sheepdog
- Fix Travis coverage reports (combine the results of all 3 tests into one coverage report). (PXP-3408 Update Sheepdog to Python3 sheepdog#314)
- ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) sheepdog#304)
- Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist2 sheepdog#298)
- Requires Arborist to be deployed (Feat/arborist2 sheepdog#298)
- look for INDEX_CLIENT instead of SIGNPOST in app.config (remove signpost from sheepdog deps sheepdog#291)
- get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (remove signpost from sheepdog deps sheepdog#291)
- Sheepdog will now create (or replace) the missing indexes during migration (PXD-2916: implement uniqueKeys sheepdog#259)
- New unique indexes have been added to all nodes PXD-2916: add unique indexes for uniqueKeys gen3datamodel#12 (PXD-2916: implement uniqueKeys sheepdog#259)
- Now uWSGI worker process will kill itself (harakiri) and restart depending on reverse proxy timeout (PXP-2441: use harakiri timeout sheepdog#261)

copying files from nci-crdc-staging.datacommons.io to nci-crdc.dataco…

840eb5b

…mmons.io

themarcelor changed the title ~~JUST A TEST, PLEASE IGNORE: Chore(QA): DCF PROD Release 2020 Q1 1585800777~~ DO NOT MERGE! - JUST A TEST, PLEASE IGNORE: Chore(QA): DCF PROD Release 2020 Q1 1585800777 Apr 2, 2020

themarcelor requested review from emalinowski and itsJiaqi April 2, 2020 17:21

themarcelor changed the title ~~DO NOT MERGE! - JUST A TEST, PLEASE IGNORE: Chore(QA): DCF PROD Release 2020 Q1 1585800777~~ Waiting for review: Chore(QA): DCF PROD Release 2020 Q1 1585800777 Apr 7, 2020

itsJiaqi reviewed Apr 7, 2020

View reviewed changes

Marcelo Costa added 3 commits April 7, 2020 10:41

cannot remove buckets from prod + sower jobs are not prod-ready yet

9f62d5e

restore portal_app:dev config

49bf0ac

restore both fence public config files

4790b10

itsJiaqi reviewed Apr 7, 2020

View reviewed changes

promoting the correct version of the metadata service

13b56d0

re-insert sower block and remove metadata-service

e9c80f6

emalinowski approved these changes Apr 7, 2020

View reviewed changes

remove sower as it is not prod-ready

b2c0619

themarcelor merged commit 9f3541a into master Apr 7, 2020

themarcelor deleted the chore/copy_nci-crdc-staging_datacommons_io_to_nci-crdc_datacommons_io_1585801082 branch April 29, 2020 15:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Waiting for review: Chore(QA): DCF PROD Release 2020 Q1 1585800777 #1330

Waiting for review: Chore(QA): DCF PROD Release 2020 Q1 1585800777 #1330

themarcelor commented Apr 2, 2020 •

edited

Loading

PlanXCyborg commented Apr 2, 2020

itsJiaqi Apr 7, 2020

themarcelor Apr 7, 2020

PlanXCyborg commented Apr 7, 2020

PlanXCyborg commented Apr 7, 2020

itsJiaqi Apr 7, 2020

themarcelor Apr 7, 2020

PlanXCyborg commented Apr 7, 2020

emalinowski Apr 7, 2020

PlanXCyborg commented Apr 7, 2020

PlanXCyborg commented Apr 7, 2020

		@@ -1,137 +1,14 @@
		S3_BUCKETS:
		tcga-protected-dcf-databucket-gen3:

Waiting for review: Chore(QA): DCF PROD Release 2020 Q1 1585800777 #1330

Waiting for review: Chore(QA): DCF PROD Release 2020 Q1 1585800777 #1330

Conversation

themarcelor commented Apr 2, 2020 • edited Loading

PlanXCyborg commented Apr 2, 2020

nci-crdc.datacommons.io/manifest.json

Deployment changes

itsJiaqi Apr 7, 2020

Choose a reason for hiding this comment

themarcelor Apr 7, 2020

Choose a reason for hiding this comment

PlanXCyborg commented Apr 7, 2020

nci-crdc.datacommons.io/manifest.json

Deployment changes

PlanXCyborg commented Apr 7, 2020

nci-crdc.datacommons.io/manifest.json

Deployment changes

itsJiaqi Apr 7, 2020

Choose a reason for hiding this comment

themarcelor Apr 7, 2020

Choose a reason for hiding this comment

PlanXCyborg commented Apr 7, 2020

nci-crdc.datacommons.io/manifest.json

⚠️ Services on branch

Deployment changes

emalinowski Apr 7, 2020

Choose a reason for hiding this comment

PlanXCyborg commented Apr 7, 2020

nci-crdc.datacommons.io/manifest.json

Deployment changes

PlanXCyborg commented Apr 7, 2020

nci-crdc.datacommons.io/manifest.json

Deployment changes

themarcelor commented Apr 2, 2020 •

edited

Loading