chore : Deploy release 2021.09 and audit-service to IBDGC PROD

[atharvar28]

This PR

1. deploying release version 2021.09 to IBD PROD
2. deploying audit service
3. splitting hatchery config from the manifest.json file

[PlanXCyborg]

ibdgc.datacommons.io/manifest.json

Deployment changes

• fence
  • Requires these cloud-automation updates to revproxy: (PXP-7565): Exclude Fence register-user blueprint from CSRF check cloud-automation#1580 ((PXP-7565): User registration, plus CSRF changes fence#906)
• metadata
  • Network rules are updated to allow for elasticsearch to work with metadata-service feat(ecosystems): elasticsearch for MDS cloud-automation#1638 (Feat/aggregation metadata-service aggregate APIs metadata-service#27)
  • USE_AGG_MDS must be set to true to enable aggregate metadata APIs (Feat/aggregation metadata-service aggregate APIs metadata-service#27)
• portal
  • Requires loginForDownload flag set to true in gitops (PXP-8448 - Register users for download data-portal#912)
  • User registration before download feature depends on (PXP-7565): User registration, plus CSRF changes fence#906 (PXP-8448 - Register users for download data-portal#912)
  • Requires the configuration of the batch-export sower job: Batch export sower job sower-jobs#29 in the environment's manifest. This job also has its own setup in cloud automation: Feat/batch export cloud-automation#1708 (Add UI components and polling for zip downloads data-portal#906)
  • 2 new fields in gitops.json: discoveryconfig.features.exportToWorkspace.[enableDownloadZip (bool) downloadZipButtonText (string)] (Add UI components and polling for zip downloads data-portal#906)
  • Requires PXP-8364 Fix/portal csp cloud-automation#1687, otherwsie Indexing page may not work properly (PXP-8364 fix: strict csp data-portal#893)
  • Portal config field discoveryConfig.features.exportToWorkspaceBETA has been renamed to discoveryConfig.features.exportToWorkspace (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • Note that there are several other new configurations introduced for the Discovery page, please refer to this diff (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • New GEN3_BUNDLE option heal introduced for buliding a HEAL bundle (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • To enable DD RUM, must create a portal-datadog-config k8s secret with datadog_application_id and datadog_client_token in it, see DEVOPS-62 feat: add datadog to portal cloud-automation#1651 (DEVOPS-62 Feat/datadog data-portal#881)
  • envs with customized color schemes no longer need to overwrite dictionary button color in their gitops.css ((PXP-6570): refactor/dictionaryBtnColor: change dictionary btn to use defined global colors data-portal#822)
  • New optional config property, tierAccessLevel, in each tab of the explorerConfig block in the portal config. Used to enable index-scoped tiered-access, which is only available starting with Guppy server version 0.11.0. See README for more information. (Feat/index scoped tiered access data-portal#802)
  • To deploy Discovery page: ((PXP-7402) Discovery page data-portal#803)
  • "featureFlags": { "discovery": true } must be set in the portal config. ((PXP-7402) Discovery page data-portal#803)
  • If discoveryConfig.features.authorization.enabled: true, "useArboristUI": true must be set in the portal config. ((PXP-7402) Discovery page data-portal#803)
  • "discoveryConfig": <DiscoveryConfig> must be set in the portal config. DiscoveryConfig documentation can be found at https://github.com/uc-cdis/data-portal/blob/master/docs/portal_config.md ((PXP-7402) Discovery page data-portal#803)
  • To upload data to Discovery page, data must be uploaded to environment's metadata-service instance with guid_type: "discovery_metadata". The metadata must all be under the top-level property gen3_discovery. (No documentation on this process yet, see this gist for right now https://gist.github.com/Avantol13/a53593837503fede2881f315198247c4) ((PXP-7402) Discovery page data-portal#803)
  • Added new entry GWASApp for analysisTools for this new GWAS app for VA. Beaware not to confuse with the old "GWAS PoC" app, which bears the entry vaGWAS (PXP-7296 PXP-7180 Feat/va gwas app data-portal#777)
  • antd modal now by default will use g3-color__base-blue as header background color, this is consistant with our color schemes for other UI components. If an env has customized color scheme, please override .ant-modal-header in gitops.css (PXP-7296 PXP-7180 Feat/va gwas app data-portal#777)
  • The new feature of Portal requires Guppy v0.10.0 ((PXP-6540): Feat/explorer download formats data-portal#780)
  • Added new button types in the Portal config. See portal_config.md for an example. ((PXP-6540): Feat/explorer download formats data-portal#780)
  • At time of writing, qa-midrc is only env using the new feature flag. These changes should be deployed alongside each other to production, but nothing will break if done separately. ((PXP-7097) Add optional alternate TopBar profile menu data-portal#759)
  • New value for GEN3_BUNDLE env var GEN3_BUNDLE=nct (PXP-6895 NIAID CT bundle data-portal#742)
  • New optional portal config variable injectDAPTag (default value is false). If set to true, will inject DAP tag script into page header (this is required for NCT PROD env only) (PXP-6895 NIAID CT bundle data-portal#742)
  • All envs should check if the dataExplorerConfig field in its portal config has ONLY arrangerConfig field but NO guppyConfig field (it is okay to have both). If so, please update dataExplorerConfig to use Guppy (PXP-6409 Remove arranger data-portal#743)
  • A new Export to PFB button can be optionally enabled in the config.json. It is disabled by default. The button requires at least pelican:2020.10 release. (Feat/export pfb to workspace data-portal#681)
  • Add GEN3_BUNDLE=covid19 as portal env var to enable covid bundle (Feat/covid bundle data-portal#737)
  • Limited File PFB Export requires Pelican >= 0.5.0, Tube >= 0.4.1 ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export is enabled by setting fileExplorerConfig.enableLimitedFilePFBExport: { sourceNodeField: "source_node" } and by adding buttons of buttonType export-files (Export to Terra), export-files-to-pfb (Download PFB), or export-files-to-seven-bridges (Export to Seven Bridges) to fileExplorerConfig.buttons. Example: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "fileExplorerConfig": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "buttons": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • .... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enabled": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "type": "export-files-to-pfb", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "title": "Export All to PFB", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "rightIcon": "external-link", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "tooltipText": "You have not selected any cases to export. Please use the checkboxes on the left to select specific cases you would like to export." ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enabled": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "type": "export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "title": "Export All to Terra", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "rightIcon": "external-link", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "tooltipText": "You have not selected any cases to export. Please use the checkboxes on the left to select specific cases you would like to export." ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enableLimitedFilePFBExport": {"sourceNodeField": "source_node"}, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export requires the source_node property to be ETL'd -- source_node should be added to props in the file section of etlMapping.yaml. Example: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: mpingram_file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • doc_type: file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • type: collector ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • root: None ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • category: data_file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • props: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: object_id ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: md5sum ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: source_node ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export requires a new export-files job block to be added to the Sower config in manifest.json. The $ROOT_NODE environment variable must be set to "file", or the name of the Guppy file index if it isn't "file". For BioDataCatalyst, the EXTRA_NODES environment variable must be set to "". (This is for backwards compatibility: pelican-export includes reference_file by default on all BDCat PFB exports unless $EXTRA_NODES is set to an empty string). Example sower config: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "action": "export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "container": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "job-task", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "image": "quay.io/cdis/pelican-export:0.5.0", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "pull_policy": "Always", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "env": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "DICTIONARY_URL", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "valueFrom": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "configMapKeyRef": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "manifest-global", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "key": "dictionary_url" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "GEN3_HOSTNAME", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "valueFrom": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "configMapKeyRef": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "manifest-global", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "key": "hostname" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "ROOT_NODE", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "value": "file" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "EXTRA_NODES", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "value": "" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "volumeMounts": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "readOnly": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "mountPath": "/pelican-creds.json", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "subPath": "config.json" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "peregrine-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "readOnly": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "mountPath": "/peregrine-creds.json", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "subPath": "creds.json" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "cpu-limit": "1", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "memory-limit": "4Gi" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "volumes": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secret": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secretName": "pelicanservice-g3auto" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "peregrine-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secret": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secretName": "peregrine-creds" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "restart_policy": "Never" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Portal should be updated to include this fix if Indexd is above 2.12.0 or 2020.07 and user want to use Arborist UI (fix: data upload auth check data-portal#702)

[PlanXCyborg]

ibdgc.datacommons.io/manifest.json

Deployment changes

• fence
  • Requires these cloud-automation updates to revproxy: (PXP-7565): Exclude Fence register-user blueprint from CSRF check cloud-automation#1580 ((PXP-7565): User registration, plus CSRF changes fence#906)
• metadata
  • Network rules are updated to allow for elasticsearch to work with metadata-service feat(ecosystems): elasticsearch for MDS cloud-automation#1638 (Feat/aggregation metadata-service aggregate APIs metadata-service#27)
  • USE_AGG_MDS must be set to true to enable aggregate metadata APIs (Feat/aggregation metadata-service aggregate APIs metadata-service#27)
• portal
  • Requires loginForDownload flag set to true in gitops (PXP-8448 - Register users for download data-portal#912)
  • User registration before download feature depends on (PXP-7565): User registration, plus CSRF changes fence#906 (PXP-8448 - Register users for download data-portal#912)
  • Requires the configuration of the batch-export sower job: Batch export sower job sower-jobs#29 in the environment's manifest. This job also has its own setup in cloud automation: Feat/batch export cloud-automation#1708 (Add UI components and polling for zip downloads data-portal#906)
  • 2 new fields in gitops.json: discoveryconfig.features.exportToWorkspace.[enableDownloadZip (bool) downloadZipButtonText (string)] (Add UI components and polling for zip downloads data-portal#906)
  • Requires PXP-8364 Fix/portal csp cloud-automation#1687, otherwsie Indexing page may not work properly (PXP-8364 fix: strict csp data-portal#893)
  • Portal config field discoveryConfig.features.exportToWorkspaceBETA has been renamed to discoveryConfig.features.exportToWorkspace (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • Note that there are several other new configurations introduced for the Discovery page, please refer to this diff (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • New GEN3_BUNDLE option heal introduced for buliding a HEAL bundle (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • To enable DD RUM, must create a portal-datadog-config k8s secret with datadog_application_id and datadog_client_token in it, see DEVOPS-62 feat: add datadog to portal cloud-automation#1651 (DEVOPS-62 Feat/datadog data-portal#881)
  • envs with customized color schemes no longer need to overwrite dictionary button color in their gitops.css ((PXP-6570): refactor/dictionaryBtnColor: change dictionary btn to use defined global colors data-portal#822)
  • New optional config property, tierAccessLevel, in each tab of the explorerConfig block in the portal config. Used to enable index-scoped tiered-access, which is only available starting with Guppy server version 0.11.0. See README for more information. (Feat/index scoped tiered access data-portal#802)
  • To deploy Discovery page: ((PXP-7402) Discovery page data-portal#803)
  • "featureFlags": { "discovery": true } must be set in the portal config. ((PXP-7402) Discovery page data-portal#803)
  • If discoveryConfig.features.authorization.enabled: true, "useArboristUI": true must be set in the portal config. ((PXP-7402) Discovery page data-portal#803)
  • "discoveryConfig": <DiscoveryConfig> must be set in the portal config. DiscoveryConfig documentation can be found at https://github.com/uc-cdis/data-portal/blob/master/docs/portal_config.md ((PXP-7402) Discovery page data-portal#803)
  • To upload data to Discovery page, data must be uploaded to environment's metadata-service instance with guid_type: "discovery_metadata". The metadata must all be under the top-level property gen3_discovery. (No documentation on this process yet, see this gist for right now https://gist.github.com/Avantol13/a53593837503fede2881f315198247c4) ((PXP-7402) Discovery page data-portal#803)
  • Added new entry GWASApp for analysisTools for this new GWAS app for VA. Beaware not to confuse with the old "GWAS PoC" app, which bears the entry vaGWAS (PXP-7296 PXP-7180 Feat/va gwas app data-portal#777)
  • antd modal now by default will use g3-color__base-blue as header background color, this is consistant with our color schemes for other UI components. If an env has customized color scheme, please override .ant-modal-header in gitops.css (PXP-7296 PXP-7180 Feat/va gwas app data-portal#777)
  • The new feature of Portal requires Guppy v0.10.0 ((PXP-6540): Feat/explorer download formats data-portal#780)
  • Added new button types in the Portal config. See portal_config.md for an example. ((PXP-6540): Feat/explorer download formats data-portal#780)
  • At time of writing, qa-midrc is only env using the new feature flag. These changes should be deployed alongside each other to production, but nothing will break if done separately. ((PXP-7097) Add optional alternate TopBar profile menu data-portal#759)
  • New value for GEN3_BUNDLE env var GEN3_BUNDLE=nct (PXP-6895 NIAID CT bundle data-portal#742)
  • New optional portal config variable injectDAPTag (default value is false). If set to true, will inject DAP tag script into page header (this is required for NCT PROD env only) (PXP-6895 NIAID CT bundle data-portal#742)
  • All envs should check if the dataExplorerConfig field in its portal config has ONLY arrangerConfig field but NO guppyConfig field (it is okay to have both). If so, please update dataExplorerConfig to use Guppy (PXP-6409 Remove arranger data-portal#743)
  • A new Export to PFB button can be optionally enabled in the config.json. It is disabled by default. The button requires at least pelican:2020.10 release. (Feat/export pfb to workspace data-portal#681)
  • Add GEN3_BUNDLE=covid19 as portal env var to enable covid bundle (Feat/covid bundle data-portal#737)
  • Limited File PFB Export requires Pelican >= 0.5.0, Tube >= 0.4.1 ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export is enabled by setting fileExplorerConfig.enableLimitedFilePFBExport: { sourceNodeField: "source_node" } and by adding buttons of buttonType export-files (Export to Terra), export-files-to-pfb (Download PFB), or export-files-to-seven-bridges (Export to Seven Bridges) to fileExplorerConfig.buttons. Example: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "fileExplorerConfig": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "buttons": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • .... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enabled": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "type": "export-files-to-pfb", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "title": "Export All to PFB", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "rightIcon": "external-link", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "tooltipText": "You have not selected any cases to export. Please use the checkboxes on the left to select specific cases you would like to export." ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enabled": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "type": "export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "title": "Export All to Terra", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "rightIcon": "external-link", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "tooltipText": "You have not selected any cases to export. Please use the checkboxes on the left to select specific cases you would like to export." ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enableLimitedFilePFBExport": {"sourceNodeField": "source_node"}, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export requires the source_node property to be ETL'd -- source_node should be added to props in the file section of etlMapping.yaml. Example: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: mpingram_file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • doc_type: file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • type: collector ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • root: None ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • category: data_file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • props: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: object_id ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: md5sum ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: source_node ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export requires a new export-files job block to be added to the Sower config in manifest.json. The $ROOT_NODE environment variable must be set to "file", or the name of the Guppy file index if it isn't "file". For BioDataCatalyst, the EXTRA_NODES environment variable must be set to "". (This is for backwards compatibility: pelican-export includes reference_file by default on all BDCat PFB exports unless $EXTRA_NODES is set to an empty string). Example sower config: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "action": "export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "container": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "job-task", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "image": "quay.io/cdis/pelican-export:0.5.0", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "pull_policy": "Always", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "env": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "DICTIONARY_URL", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "valueFrom": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "configMapKeyRef": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "manifest-global", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "key": "dictionary_url" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "GEN3_HOSTNAME", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "valueFrom": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "configMapKeyRef": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "manifest-global", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "key": "hostname" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "ROOT_NODE", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "value": "file" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "EXTRA_NODES", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "value": "" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "volumeMounts": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "readOnly": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "mountPath": "/pelican-creds.json", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "subPath": "config.json" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "peregrine-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "readOnly": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "mountPath": "/peregrine-creds.json", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "subPath": "creds.json" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "cpu-limit": "1", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "memory-limit": "4Gi" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "volumes": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secret": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secretName": "pelicanservice-g3auto" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "peregrine-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secret": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secretName": "peregrine-creds" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "restart_policy": "Never" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Portal should be updated to include this fix if Indexd is above 2.12.0 or 2020.07 and user want to use Arborist UI (fix: data upload auth check data-portal#702)

[paulineribeyre]

Portal, pidgin and peregrine versions are not correct.
See https://cdis.slack.com/archives/CARERJJDV/p1629219673046500?thread_ts=1629203212.043800&cid=CARERJJDV
The patches are not included in 2021.09.
Monthly release deployments should NOT downgrade service versions!

[PlanXCyborg]

ibdgc.datacommons.io/manifest.json

Deployment changes

• fence
  • Requires these cloud-automation updates to revproxy: (PXP-7565): Exclude Fence register-user blueprint from CSRF check cloud-automation#1580 ((PXP-7565): User registration, plus CSRF changes fence#906)
• metadata
  • Network rules are updated to allow for elasticsearch to work with metadata-service feat(ecosystems): elasticsearch for MDS cloud-automation#1638 (Feat/aggregation metadata-service aggregate APIs metadata-service#27)
  • USE_AGG_MDS must be set to true to enable aggregate metadata APIs (Feat/aggregation metadata-service aggregate APIs metadata-service#27)

[paulineribeyre]

thanks