Update filter metadata compoment in HEAL CEDAR ingestion script (#2093)

* update cedar ingestion script for filter component

* update comment

* avoid retry

* fix

* update

* - adding networkpolicies for argocd (#2100)

- annotating the argocd namespace for network policy use
- fixing the indentation in the argocd nginx conf file

* feat(waf-enabled): Enable waf through a flag in the manifest (#1973)

* feat(waf-enabled): Enable waf through a flag in the manifest

* Added the ingress_setup_waf function. Tested and working in my dev commons to be implemented everywhere once the waf rules are tested.

* removing the "AWSManagedRulesAnonymousIpList" rule

Co-authored-by: Edward Malinowski <[email protected]>
Co-authored-by: EliseCastle23 <[email protected]>
Co-authored-by: EliseCastle23 <[email protected]>
Co-authored-by: cmlsn <[email protected]>

* the gitops-sa account was not able to access the argo namespace because the default namespace was not provided for the gitops-sa (#2104)

* chore(revproxy): update nginx config to increase max request body size (manifestservice has no limit and we're running into issues with it being at 1m default) (#2052)

Co-authored-by: Alexander VT <[email protected]>

* chore(fail-fast): Update gen3 reset to exit on error (#2088)

* chore(fail-fast): Update gen3 reset to exit on error

* chore(fail-fast): Update gen3 reset to exit on error

Co-authored-by: Edward Malinowski <[email protected]>
Co-authored-by: J. Q <[email protected]>

* fix: upgrade lit-html from 1.3.0 to 1.4.1 (#2106)

Snyk has created this PR to upgrade lit-html from 1.3.0 to 1.4.1.

See this package in npm:
https://www.npmjs.com/package/lit-html

See this project in Snyk:
https://app.snyk.io/org/plan-x/project/2334b58a-d787-40b8-8f93-2aa7a812c519?utm_source=github&utm_medium=referral&page=upgrade-pr

Co-authored-by: snyk-bot <[email protected]>

* fix: upgrade jasmine-core from 3.6.0 to 3.99.1 (#2105)

Snyk has created this PR to upgrade jasmine-core from 3.6.0 to 3.99.1.

See this package in npm:
https://www.npmjs.com/package/jasmine-core

See this project in Snyk:
https://app.snyk.io/org/plan-x/project/2334b58a-d787-40b8-8f93-2aa7a812c519?utm_source=github&utm_medium=referral&page=upgrade-pr

Co-authored-by: snyk-bot <[email protected]>
Co-authored-by: J. Q <[email protected]>

* Revert "chore(fail-fast): Update gen3 reset to exit on error (#2088)" (#2108)

This reverts commit 45e9d1f.

* changing from an annotation to a label (#2109)

* exit on failure but no retry

Co-authored-by: EliseCastle23 <[email protected]>
Co-authored-by: emalinowski <[email protected]>
Co-authored-by: Edward Malinowski <[email protected]>
Co-authored-by: EliseCastle23 <[email protected]>
Co-authored-by: cmlsn <[email protected]>
Co-authored-by: Alexander VanTol <[email protected]>
Co-authored-by: Alexander VT <[email protected]>
Co-authored-by: J. Q <[email protected]>
Co-authored-by: Michael Lukowski <[email protected]>
Co-authored-by: snyk-bot <[email protected]>

.gitignore

@@ -14,6 +14,7 @@ terraform
 *~
 *.swp
 .DS_Store
+.dccache
 kube/services/fluentd/varlogs/
 kube/services/fluentd/dockerlogs/
 

files/scripts/healdata/heal-cedar-data-ingest.py

@@ -1,9 +1,57 @@
 import argparse
-import json
+import sys
 import requests
 import pydash
-import os
 
+# Defines how a field in metadata is going to be mapped into a key in filters
+FILTER_FIELD_MAPPINGS = {
+    "Study Type.study_stage": "Study Type",
+    "Data.data_type": "Data Type",
+    "Study Type.study_subject_type": "Subject Type",
+    "Human Subject Applicability.gender_applicability": "Gender",
+    "Human Subject Applicability.age_applicability": "Age"
+}
+
+# Defines how to handle special cases for values in filters
+SPECIAL_VALUE_MAPPINGS = {
+    "Interview/Focus Group - structured": "Interview/Focus Group",
+    "Interview/Focus Group - semi-structured": "Interview/Focus Group",
+    "Interview/Focus Group - unstructured": "Interview/Focus Group",
+    "Questionnaire/Survey/Assessment - validated instrument": "Questionnaire/Survey/Assessment",
+    "Questionnaire/Survey/Assessment - unvalidated instrument": "Questionnaire/Survey/Assessment",
+    "Cis Male": "Male",
+    "Cis Female": "Female",
+    "Trans Male": "Female-to-male transsexual",
+    "Trans Female": "Male-to-female transsexual",
+    "Agender, Non-binary, gender non-conforming": "Other",
+    "Gender Queer": "Other",
+    "Intersex": "Intersexed"
+}
+
+# Defines field that we don't want to include in the filters
+OMITTED_VALUES_MAPPING = {
+    "Human Subject Applicability.gender_applicability": "Not applicable"
+}
+
+def update_filter_metadata(metadata_to_update):
+    filter_metadata = []
+    for metadata_field_key, filter_field_key in FILTER_FIELD_MAPPINGS.items():
+        filter_field_values = pydash.get(metadata_to_update, metadata_field_key)
+        if filter_field_values:
+            if isinstance(filter_field_values, str):
+                filter_field_values = [filter_field_values]
+            if not isinstance(filter_field_values, list):
+                print(filter_field_values)
+                raise TypeError("Neither a string nor a list")
+            for filter_field_value in filter_field_values:
+                if (metadata_field_key, filter_field_value) in OMITTED_VALUES_MAPPING.items():
+                    continue
+                if filter_field_value in SPECIAL_VALUE_MAPPINGS:
+                    filter_field_value = SPECIAL_VALUE_MAPPINGS[filter_field_value]
+                filter_metadata.append({"key": filter_field_key, "value": filter_field_value})
+    filter_metadata = pydash.uniq(filter_metadata)
+    metadata_to_update["advSearchFilters"] = filter_metadata
+    return metadata_to_update
 
 parser = argparse.ArgumentParser()
 
@@ -16,13 +64,13 @@
 
 if not args.directory:
     print("Directory ID is required!")
-    exit(1)
+    sys.exit(1)
 if not args.access_token:
     print("User access token is required!")
-    exit(1)
+    sys.exit(1)
 if not args.hostname:
     print("Hostname is required!")
-    exit(1)
+    sys.exit(1)
 
 dir_id = args.directory
 access_token = args.access_token
@@ -39,7 +87,7 @@
     metadata_return = cedar.json()
     if "metadata" not in metadata_return:
         print("Got 200 from CEDAR wrapper but no metadata in body, something is not right!")
-        exit(1)
+        sys.exit(1)
 
     print(f"Successfully got {len(metadata_return['metadata'])} record(s) from CEDAR directory")
     for cedar_record in metadata_return["metadata"]:
@@ -60,7 +108,9 @@
                 print("Metadata is already registered. Updating MDS record")
             elif mds_res["_guid_type"] == "unregistered_discovery_metadata":
                 print("Metadata is has not been registered. Registering it in MDS record")
+                continue
             pydash.merge(mds_discovery_data_body, mds_res["gen3_discovery"], cedar_record)
+            mds_discovery_data_body = update_filter_metadata(mds_discovery_data_body)
             mds_cedar_register_data_body["gen3_discovery"] = mds_discovery_data_body
             mds_cedar_register_data_body["_guid_type"] = "discovery_metadata"
 

kube/services/jobs/cedar-ingestion-job.yaml

@@ -19,6 +19,7 @@ kind: Job
 metadata:
   name: cedar-ingestion
 spec:
+  backoffLimit: 0
   template:
     metadata:
       labels:
@@ -106,6 +107,7 @@ spec:
             export ACCESS_TOKEN="$(cat /mnt/shared/access_token.txt)"
             python ${GEN3_HOME}/files/scripts/healdata/heal-cedar-data-ingest.py --access_token $ACCESS_TOKEN --directory $CEDAR_DIRECTORY_ID --hostname $HOSTNAME
             echo "All done - exit status $?"
+        restartPolicy: Never
       - name: fence
         GEN3_FENCE_IMAGE
         imagePullPolicy: Always