Release/0.2.0 #49

frankinspace · 2025-01-21T22:18:15Z

Release Notes

Two configuration options have changed names. Any existing configuration files will need to be updated to reflect these changes.
- operaTreatment changed to operaHLSTreatment
- convertToPNG changed to sendToHarmony.
Input variable names changed:
- pobit_audit_bucket -> bignbit_audit_bucket
- pobit_audit_path -> bignbit_audit_path
Default values for variables config_dir and bignbit_audit_path have changed as shown in the table below. This change can be ignored by explicitly setting the module input variables to the old default values.

Module Input Variable Name	Old default	New Default
config_dir	dataset-config	big-config
bignbit_audit_path	pobit-cma-output	bignbit-cnm-output

There is a new variable bignbit_staging_bucket for the module. If not set (recommended), a new s3 bucket will be created for storing harmony results via https://harmony.earthdata.nasa.gov/docs#user-owned-buckets-for-harmony-output. Alternatively, to use an s3 bucket that already exists, set the variable bignbit_staging_bucket to the name of the bucket to be used. Be sure the bucket policy permissions are set correctly to allow Harmony write and GIBS read permission to this bucket.
lambda_role variable has been removed and instead replaced by permissions_boundary_arn, this will need to be updated in the cumulus deployment terraform for the bignbit module.
The prevent_destroy flag for gibs_response_topic has been turned on to help avoid accidental deletion of the topic used for responses from GIBS
Terraform v1.5.3 should be used for deployment

Test Results

Summary

Two collections were tested in the PO.DAAC Services UAT account:

OPERA_L3_DSWX-HLS_V1
OPERA_L3_DSWX-S1_V1

One granule from each collection was used as input to an execution of the bignbit module. The OPERA_L3_DSWX-HLS_V1 collection tests the operaHLSTreatment branch of the workflow while the OPERA_L3_DSWX-S1_V1 collection tests the sendToHarmony branch of the workflow.
Both granules succeeding in generating images and sending CNM messages to GIBS
One CNM-R message with SUCCESS status was received from GIBS for each image generated
GIBS confirmed processing of images was successful in the GIBS UAT environment

OPERA_L3_DSWX-HLS_V1

Input data

Granule used for test:
OPERA_L3_DSWx-HLS_T43RDH_20241122T054141Z_20241206T215709Z_S2A_30_v1.0
https://cmr.uat.earthdata.nasa.gov/search/concepts/G1272632743-POCLOUD.umm_json

CMA message used to trigger the workflow execution:
https://github.com/podaac/bignbit/blob/17ca38cd96e5a8dad994614e2e654aed15475bcb/tests/sample_messages/cma.uat.workflow-input.OPERA_L3_DSWx-HLS_T55MDS_20250124T001834Z_20250128T131415Z_L8_30_v1.0.json

Results

Execution completed successfully:

Note: because of the split to MGRS grid, this granule was split into 4 images before being sent to GIBS. For this collection, one granule can result in 10's of images; this particular granule required 4. Each image is sent to GIBS in a distinct CNM message which is why there was 4 iterations of the TransferImageSets group.

CNM and CNM-R with GIBS:

OPERA_L3_DSWX-HLS_V1_none.zip

Result in GIBS UAT Worldview:

https://worldview.gitc.uat.earthdatacloud.nasa.gov/?v=65.38591321601064,20.05699438786398,83.26940179774141,27.83531470036398&l=HLS_MGRS_Granule_Grid,OPERA_L3_Dynamic_Surface_Water_Extent-HLS_v1_STD(disabled=5),Coastlines,BlueMarble_NextGeneration(hidden)&lg=true&t=2024-11-22-T12%3A39%3A37Z

OPERA_L3_DSWX-S1_V1

Input data

Granule used for test:
OPERA_L3_DSWx-S1_T45QYD_20241001T121219Z_20241206T065726Z_S1A_30_v1.0
https://cmr.uat.earthdata.nasa.gov/search/concepts/G1272632541-POCLOUD.umm_json

CMA message used to trigger the workflow execution:
https://github.com/podaac/bignbit/blob/17ca38cd96e5a8dad994614e2e654aed15475bcb/tests/sample_messages/cma.uat.workflow-input.OPERA_L3_DSWx-S1_T45QYD_20241001T121219Z_20241206T065726Z_S1A_30_v1.0.json

Results

Execution completed successfully:

Harmony Job completed successfully:

https://harmony.uat.earthdata.nasa.gov/jobs/d5ac5a80-0ebd-4741-b038-c132c37d517a

CNM and CNM-R with GIBS:

OPERA_L3_DSWX-S1_V1_none.zip

Result in GIBS UAT Worldview:

https://worldview.gitc.uat.earthdatacloud.nasa.gov/?v=82.20013623040202,16.854485801309316,97.10638623040202,24.986310211090093&l=HLS_MGRS_Granule_Grid,OPERA_L3_Dynamic_Surface_Water_Extent-Sentinel-1_v1_STD(disabled=5),Coastlines,BlueMarble_NextGeneration(hidden)&lg=true&t=2024-10-01-T12%3A39%3A37Z

Changelog

[0.2.0]

Added

issues/40: New message attribute response_topic_arn will be added to every message sent to GIBS
issues/9: Added some documentation for installing as cumulus module

Changed

issues/15: Change 'convertToPNG' choice to a generic send to harmony choice
issues/16: Change apply opera treatment choice and lambda to be specific to HLS
issues/23: Harmony requests now include destinationUrl parameter to place output directly in s3 bucket instead of requiring data to be copied.
issues/41: Module no longer depends on deprecated hashicorp/template provider
issues/42: Terraform version upgraded to v1.5.3
Default values for config_dir and bignbit_audit_path have changed to big-config and bignbit-cnm-output respectively

Deprecated

Removed

issues/7: Remove the wait for GITC response
issues/23: Removed lambda_role module variable. The lambda role is now created as part of the module, permissions_boundary_arn is required instead.

Fixed

issues/36: Support datetimes without microseconds

Security

* update state_machine, sendToHarmony * --changelog Issue-15 * moved to unreleased, changelog

…ic to HLS (#24) * update for HLS -- lambda and state machine * py lint error * update file name

# Conflicts: # pyproject.toml

* fix type in readme * change task token to uuid in send to gitc * remove wait for task token & gitc response handler from state machine * change gitc response handler to invoke save cma message * fix typo * fix context parameter * convert uuid to string * fix json typo * remove uuid lib to use stdlib uuid * change identifier to image set name instead of uuid * remove uuid import * fix json formatting of cma invoke * add granule concept ID to identifier * update image set name with granule conceptid * remove whitespace * move get umm json to utils and save gitc outgoing cnm * remove unused import * try setting cmr query env based on stage * remove task token from tests * fix granule index * change how image set name defined * add region to ssm client in utils * add ssm parameters to gitc lambdas * change region reference * update how granule name referenced in send to gitc * remove unused cmr var * fix case * fix cnm parsing * reformat save cnm to separate step * update tf vars for save cnm * fix type in module definition * fix cnm in cma * fix collection reference in save cnm * add debugging log statements * fix input * update save cnm input * debugging * debugging * linting * debugging * debugging * linting * fix state machine * update state machine * reorg state machine * missing comma * move save cnm into map * fix state machine transitions * fix sm * fix boolean * debugging cnm input * fix input * change input * debug input * add cnm as config parameter * lint * fix cnm payload * remove debugging statements * add prefix to cnm path and fix gitc response * increase handle gitc response timeout * update changelog * add original shortname as cnm prefix * revert shortname and change collection ref to save cnm & cnm-r in same location * remove save cma lambda no longer used * remove EDL env params from sendtogitc * change parsing of granule concept id

* Support datetimes without microseconds * changelog

add docstring

# Conflicts: # CHANGELOG.md # pyproject.toml # terraform/lambda_functions.tf

* add s3 destination to harmony job * syntax * job success * harmony block update * --variable * temp test in uat * manual to uat * test s1-v1 * lint * deploy --sit * Update to place harmony results in our bucket * Update to place harmony results in our bucket * Update to place harmony results in our bucket * Update to place harmony results in our bucket * Update to place harmony results in our bucket * Update to place harmony results in our bucket --------- Co-authored-by: Josh <[email protected]>

… to GIBS (#47)

torimcd

Looks good, pending successful UAT tests

…ng, fix type error

frankinspace and others added 30 commits April 23, 2024 17:58

/version 0.2.0a0

dcf15a0

feature: convertToPNG --> sendToHarmony (#21)

341c23b

* update state_machine, sendToHarmony * --changelog Issue-15 * moved to unreleased, changelog

/version 0.2.0a1

9522d7f

Update README.md (#22)

18d494c

/version 0.2.0a2

2ceb142

Use -j to remove top-level directory in zip artifact (#25)

f62ab5b

/version 0.2.0a3

258e752

Issue#16: Change apply opera treatment choice and lambda to be specif…

7bda932

…ic to HLS (#24) * update for HLS -- lambda and state machine * py lint error * update file name

/version 0.2.0a4

bf574e5

Merge remote-tracking branch 'origin/main' into develop

c4d54bb

# Conflicts: # pyproject.toml

/version 0.2.0a5

d47a467

/version 0.2.0a6

b7fa282

Update README.md

33034e8

/version 0.2.0a7

752c2a3

Support datetimes without microseconds (#37)

9c97fbc

* Support datetimes without microseconds * changelog

Update generate_image_metadata.py (#39)

8b463ea

add docstring

/version 0.2.0a8

195fe06

Merge branch 'main' into develop

26cb2b8

# Conflicts: # CHANGELOG.md # pyproject.toml # terraform/lambda_functions.tf

/version 0.2.0a9

c36a137

/version 0.2.0a10

3b230bb

fix deploy failure

8c4a7e4

/version 0.2.0a11

34d6019

Add new message attribute response_topic_arn to all outgoing messages…

37f6d08

… to GIBS (#47)

/version 0.2.0a12

dfb5dd3

prep for 0.2.0

802a271

/version 0.2.0a13

2b705af

Fix build error

b916a00

/version 0.2.0a14

b6de839

frankinspace added this to the Add support for OPERA_L3_DSWX-S1_V1 milestone Jan 21, 2025

frankinspace requested a review from torimcd January 21, 2025 22:18

frankinspace and others added 8 commits January 21, 2025 16:36

Update graph

5399a01

/version 0.2.0rc1

533785c

Turn prevent_destroy back on for gibs response topic

5225446

/version 0.2.0rc2

902a8b9

Improve error message when missing config

b3d8079

/version 0.2.0rc3

e2845cd

Improve error message when missing config

85bf782

/version 0.2.0rc4

df3df5a

torimcd approved these changes Jan 22, 2025

View reviewed changes

frankinspace and others added 17 commits January 22, 2025 15:39

UAT is in east-1 region

436cde9

/version 0.2.0rc5

4f90d0e

fix gibs queue name

fe8ac43

/version 0.2.0rc6

807841f

calculate md5sum for files from harmony

5b7103a

calculate md5sum for files from harmony

38dadb7

/version 0.2.0rc7

090e48d

bump chunk size to 100 MB

fd97537

/version 0.2.0rc8

e18f32a

Update test message for UAT, add checksum to generated tif, add loggi…

c401739

…ng, fix type error

Lint

063f38a

Lint

ed2c4b0

/version 0.2.0rc9

17ca38c

Change named variables for audit path from pobit to bignbit

dc209f2

/version 0.2.0rc10

400855e

Update docs

76f0cbe

/version 0.2.0rc11

b2d163c

frankinspace merged commit 43d56cb into main Feb 4, 2025
1 check passed

frankinspace deleted the release/0.2.0 branch February 4, 2025 14:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release/0.2.0 #49

Release/0.2.0 #49

frankinspace commented Jan 21, 2025 •

edited

Loading

torimcd left a comment

Release/0.2.0 #49

Release/0.2.0 #49

Conversation

frankinspace commented Jan 21, 2025 • edited Loading

Release Notes

Test Results

Summary

OPERA_L3_DSWX-HLS_V1

Input data

Results

OPERA_L3_DSWX-S1_V1

Input data

Results

Changelog

[0.2.0]

Added

Changed

Deprecated

Removed

Fixed

Security

torimcd left a comment

Choose a reason for hiding this comment

frankinspace commented Jan 21, 2025 •

edited

Loading