Feature: GeoMetaMaker integration #1753

davemfish · 2025-01-28T21:57:56Z

The invest run CLI now uses geometamaker to generate metadata documents for all files listed in a model's OUTPUT_SPEC.

The Workbench includes a new feature to allow users to configure geometamaker with a "user profile".

There's a corresponding User's Guide PR which we could merge first: natcap/invest.users-guide#157

Fixes #1662

Checklist

Updated HISTORY.rst and link to any relevant issue (if these changes are user-facing)
Updated the user's guide (if needed)
Tested the Workbench UI (if relevant)

…662-metadata

davemfish · 2025-01-31T20:36:50Z

tests/test_model_specs.py

@@ -505,29 +504,5 @@ def test_model_specs_serialize(self):
                    f'{error}')


-class SpecUtilsTests(unittest.TestCase):


I moved this set of tests to test_spec_utils

claire-simpson · 2025-02-07T22:13:27Z

workbench/src/renderer/server_requests.js

+ *   }
+ *   license: {
+ *     title: string
+ *     url: string


Looks like in other places url is actually called path

Yeah, not ideal. geometamaker is following the DataPackage schema, which uses path. But url is more meaningful so that's why the Workbench components are using it.

claire-simpson

This looks fantastic! I was also able to try this out in the workbench and CLI and everything worked as expected (including that the metadata was recreated each time the model was run even with the same inputs/params, which is great).

claire-simpson · 2025-02-07T22:19:56Z

workbench/src/renderer/components/SettingsModal/MetadataForm/index.jsx

+        in the output workspace after running a model.
+      </p>
+      <p>
+        Open a YAML file in a text editor to read the metadata. It includes


Would it be worth mentioning in this info panel that users can also edit the metadata in a text editor (and give an example of why they might wish to do so)?

Good idea, for now I only changed the sentence to
Open a YAML file in a text editor to read the metadata and even add to it.

Maybe we'll get more feedback at some point and possibly have some other documentation to point to. For now I don't want to distract too much from just giving people the context they need to fill out the form in the Workbench.

davemfish · 2025-02-11T15:19:25Z

@claire-simpson and @emilyanndavis , I added a small addition to write keywords into the metadata docs. These changes are in d97435e and 6901e34. Sorry that wasn't included from the beginning!

emilyanndavis

Very cool, Dave! Please don't be alarmed by the number of comments. Pretty minor stuff for the most part!

emilyanndavis · 2025-02-12T21:34:25Z

workbench/src/renderer/components/Changelog/index.jsx

@@ -5,6 +5,7 @@ import Modal from 'react-bootstrap/Modal';
 import { MdClose } from 'react-icons/md';
 import { useTranslation } from 'react-i18next';

+import { openLinkInBrowser } from '../../utils';


Good call moving this to utils; thanks!

emilyanndavis · 2025-02-12T21:56:57Z

workbench/src/renderer/components/SettingsModal/MetadataForm/index.jsx

+        the bands in a raster, and other useful information.
+      </p>
+      <p>
+        Some properties of the metadata are configureable here. You may


configureable => configurable

emilyanndavis · 2025-02-12T22:07:12Z

workbench/src/renderer/components/SettingsModal/MetadataForm/index.jsx

+        onClick={openLinkInBrowser}
+      >
+        https://github.com/natcap/geometamaker
+      </a>


Can we give this link user-friendly link text instead of using the raw URL? Maybe something like this?

<p> InVEST uses <a href="https://github.com/natcap/geometamaker" onClick={openLinkInBrowser} >GeoMetaMaker</a> to generate metadata. </p>

... or like this?

<p> InVEST uses GeoMetaMaker to generate metadata. Learn more about <a href="https://github.com/natcap/geometamaker" onClick={openLinkInBrowser} >GeoMetaMaker on Github</a>. </p>

I like the second one. One thing I like about the bare URL is that it lets you know where you're headed. But this second option is a nice improvement on that.

emilyanndavis · 2025-02-12T22:09:32Z

workbench/src/renderer/components/SettingsModal/MetadataForm/index.jsx

+
+function AboutMetadataDiv() {
+  return (
+    <div>


I'm assuming we'll eventually have translations of this copy into other languages. Is it worth adding support for that now?

Yes, thanks for the suggestion.

emilyanndavis · 2025-02-12T22:22:37Z

workbench/src/renderer/components/SettingsModal/MetadataForm/index.jsx

+          ? <AboutMetadataDiv />
+          : (
+            <Form onSubmit={handleSubmit}>
+              <fieldset>


Awesome use of fieldsets & legends! 🌟

I learned something new!

emilyanndavis · 2025-02-12T22:44:49Z

workbench/src/renderer/styles/style.css

+}
+
+#metadata-form fieldset {
+}


Looks like an empty ruleset that can be removed.

emilyanndavis · 2025-02-12T22:56:30Z

src/natcap/invest/spec_utils.py

+    resource.set_lineage(lineage_statement)
+    # a pre-existing metadata doc could have keywords
+    words = resource.get_keywords()
+    resource.set_keywords(words + keywords_list)


Is there any chance any of the words could also be in keywords_list? Would it be worth checking for duplicates?

After testing this locally, and looking at the code again, I can confirm the answer to the first question is yes! And I would recommend checking for and removing duplicates. I ran the carbon model a few times, and I ended up with the keywords list carbon, InVEST, carbon, InVEST, carbon, InVEST. That could add up quickly. 🙃

Thanks for catching this!

emilyanndavis · 2025-02-12T23:02:02Z

tests/test_spec_utils.py

+            spec_utils.format_unit({})
+
+
+class TestDescribeArgFomSpec(unittest.TestCase):


Should this be TestDescribeArgFromSpec?

emilyanndavis · 2025-02-12T23:41:58Z

workbench/src/renderer/components/SettingsModal/index.jsx

+                eventKey="0"
+                className="mr-2 w-50"
+              >
+                <span><u>{t('Configure Metadata')}</u></span>


I was surprised to see this text underlined: it had the effect of making me think it was a link embedded in a button. Would you be OK with removing the underline?

I also have a couple other ideas to help make it look more obviously like an accordion; let me know if you want my help, either now or in a follow-up PR!

We talked about this on Slack. I implemented some of your ideas.

emilyanndavis · 2025-02-12T23:52:09Z

workbench/src/renderer/components/SettingsModal/MetadataForm/index.jsx

+      if (profile.license) {
+        setLicenseTitle(profile.license.title);
+        setLicenseURL(profile.license.path);
+      }


I tried this out locally and was seeing a Cannot read properties of undefined (reading 'contact') error. The root cause was that I'd forgotten to install invest, so the get profile endpoint was returning a 404. Once I installed invest, the errors went away, so in theory this shouldn't happen IRL, but just in case, it wouldn't hurt to add a null check on profile to these if statements, i.e., if (profile && profile.contact) and if (profile && profile.license).

…ime. natcap#1662

emilyanndavis

Awesome! Thanks, Dave!

davemfish added 18 commits August 13, 2024 11:24

add utils function for metadata creation; decorated carbon execute

78e3262

moved to a plain function rather than a decorator

edfda1a

merged upstream main and resolved trivial conflict

3683455

docstrings for the new utils functions. natcap#1662

f11c624

Merge remote-tracking branch 'upstream/main' into feature/geometamaker

c0735eb

move the utility functions to spec_utils.

4a969fe

demote an expected KeyError message to debug-level

7aa56b9

merge feature branch and resolve conflict. natcap#1662

feb5eae

added a form component for configuring geometamaker. natcap#1662

05da34f

add flask app endpoints for geometamaker configuration. natcap#1662

5bbbcc4

setup requests to geometamaker endpoints. natcap#1662

e802bd5

making the form functional; some styling. natcap#1662

865b2da

add an alert message after form submission. natcap#1662

8acd21a

added tests for new flask app endpoints. natcap#1662

24ca23e

added renderer unit and integration tests. natcap#1662

5052e9f

add geometamaker as a requirement. natcap#1662

0ffa0e7

added tests for metadata creation functions. natcap#1662

fc99a22

Merge remote-tracking branch 'upstream/release/3.15.0' into feature/1…

fd96e5b

…662-metadata

davemfish commented Jan 31, 2025

View reviewed changes

davemfish added 11 commits January 31, 2025 15:55

some cleanup and docstrings. natcap#1662

a77f431

wrap a handler in useCallback to memoize it. natcap#1662

6eb578e

a note for HISTORY. natcap#1662

24bbac9

include frictionless assets in pyinstaller builds. natcap#1662

ed0a2d0

renamed a function. natcap#1662

28c4b7a

revert the useCallback optimization - not worth the cost. natcap#1662

4648fda

install an experimental bugfix version of geometamaker. natcap#1662

19d684c

add an informational component to the metadata form. natcap#1662

5cda023

testing the information component. natcap#1662

e7e8179

add geometamaker to Workbench About page. natcap#1662

c5f2ee0

revert change to geometamaker requirement. natcap#1662

113300a

fix pyinstaller script for macos. natcap#1662

d8a1013

davemfish marked this pull request as ready for review February 6, 2025 20:45

davemfish requested review from emilyanndavis and claire-simpson February 6, 2025 20:45

claire-simpson reviewed Feb 7, 2025

View reviewed changes

claire-simpson approved these changes Feb 10, 2025

View reviewed changes

davemfish added 5 commits February 11, 2025 09:43

add model_id to each MODEL_SPEC. natcap#1662

d97435e

add model keywords to metadata documents. natcap#1662

6901e34

fix bug in dict access. natcap#1662

05ae7bc

edited the workbench help text. natcap#1662

7cbec47

merge upstream release branch and resolve history conflict. natcap#1662

5159423

davemfish added 2 commits February 11, 2025 10:22

update test for the new model_id key in MODEL_SPECs. natcap#1662

29d72f2

updated another test for new model_id key. natcap#1662

8429da1

emilyanndavis requested changes Feb 13, 2025

View reviewed changes

davemfish added 11 commits February 18, 2025 09:13

cleanup modal text and allow translation. natcap#1662

23c36fb

move styles out to stylesheet. natcap#1662

7579adc

avoid adding keywords that already exist. natcap#1662

c65804f

check for an undefined variable. natcap#1662

e7c5e1c

fix assertion of same keywords. natcap#1662

43f2441

fix syntax error in css. natcap#1662

e8e7f72

clear the alert on changes rather than after a designated length of t…

b1edbbb

…ime. natcap#1662

handle error and success alerts differently. natcap#1662

8dae658

make the accordion icon dynamic, and some other styling. natcap#1662

194f87a

remove extra padding for better alignment. natcap#1662

3643d9f

some cleanup; re-arringing css rules. natcap#1662

c86fbeb

davemfish requested a review from emilyanndavis February 19, 2025 17:43

emilyanndavis approved these changes Feb 19, 2025

View reviewed changes

emilyanndavis merged commit 0399389 into natcap:release/3.15.0 Feb 20, 2025
29 checks passed

davemfish mentioned this pull request Feb 25, 2025

Metadata: generate metadata sidecar files for invest model outputs #1662

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: GeoMetaMaker integration #1753

Feature: GeoMetaMaker integration #1753

davemfish commented Jan 28, 2025 •

edited

Loading

davemfish Jan 31, 2025

claire-simpson Feb 7, 2025

davemfish Feb 11, 2025

claire-simpson left a comment

claire-simpson Feb 7, 2025

davemfish Feb 11, 2025

davemfish commented Feb 11, 2025

emilyanndavis left a comment

emilyanndavis Feb 12, 2025

emilyanndavis Feb 12, 2025

emilyanndavis Feb 12, 2025

davemfish Feb 18, 2025

emilyanndavis Feb 12, 2025

davemfish Feb 18, 2025

emilyanndavis Feb 12, 2025

davemfish Feb 13, 2025

emilyanndavis Feb 12, 2025

emilyanndavis Feb 12, 2025

emilyanndavis Feb 13, 2025 •

edited

Loading

davemfish Feb 18, 2025

emilyanndavis Feb 12, 2025

emilyanndavis Feb 12, 2025

davemfish Feb 19, 2025

emilyanndavis Feb 12, 2025

davemfish Feb 18, 2025

emilyanndavis left a comment

		@@ -505,29 +504,5 @@ def test_model_specs_serialize(self):
		f'{error}')


		class SpecUtilsTests(unittest.TestCase):

		spec_utils.format_unit({})


		class TestDescribeArgFomSpec(unittest.TestCase):

Feature: GeoMetaMaker integration #1753

Feature: GeoMetaMaker integration #1753

Conversation

davemfish commented Jan 28, 2025 • edited Loading

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

claire-simpson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davemfish commented Feb 11, 2025

emilyanndavis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emilyanndavis Feb 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emilyanndavis left a comment

Choose a reason for hiding this comment

davemfish commented Jan 28, 2025 •

edited

Loading

emilyanndavis Feb 13, 2025 •

edited

Loading