Locality and Geocoorddetail batch import tool #4548

melton-jason · 2024-02-15T17:20:29Z

The Locality and GeoCoordDetail batch import tool allows users to match and update the Latitude1, Longitidude1, and Datum fields for Locality records and associate new GeoCoordDetails for the matched Locality records.

The primary motivation for the tool is such that the georefrencing for existing records in the database can take place outside of Specify (such as with CoGe), and there is no easy way to currently repatriate the data back into Specify and update those records.

This process is facilitated by requiring the guids of the Locality records in the CSV. Specify will use these guids to match/find the corresponding Locality record(s).
With the Locality record, the tool does the following:

If any latitude1, longitude1, and/or datum fields have values for the Locality record in the CSV, Specify will overwrite the current values in the database with those specified in the CSV
If any other column in the CSV matches a non-relationship field name in the GeoCoordDetail table, Specify will delete the previous GeoCoordDetail associated with the Locality record and create a new one populated with field values from the CSV
If there are any other columns in the CSV, Specify will display a warning to the user indicating there are unknown columns, but allow the user to proceed; the unknown columns will be ignored if the user proceeds.

Demo

Currently, access to the tool can be found in the User Tools menu:

The primary interface is very similar to that of importing a file to be used as a DataSet in the WorkBench. Specify supports different encodings and delimiters, and displays the first 100 rows of the file:

The Import File button can be pressed to initiate the parsing/upload.

If there are any unknown columns in the dataset, a warning will first be displayed to the user so they can decide how to proceed:

If there are any bad values discovered when parsing the results, Specify will display a downloadable message stating which line the error was found on along with a message with more information about how to resolve the parsing error:

If the parsing and upload is successful, Specify will display a brief results page stating how many records of each table were affected, along with the option to create a RecordSet of modified Locality Records:

Checklist

Self-review the PR after opening it to make sure the changes look good
and self-explanatory (or properly documented)
Add automated tests
Add relevant issue to release milestone

Testing instructions

For testing purposes, here is an example set of data for use on the KUFish database:
kufish_coge_sample.csv

Other datasets have been made for FITZ_NHMD, fwri, KUBirds, and lsumzmammals and are available in the following folder in the google drive:
https://drive.google.com/drive/folders/1B0hKQMBaX82nBUOB95uEa3H0MIrGrBv7?usp=drive_link

A foundation for other Datasets for use in other databases can be made by creating a Locality -> GeoCoordDetail query and exporting the results. For an example, here is the Query used to populate most of kufish_coge_sample.csv:
https://kufish51623-edge.test.specifysystems.org/specify/query/191/

There should be no errors in the kufish_coge_sample dataset presently, so modifications can be made to accomplish the desired errors and edge cases.

Generally, the cases which would need to be tested are:

Ensure an error message is displayed to the user when there is no Locality which has a guid which exists in the dataset
Data in unknown columns is properly ignored and not accounted for when uploading
If a Locality contains existing data for GeoCoordDetails, it should be deleted if there is at least one GeoCoordDetail field in the dataset
If there are no values in any GeoCoordDetail field for a Locality in the dataset and the Locality contains a GeoCoordDetail in Specify, the GeoCoordDetail should not be overwritten
Ensure that only the Locality fields which contain data in the dataset get updated for a Locality in the dataset (i.e., if longitude1 and datum have values in the dataset but latitude1 is either not a column or empty, ensure that latitude1 is never overwritten)
Ensure parsing error messages are intuitive enough to diagnose and resolve the problem with the dataset
- Potential ways to introduce invalid values include:
  - Entering a value in a field which exceeds the maximum length for the field (the maximum length for a field can be found in the Schema Configuration)
  - Entering an incorrect type of value for the field (such as entering a letter into a number/integer field)
  - Not following a UIFormatter for a field

specifyweb/frontend/js_src/lib/components/Header/ImportLocalitySet.tsx

specifyweb/frontend/js_src/lib/components/Molecules/FilePicker.tsx

specifyweb/frontend/js_src/lib/components/WbImport/index.tsx

specifyweb/frontend/js_src/lib/localization/header.ts

specifyweb/frontend/js_src/lib/localization/locality.ts

combs-a

Was doing some light testing on this and got this error. These coordinates should be fine in Specify, since Locality is fine with it. Looks like the issue is just the >= in the parse_latlong function in parse.py when it should just be >!

Good work so far btw Jason! While I haven't done full testing yet, what I've gotten through so far is great. 👍

melton-jason · 2024-06-18T16:52:31Z

Hi @specify/ux-testing!
I have merged production into this PR. Notably, this merge included changes from #4764, which included a lot of changes.

Also, if you encountered the Issue from #4998 in this PR before, it should now additionally be fixed.

Was doing some light testing on this and got this error. These coordinates should be fine in Specify, since Locality is fine with it. Looks like the issue is just the >= in the parse_latlong function in parse.py when it should just be >!

From #4548 (review)

Both the WorkBench and the Locality Update Tool utilizes the same parsing/validating code. (So this Issue is really the same as #4914).
This has been fixed as well!

lexiclevenger

Generally, the cases which would need to be tested are:

Ensure an error message is displayed to the user when there is no Locality which has a guid which exists in the dataset
Data in unknown columns is properly ignored and not accounted for when uploading
If a Locality contains existing data for GeoCoordDetails, it should be deleted if there is at least one GeoCoordDetail field in the dataset
If there are no values in any GeoCoordDetail field for a Locality in the dataset and the Locality contains a GeoCoordDetail in Specify, the GeoCoordDetail should not be overwritten
Ensure that only the Locality fields which contain data in the dataset get updated for a Locality in the dataset (i.e., if longitude1 and datum have values in the dataset but latitude1 is either not a column or empty, ensure that latitude1 is never overwritten)
Ensure parsing error messages are intuitive enough to diagnose and resolve the problem with the dataset
- Potential ways to introduce invalid values include:
  - Entering a value in a field which exceeds the maximum length for the field (the maximum length for a field can be found in the Schema Configuration)
  - Entering an incorrect type of value for the field (such as entering a letter into a number/integer field)
  - Not following a UIFormatter for a field

The latitude/longitude issue has been fixed! However, It looks like cells that are empty are overwriting fields that contain data.

Before Import:

Imported (blank) value:

After Import

Link to record: https://fwri1924-coge-import.test.specifysystems.org/specify/view/locality/35357/?recordsetid=1920
Imported file: fwri_coge_sample - fwri_coge_sample (8).csv

emenslin

Generally, the cases which would need to be tested are:

Ensure an error message is displayed to the user when there is no Locality which has a guid which exists in the dataset
Data in unknown columns is properly ignored and not accounted for when uploading
If a Locality contains existing data for GeoCoordDetails, it should be deleted if there is at least one GeoCoordDetail field in the dataset
If there are no values in any GeoCoordDetail field for a Locality in the dataset and the Locality contains a GeoCoordDetail in Specify, the GeoCoordDetail should not be overwritten
Ensure that only the Locality fields which contain data in the dataset get updated for a Locality in the dataset (i.e., if longitude1 and datum have values in the dataset but latitude1 is either not a column or empty, ensure that latitude1 is never overwritten)
Ensure parsing error messages are intuitive enough to diagnose and resolve the problem with the dataset
- Potential ways to introduce invalid values include:
  - Entering a value in a field which exceeds the maximum length for the field (the maximum length for a field can be found in the Schema Configuration)
  - Entering an incorrect type of value for the field (such as entering a letter into a number/integer field)
  - Not following a UIFormatter for a field

Looks great!

lexiclevenger

My previous comment was the expected behavior; everything looks good!

Areyes42

Generally, the cases which would need to be tested are:

Ensure an error message is displayed to the user when there is no Locality which has a guid which exists in the dataset
Data in unknown columns is properly ignored and not accounted for when uploading
If a Locality contains existing data for GeoCoordDetails, it should be deleted if there is at least one GeoCoordDetail field in the dataset
If there are no values in any GeoCoordDetail field for a Locality in the dataset and the Locality contains a GeoCoordDetail in Specify, the GeoCoordDetail should not be overwritten
Ensure that only the Locality fields which contain data in the dataset get updated for a Locality in the dataset (i.e., if longitude1 and datum have values in the dataset but latitude1 is either not a column or empty, ensure that latitude1 is never overwritten)
Ensure parsing error messages are intuitive enough to diagnose and resolve the problem with the dataset
- Potential ways to introduce invalid values include:
  - Entering a value in a field which exceeds the maximum length for the field (the maximum length for a field can be found in the Schema Configuration)
  - Entering an incorrect type of value for the field (such as entering a letter into a number/integer field)
  - Not following a UIFormatter for a field

Looks good! I also tested previous issues, and it seems like they all got fixed!

combs-a

Generally, the cases which would need to be tested are:

Ensure an error message is displayed to the user when there is no Locality which has a guid which exists in the dataset
Data in unknown columns is properly ignored and not accounted for when uploading
If a Locality contains existing data for GeoCoordDetails, it should be deleted if there is at least one GeoCoordDetail field in the dataset
If there are no values in any GeoCoordDetail field for a Locality in the dataset and the Locality contains a GeoCoordDetail in Specify, the GeoCoordDetail should not be overwritten
Ensure that only the Locality fields which contain data in the dataset get updated for a Locality in the dataset (i.e., if longitude1 and datum have values in the dataset but latitude1 is either not a column or empty, ensure that latitude1 is never overwritten)
Ensure parsing error messages are intuitive enough to diagnose and resolve the problem with the dataset
- Potential ways to introduce invalid values include:
  - Entering a value in a field which exceeds the maximum length for the field (the maximum length for a field can be found in the Schema Configuration)
  - Entering an incorrect type of value for the field (such as entering a letter into a number/integer field)
  - Not following a UIFormatter for a field

Some of the error messages have typos/etc, such as below:

Additionally, I couldn't do the Locality import on a user account that is a Specify7 Admin but didn't have a role defined in the collection:

18_batchadmin.mp4

Still, the majority of functionality is there and looking good Jason. Nice work! 🎉

melton-jason · 2024-06-18T20:32:44Z

@specify/ux-testing
The typos and "Additionally, I couldn't do the Locality import on a user account that is a Specify7 Admin but didn't have a role defined in the collection" have been fixed!

Previously, the backend was checking if the user was a specify 6 admin, not specify 7 admin...

combs-a

Looks good now! The issue with institution admins w/o an admin role in the collection not being able to upload is fixed now. 🎉

stale review

maxpatiiuk

sorry that I didn't get to reviewing this yesterday.
looking great!

maxpatiiuk · 2024-06-19T03:02:57Z

specifyweb/frontend/js_src/lib/components/LocalityUpdate/index.tsx

+  }
+
+  return (
+    <ProtectedAction action="%" resource="%">


uh interesting
is this to restict this to institutional admins only?
if so, could you put a short comment here?

maxpatiiuk · 2024-06-19T03:04:03Z

specifyweb/frontend/js_src/lib/components/LocalityUpdate/types.ts

+};
+
+export type LocalityUpdateTaskStatus =
+  | 'ABORTED'


for consistency, we usually name states in PascalCase not UPPER_CASE

maxpatiiuk · 2024-06-19T03:04:28Z

specifyweb/frontend/js_src/lib/components/LocalityUpdate/types.ts

+  readonly rowNumber: number;
+};
+
+export type LocalityUpdateTaskStatus =


no need to duplicate it. can infer it

Suggested change

export type LocalityUpdateTaskStatus =

export type LocalityUpdateTaskStatus = LocalityUpdateState['state'];

maxpatiiuk · 2024-06-19T03:05:18Z

specifyweb/frontend/js_src/lib/components/LocalityUpdate/types.ts

+      'ABORTED',
+      { readonly taskstatus: 'ABORTED'; readonly taskinfo: string }


hmm, this duplicates the fields
the State<> helper adds a {state:'ABORTED'}, and then you redundantly add taskstatus
Ideally you would pick one or another
If you want to keep the taskstatus name, then get rid of State<>

maxpatiiuk · 2024-06-19T03:06:20Z

specifyweb/frontend/js_src/lib/components/LocalityUpdate/utils.ts

+  } else if (key === 'multipleLocalitiesWithGuid') {
+    return localityText.multipleLocalitiesWithGuid({
+      guid: payload.guid as string,
+      localityIds: (payload.localityIds as RA<number>).join(', '),


call formatConjunction instead of join?

maxpatiiuk · 2024-06-19T03:06:57Z

specifyweb/frontend/js_src/lib/components/Merging/types.ts

@@ -1,6 +1,6 @@
-export type MergeStatus = 'ABORTED' | 'FAILED' | 'MERGING' | 'SUCCEEDED';
+export type MergingStatus = 'ABORTED' | 'FAILED' | 'MERGING' | 'SUCCEEDED';


(optional) use PascalCase for statuses?

maxpatiiuk · 2024-06-19T03:10:50Z

specifyweb/frontend/js_src/lib/components/QueryBuilder/CreateRecordSet.tsx

-    queryResource === undefined || queryResource.isNew()
-      ? undefined
-      : queryResource.get('name');
+function LoadingDialog(): JSX.Element {


(optional) add optional header parameter to <LoadingScreen /> since we needed that in several places
and then look for usages of loadingBar in dialog that can be replaced with it (there would be one in the react workbench I think)

maxpatiiuk · 2024-06-19T03:12:01Z

specifyweb/frontend/js_src/lib/components/WorkBench/Results.tsx

+  const resolvedRecords =
+    typeof rawSortFunction === 'function'
+      ? Object.entries(recordCounts).sort(sortFunction(rawSortFunction))
+      : Object.entries(recordCounts);


melton-jason added 14 commits December 27, 2023 20:45

Generalize CSV file picker

0fe9e2c

Setup routes and define 'skeleton' component

88efe08

Merge remote-tracking branch 'origin/xml-editor' into coge-import

1f4747e

Merge remote-tracking branch 'origin/xml-editor' into coge-import

ad1500b

Merge remote-tracking branch 'origin/production' into coge-import

6969ac5

Merge remote-tracking branch 'origin/production' into coge-import

0bb0373

Finish frontend implementation

ee69e2e

Merge remote-tracking branch 'origin/production' into coge-import

c523ec3

Merge remote-tracking branch 'origin/production' into coge-import

ad64121

Merge remote-tracking branch 'origin/production' into coge-import

e7310f3

Merge remote-tracking branch 'origin/production' into coge-import

d216a70

Generalize backend parsing behavior

1c7f0e3

Merge remote-tracking branch 'origin/production' into coge-import

b911d95

Finish parsing implementation

3bc5f7f

melton-jason added this to the 7.9.4 milestone Feb 15, 2024

melton-jason added 5 commits February 15, 2024 11:25

Resolve backend tests

f504503

Fix typo in ParseFailure

96cbf7d

Use the correct parse result class in parse date tests

af253ea

Merge remote-tracking branch 'origin/production' into coge-import

8aec0d3

Merge remote-tracking branch 'origin/production' into coge-import

88ee70e

melton-jason requested a review from maxpatiiuk February 21, 2024 21:13

Merge remote-tracking branch 'origin/production' into coge-import

fe130e2

melton-jason removed the request for review from maxpatiiuk February 21, 2024 21:13

maxpatiiuk previously requested changes Feb 23, 2024

View reviewed changes

melton-jason and others added 6 commits March 1, 2024 10:25

Merge branch 'production' into coge-import

39a4290

Merge remote-tracking branch 'origin/production' into coge-import

064dea0

Finish backend implementation and improve Parsing types

c017fb0

Improve frontend localizationKey for backend parse results

fbcf763

Extract 'Create RecordSet' button into a general component

6d963b6

Merge remote-tracking branch 'origin/production' into coge-import

b260c32

combs-a requested changes Jun 17, 2024

View reviewed changes

melton-jason and others added 3 commits June 18, 2024 08:39

Rename tool to 'Locality Update Tool'

66fca8f

Merge remote-tracking branch 'origin/production' into coge-import

9d647e5

Merge branch 'production' into coge-import

fbce549

melton-jason mentioned this pull request Jun 18, 2024

Workbench does not allow highest and lowest lat/long values #4914

Closed

melton-jason added 2 commits June 18, 2024 11:36

Allow hasPermission to check explict admin status

cfd4ec5

Only allow admins to use the Locality Update tool

deb10fa

melton-jason requested review from a team June 18, 2024 16:43

CarolineDenis approved these changes Jun 18, 2024

View reviewed changes

lexiclevenger requested changes Jun 18, 2024

View reviewed changes

emenslin approved these changes Jun 18, 2024

View reviewed changes

melton-jason added 2 commits June 18, 2024 13:19

Fix typo in LocalityUpdate recordset relationship

6f1731b

Provide additional data for SQLAlchemyModelTest

5b3d04e

lexiclevenger self-requested a review June 18, 2024 19:18

lexiclevenger approved these changes Jun 18, 2024

View reviewed changes

Merge branch 'production' into coge-import

608dd80

Areyes42 self-requested a review June 18, 2024 19:58

Areyes42 approved these changes Jun 18, 2024

View reviewed changes

combs-a requested changes Jun 18, 2024

View reviewed changes

Misc bug/typo fixes

8024c08

melton-jason requested a review from combs-a June 18, 2024 20:26

combs-a approved these changes Jun 18, 2024

View reviewed changes

melton-jason merged commit 0aed686 into production Jun 18, 2024
9 checks passed

melton-jason deleted the coge-import branch June 18, 2024 21:18

maxpatiiuk reviewed Jun 19, 2024

View reviewed changes

realVinayak mentioned this pull request Feb 9, 2025

Batch Edit: Support for editing basic fields #5417

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Locality and Geocoorddetail batch import tool #4548

Locality and Geocoorddetail batch import tool #4548

melton-jason commented Feb 15, 2024 •

edited

Loading

combs-a left a comment

melton-jason commented Jun 18, 2024

lexiclevenger left a comment •

edited

Loading

emenslin left a comment

lexiclevenger left a comment

Areyes42 left a comment

combs-a left a comment

melton-jason commented Jun 18, 2024 •

edited

Loading

combs-a left a comment

maxpatiiuk left a comment •

edited

Loading

maxpatiiuk Jun 19, 2024

maxpatiiuk Jun 19, 2024

maxpatiiuk Jun 19, 2024

maxpatiiuk Jun 19, 2024

maxpatiiuk Jun 19, 2024

maxpatiiuk Jun 19, 2024

maxpatiiuk Jun 19, 2024

maxpatiiuk Jun 19, 2024

	export type LocalityUpdateTaskStatus =
	export type LocalityUpdateTaskStatus = LocalityUpdateState['state'];

		'ABORTED',
		{ readonly taskstatus: 'ABORTED'; readonly taskinfo: string }

		@@ -1,6 +1,6 @@
		export type MergeStatus = 'ABORTED' \| 'FAILED' \| 'MERGING' \| 'SUCCEEDED';
		export type MergingStatus = 'ABORTED' \| 'FAILED' \| 'MERGING' \| 'SUCCEEDED';

Locality and Geocoorddetail batch import tool #4548

Locality and Geocoorddetail batch import tool #4548

Conversation

melton-jason commented Feb 15, 2024 • edited Loading

Demo

Checklist

Testing instructions

combs-a left a comment

Choose a reason for hiding this comment

melton-jason commented Jun 18, 2024

lexiclevenger left a comment • edited Loading

Choose a reason for hiding this comment

emenslin left a comment

Choose a reason for hiding this comment

lexiclevenger left a comment

Choose a reason for hiding this comment

Areyes42 left a comment

Choose a reason for hiding this comment

combs-a left a comment

Choose a reason for hiding this comment

melton-jason commented Jun 18, 2024 • edited Loading

combs-a left a comment

Choose a reason for hiding this comment

maxpatiiuk left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

melton-jason commented Feb 15, 2024 •

edited

Loading

lexiclevenger left a comment •

edited

Loading

melton-jason commented Jun 18, 2024 •

edited

Loading

maxpatiiuk left a comment •

edited

Loading