Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Locality and Geocoorddetail batch import tool #4548

Merged
merged 108 commits into from
Jun 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
0fe9e2c
Generalize CSV file picker
melton-jason Dec 28, 2023
88efe08
Setup routes and define 'skeleton' component
melton-jason Dec 28, 2023
1f4747e
Merge remote-tracking branch 'origin/xml-editor' into coge-import
melton-jason Dec 28, 2023
ad1500b
Merge remote-tracking branch 'origin/xml-editor' into coge-import
melton-jason Jan 7, 2024
6969ac5
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Jan 24, 2024
0bb0373
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Jan 25, 2024
ee69e2e
Finish frontend implementation
melton-jason Jan 26, 2024
c523ec3
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Jan 26, 2024
ad64121
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Jan 30, 2024
e7310f3
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Feb 9, 2024
d216a70
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Feb 14, 2024
1c7f0e3
Generalize backend parsing behavior
melton-jason Feb 14, 2024
b911d95
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Feb 14, 2024
3bc5f7f
Finish parsing implementation
melton-jason Feb 15, 2024
f504503
Resolve backend tests
melton-jason Feb 15, 2024
96cbf7d
Fix typo in ParseFailure
melton-jason Feb 15, 2024
af253ea
Use the correct parse result class in parse date tests
melton-jason Feb 15, 2024
8aec0d3
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Feb 16, 2024
88ee70e
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Feb 20, 2024
fe130e2
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Feb 21, 2024
39a4290
Merge branch 'production' into coge-import
melton-jason Mar 1, 2024
064dea0
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Mar 8, 2024
c017fb0
Finish backend implementation and improve Parsing types
melton-jason Mar 13, 2024
fbcf763
Improve frontend localizationKey for backend parse results
melton-jason Mar 13, 2024
6d963b6
Extract 'Create RecordSet' button into a general component
melton-jason Mar 13, 2024
b260c32
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Mar 13, 2024
6c4e781
Add localization for backend parse errors
melton-jason Mar 13, 2024
af383e0
Restore coordinateBadFormat ParseFailure
melton-jason Mar 13, 2024
fac170e
Finish frontend implementation
melton-jason Mar 13, 2024
09c5881
Cleanup frontend code
melton-jason Mar 15, 2024
2eefce0
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Mar 23, 2024
80bb476
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Mar 25, 2024
46b8355
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Apr 15, 2024
f0e372f
Reorganize http error messages
melton-jason Apr 15, 2024
b844b88
Remove CreateRecordSetFromQuery component
melton-jason Apr 15, 2024
6741ebd
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Apr 17, 2024
1e407fd
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Apr 30, 2024
37b71f1
Reset all context when closing dialogs
melton-jason Apr 30, 2024
d2d3c19
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason May 6, 2024
07a8393
Return 422 status when Parse Error occurs
melton-jason May 8, 2024
0bcea7f
Include geocoorddetail ids in response
melton-jason May 9, 2024
2baf5c2
Show number of affected records after import
melton-jason May 9, 2024
c53c396
Remove redundant initial import dialog
melton-jason May 9, 2024
fd1be30
Fix infinite loading bug in query results
melton-jason May 9, 2024
1f21835
Simplify onFileImport for CSV file picker
melton-jason May 9, 2024
0a86329
Return in onFileImport promise
melton-jason May 9, 2024
f45b66c
Fix misc. parsing bugs
melton-jason May 9, 2024
1498f42
Add field to valueTooLong payload in upload test
melton-jason May 9, 2024
fb6a5ce
Add separate parsing endpoint
melton-jason May 14, 2024
c3a5aae
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason May 16, 2024
618c805
Hide the create record set button if has no permission
melton-jason May 16, 2024
f532854
Allow specifying button type for CreateRecordSet button
melton-jason May 16, 2024
c1e67c8
Remove unuesed imports
melton-jason May 16, 2024
0e64f0c
Make buttonType optional for CreateRecordSet button
melton-jason May 16, 2024
3d8c4eb
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason May 23, 2024
b0f549f
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason May 23, 2024
b6abff0
Automatically create record set on import
melton-jason May 23, 2024
6e5330a
Merge branch 'production' into coge-import
melton-jason May 26, 2024
050f38b
Always include field which caused error
melton-jason May 27, 2024
f746ee5
Rename generic instances of 'coge' to locality dataset
melton-jason May 27, 2024
fc660fa
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason May 27, 2024
a9a3975
Close columnHeaders error dialog on import
melton-jason May 27, 2024
4444909
Merge branch 'production' into coge-import
melton-jason May 28, 2024
56f521b
Define generic AsyncTask abstract model
melton-jason May 29, 2024
958196c
Ensure consistency with AsyncTask models and other tables
melton-jason May 30, 2024
29f8f18
Add LocalityImport AsyncTask model
melton-jason May 30, 2024
2a8e33d
backend: Integrate Import Locality with worker and notifications
melton-jason May 30, 2024
93e228f
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason May 30, 2024
f0edc66
frontend: Implement Loading/Progress bar
melton-jason Jun 4, 2024
1f640c3
Reorganize Locality Import Data Set Code
melton-jason Jun 4, 2024
453e865
Merge branch 'production' into coge-import
melton-jason Jun 4, 2024
25b1dc5
Add notification renderers for locality import tool
melton-jason Jun 5, 2024
b935df7
Use 1-index start for rowNumbers
melton-jason Jun 5, 2024
5ce827b
Merge branch 'production' into coge-import
melton-jason Jun 5, 2024
94c2e03
Merge Parsing and Upload states
melton-jason Jun 6, 2024
1a75a2e
Always parse the locality and geocoordetail results as objects
melton-jason Jun 6, 2024
fcaf839
Don't sync the Error dailog dimensions with progress dialogs
melton-jason Jun 6, 2024
a3ad3be
Improve appearance of the Parse Error dialog
melton-jason Jun 6, 2024
67dee64
Resolve Typescript errors
melton-jason Jun 6, 2024
2c2be23
Remove unused import
melton-jason Jun 6, 2024
df949e8
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Jun 6, 2024
14a4987
Don't show loading dialog while cancelling task
melton-jason Jun 6, 2024
e97b526
Add option to disable 'First Row is Header' for CSVFilePreview
melton-jason Jun 7, 2024
2b4d4c5
Improve appearance of LocalityImport Error dialog
melton-jason Jun 7, 2024
8ccb406
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Jun 10, 2024
7c4360e
Restore Parsing State
melton-jason Jun 10, 2024
0c62335
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Jun 10, 2024
618402e
Add csv-parse to Jest transformIgnorePatterns
melton-jason Jun 12, 2024
6d1ac8b
Use WebPack v5 native Workers over loader
melton-jason Jun 12, 2024
7682fec
Add ts-expect error and comment on import.meta.url
melton-jason Jun 12, 2024
fef787b
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Jun 12, 2024
cf65af8
backend: split row results and allow parsing with worker
melton-jason Jun 13, 2024
e53af54
Show confirmation dialog with number of affected records
melton-jason Jun 13, 2024
59bcd42
Fix OpenAPI schema errors
melton-jason Jun 13, 2024
c411e0d
Only reset headerError context on import
melton-jason Jun 13, 2024
177ae7a
Update LocalityImport Notification renderers
melton-jason Jun 13, 2024
4d6f589
Insert a LocalityImportRowResult for each Parsed Row
melton-jason Jun 17, 2024
e8a2ced
Handle backend exceptions and refactor Parsed return
melton-jason Jun 17, 2024
13c5b59
Extract Unknown State to own component
melton-jason Jun 17, 2024
66fca8f
Rename tool to 'Locality Update Tool'
melton-jason Jun 18, 2024
9d647e5
Merge remote-tracking branch 'origin/production' into coge-import
melton-jason Jun 18, 2024
fbce549
Merge branch 'production' into coge-import
melton-jason Jun 18, 2024
cfd4ec5
Allow hasPermission to check explict admin status
melton-jason Jun 18, 2024
deb10fa
Only allow admins to use the Locality Update tool
melton-jason Jun 18, 2024
6f1731b
Fix typo in LocalityUpdate recordset relationship
melton-jason Jun 18, 2024
5b3d04e
Provide additional data for SQLAlchemyModelTest
melton-jason Jun 18, 2024
608dd80
Merge branch 'production' into coge-import
melton-jason Jun 18, 2024
8024c08
Misc bug/typo fixes
melton-jason Jun 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion specifyweb/celery_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,28 @@
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

class CELERY_TASK_STATE:
""" Built-In Celery Task States
See https://docs.celeryq.dev/en/stable/userguide/tasks.html#built-in-states
"""
FAILURE = 'FAILURE'
PENDING = 'PENDING'
RECEIVED = 'RECEIVED'
RETRY = 'RETRY'
REVOKED = 'REVOKED'
STARTED = 'STARTED'
SUCCESS = 'SUCCESS'

@app.task(bind=True)
def debug_task(self):
print(f'Request: {self.request!r}')


logger = get_task_logger(__name__)


class LogErrorsTask(Task):
def on_failure(self, exc, task_id, args, kwargs, einfo):
logger.exception('Celery task failure!!!1', exc_info=exc)
super(LogErrorsTask, self).on_failure(exc, task_id, args, kwargs, einfo)
super(LogErrorsTask, self).on_failure(
exc, task_id, args, kwargs, einfo)
1 change: 1 addition & 0 deletions specifyweb/context/openapi_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -414,6 +414,7 @@
"type": "string",
"enum": [
"array",
"null",
"boolean",
"integer",
"number",
Expand Down
2 changes: 1 addition & 1 deletion specifyweb/frontend/js_src/babel.config.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ module.exports = {
['@babel/preset-react'],
['@babel/preset-typescript'],
],
plugins: ['@babel/plugin-transform-modules-commonjs'],
plugins: ['@babel/plugin-transform-modules-commonjs', "babel-plugin-transform-import-meta"],
},
},
};
2 changes: 1 addition & 1 deletion specifyweb/frontend/js_src/jest.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,7 @@ const config: Config.InitialOptions = {

// See https://stackoverflow.com/questions/69075510/jest-tests-failing-on-d3-import
transformIgnorePatterns: [
'/node_modules/(?!d3|d3-array|internmap|delaunator|robust-predicates|csv-stringify)',
'/node_modules/(?!d3|d3-array|internmap|delaunator|robust-predicates|csv-stringify|csv-parse)',
],

// An array of regexp pattern strings that are matched against all modules before the module loader will automatically return a mock for them
Expand Down
3 changes: 3 additions & 0 deletions specifyweb/frontend/js_src/lib/components/Atoms/Icons.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,9 @@ export const icons = {
fingerPrint: <svg aria-hidden className={iconClassName} fill="currentColor" viewBox="0 0 20 20" xmlns="http://www.w3.org/2000/svg"><path clipRule="evenodd" d="M6.625 2.655A9 9 0 0119 11a1 1 0 11-2 0 7 7 0 00-9.625-6.492 1 1 0 11-.75-1.853zM4.662 4.959A1 1 0 014.75 6.37 6.97 6.97 0 003 11a1 1 0 11-2 0 8.97 8.97 0 012.25-5.953 1 1 0 011.412-.088z" fillRule="evenodd" /><path clipRule="evenodd" d="M5 11a5 5 0 1110 0 1 1 0 11-2 0 3 3 0 10-6 0c0 1.677-.345 3.276-.968 4.729a1 1 0 11-1.838-.789A9.964 9.964 0 005 11zm8.921 2.012a1 1 0 01.831 1.145 19.86 19.86 0 01-.545 2.436 1 1 0 11-1.92-.558c.207-.713.371-1.445.49-2.192a1 1 0 011.144-.83z" fillRule="evenodd" /><path clipRule="evenodd" d="M10 10a1 1 0 011 1c0 2.236-.46 4.368-1.29 6.304a1 1 0 01-1.838-.789A13.952 13.952 0 009 11a1 1 0 011-1z" fillRule="evenodd" /></svg>,
gallery:<svg aria-hidden className={iconClassName} fill="currentColor" viewBox="0 0 20 20" xmlns="http://www.w3.org/2000/svg">
<path clipRule="evenodd" d="M5 3a2 2 0 00-2 2v2a2 2 0 002 2h2a2 2 0 002-2V5a2 2 0 00-2-2H5zM5 11a2 2 0 00-2 2v2a2 2 0 002 2h2a2 2 0 002-2v-2a2 2 0 00-2-2H5zM11 5a2 2 0 012-2h2a2 2 0 012 2v2a2 2 0 01-2 2h-2a2 2 0 01-2-2V5zM11 13a2 2 0 012-2h2a2 2 0 012 2v2a2 2 0 01-2 2h-2a2 2 0 01-2-2v-2z" fillRule="evenodd" />
</svg>,
globe: <svg aria-hidden className={iconClassName} fill="currentColor" viewBox="0 0 20 20" xmlns="http://www.w3.org/2000/svg">
<path clipRule="evenodd" fillRule="evenodd" d="M4.083 9h1.946c.089-1.546.383-2.97.837-4.118A6.004 6.004 0 004.083 9zM10 2a8 8 0 100 16 8 8 0 000-16zm0 2c-.076 0-.232.032-.465.262-.238.234-.497.623-.737 1.182-.389.907-.673 2.142-.766 3.556h3.936c-.093-1.414-.377-2.649-.766-3.556-.24-.56-.5-.948-.737-1.182C10.232 4.032 10.076 4 10 4zm3.971 5c-.089-1.546-.383-2.97-.837-4.118A6.004 6.004 0 0115.917 9h-1.946zm-2.003 2H8.032c.093 1.414.377 2.649.766 3.556.24.56.5.948.737 1.182.233.23.389.262.465.262.076 0 .232-.032.465-.262.238-.234.498-.623.737-1.182.389-.907.673-2.142.766-3.556zm1.166 4.118c.454-1.147.748-2.572.837-4.118h1.946a6.004 6.004 0 01-2.783 4.118zm-6.268 0C6.412 13.97 6.118 12.546 6.03 11H4.083a6.004 6.004 0 002.783 4.118z" />
</svg>,
hashtag: <svg aria-hidden className={iconClassName} fill="currentColor" viewBox="0 0 20 20" xmlns="http://www.w3.org/2000/svg"><path clipRule="evenodd" d="M9.243 3.03a1 1 0 01.727 1.213L9.53 6h2.94l.56-2.243a1 1 0 111.94.486L14.53 6H17a1 1 0 110 2h-2.97l-1 4H15a1 1 0 110 2h-2.47l-.56 2.242a1 1 0 11-1.94-.485L10.47 14H7.53l-.56 2.242a1 1 0 11-1.94-.485L5.47 14H3a1 1 0 110-2h2.97l1-4H5a1 1 0 110-2h2.47l.56-2.243a1 1 0 011.213-.727zM9.03 8l-1 4h2.938l1-4H9.031z" fillRule="evenodd" /></svg>,
// This icon is not from Heroicons. It was drawn by @grantfitzsimmons
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,14 @@ const rawUserTools = ensure<IR<IR<Omit<MenuItem, 'name'>>>>()({
icon: icons.rss,
},
},
[commonText.import()]: {
localityUpdate: {
title: headerText.localityUpdateTool(),
enabled: () => userInformation.isadmin,
url: '/specify/import/locality-dataset/',
icon: icons.globe,
},
},
[headerText.documentation()]: {
aboutSpecify: {
title: welcomeText.aboutSpecify(),
Expand Down
Loading
Loading