Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add service to import data from remote URL vs uploading local file #5406

Merged
merged 10 commits into from
Aug 23, 2022
Merged

Add service to import data from remote URL vs uploading local file #5406

merged 10 commits into from
Aug 23, 2022

Conversation

brianhatchl
Copy link
Contributor

fixes #5391

@brianhatchl
Copy link
Contributor Author

brianhatchl commented Aug 11, 2022 •
edited
Loading

This PR adds a /remote service endpoint to import data from a URL. It makes use of the /vsicurl/, /vsis3/, and /vsizip/ features of OGR. The /vsizip/ is used to simplify expectations and also to use OGR to read osm data (vs Hoot's custom OSM reader that doesn't currently support remote URLs to osm files. However, this adds some caveats to the supported osm that can be read with OGR. For example, by default OGR doesn't like the negative IDs that would be on features not yet in the OSM db returning this error Unsupported node id value (-1560244). Use OSM_USE_CUSTOM_INDEXING=NO.

The S3 url will use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for the username and password values. This link can be used to test s3://hootenanny-remote-upload-test/dcpoi_clip.zip, I can provide the creds in a DM.

The http/ftp are currently using Basic Auth creds embedded in the hostname (e.g. http://user:pw@hostname:port/) but this is not ideal and not sure if it's supported with https and sftp/ftps.

To test the ftp w/basic auth endpoint I used this command to serve the /hoot/test-files/ directory with the zip file.
python3 -m pyftpdlib -i 192.168.56.1 -u hoot -P hoottest

The UI changes add a "Remote URL" import type:

remote_url

This choice then shows optional authentication fields:

remote_auth

@brianhatchl
allow negative IDs on osm read by ogr
24296cb
@brianhatchl
Submodule hoot-ui-2x cafbaa1ea..3718c4e74:
a7c8267 
  > add ftp
  > remove unneeded
  > undo change not needed
  > add eye icon to hide/show pw
  > use a password field
  > add username/pw, make remote url a data type
  > Merge remote-tracking branch 'origin/hoot2x' into import_remote_data
  > Merge remote-tracking branch 'origin/hoot2x' into import_remote_data
  > add a URL input to import remote data
@brianhatchl
manually set NONE translation to quick.js
c8024a0 
full path may avoid this error:
ImportError: Import by filename is not supported.
Error running convert:
Unable to find an appropriate scripting language for: translations/quick.js
@brianhatchl
Merge remote-tracking branch 'origin/master' into 5391_import_remote_…
b6d4344 
…data

# Conflicts:
#	hoot-ui-2x
@brianhatchl
Submodule hoot-ui-2x 3718c4e74..c2f24b3d7:
e8103ca 
  > update build info
  > Merge remote-tracking branch 'origin/hoot2x' into import_remote_data
  > allow sftp and ftps
@brianhatchl brianhatchl merged commit 26b2162 into master Aug 23, 2022
@brianhatchl brianhatchl deleted the 5391_import_remote_data branch August 23, 2022 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maxgrossman maxgrossman maxgrossman approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add feature to upload dataset from remote URL vs local file
2 participants
@brianhatchl @maxgrossman