Qview - Nomenclature tool broken

[lwellerastro]

ISIS version(s) affected: 4.1.1 as well as older 3.x.y versions

Description
Using nomenclature tool on mosaic produces an error dialog. Can't connect to features database.

How to reproduce
Open an mosaic (or unprojected data, I think) and turn on nomenclature tool (ID icon).

[jessemapel]

@thareUSGS Do you know if the nomenclature database/web portal has changed?

[thareUSGS]

That link does work and I don't think the current service has changed in a while (but I have never really worked on this site). https://planetarynames.wr.usgs.gov/nomenclature/SearchResults

Now I think many of our web servers and host OSs have been upgraded, so there could be something introduced like a time-out issue. Although, the error does return pretty quickly.

I have always been a little concerned on how this was implemented (instead of using the available Web Feature Server API). Looking at the code, as far as I can tell, it is actually filling out the advanced search web page/form (acting like a user) and then parsing the results. If not a time-out issue, perhaps updates to our proxy or OS/web server might not allow this any more? But I am completely guessing here. Not sure what advice to offer to fix this now besides maybe checking with IT if the host machine, web server, or any adjustments to the machine's proxy has been changed...?

[jessemapel]

@thareUSGS That's still helpful. I doubt this code has been touched in ISIS since it was written, so we'll probably need to have someone dig in on this one. It sounds like this needs to be re-written to use the API properly anyway.

[lwellerastro]

I haven't used the tool in many years, but I'm working with Titan data and in reading some papers, etc. I was trying to locate features by name (they didn't provide lat/lon info). This is not a priority, just an FYI so feel free to drop the products label if necessary. It is a handy tool when you need it, but not a necessity.

I also tried to use it under an older version of isis (3.6.0) but it failed there as well, so likely a server/IT issue as Trent suggested.

[thareUSGS]

complete shot in the dark but maybe change the address here to https://planetarynames.wr.usgs.gov/nomenclature/AdvancedSearch

[lwellerastro]

@acpaquette, any Themis IR tile the CTX folks are using is good for testing if you need something. I can point you to small mosaics/images as well.

[acpaquette]

@lwellerastro If you could point me to some small mosaics I would really appreciate it!

[jlaura]

@acpaquette /scratch/ladoramkershner/mars_quads/oxia_palus/center/test/mc11_oxia_palus_dir_final.cub - for one example.

[lwellerastro]

sorry I missed your request @paarongiroux - I didn't get the notification. I think it's a casualty of the bot onslaught the other day. Thanks @jlaura

[acpaquette]

https://github.com/USGS-Astrogeology/PlanetaryNomenclature/commit/c5ba81e74a495fda156e02a292987b6dd1594484#

Looks like a change was made for posts at the SearchResults URL

[jlaura]

@acpaquette That change alters the response that we are getting back from the API? (Update: the post isn't working so something changed - from the standup. Guessing this happened during the flask conversion.)

@amesamoyers @thareUSGS Are you all versioning the nomenclature API or could changes to the user facing side happen without warning? Also, can you all link us to docs on how we should be consuming this API?

[thareUSGS]

While nomenclature has had a WFS API for a long while (Mapserver WFS/WMS) , the ISIS tool never used it. The ISIS program simply used the advanced web form to pretend to be a user and then request a https response. That search form could be thought of as an API I guess. In that regard it probably has not been versioned. But I don't think that form hasn't been touched/updated in years. Thus I still think original issue here is a hardware upgrade, network change, or ...?

That said, we did warn against Qview using the advanced search form (really for humans) to solve this issue (hoping it would use the WFS API). Note the WFS API has always been versioned. Now as we move to Geoserver for our WFS needs (including Nomenclature), Geoserver allows the user to pick the WFS version they want (thus is good at allowing backward compatibility). Unfortunately I am not 100% comfortable with our current GeoServer implementation. Still needs some help/time to stabilize it.

[jlaura]

@thareUSGS I am not fully understanding your response. The web form that you are referencing packages the data and sends it to an endpoint. That endpoint, via a GET request, is the API that ISIS was using. That endpoint has changed how it works, which broke the nomenclature tool. The advanced search form is just an HTML page that packages data and ships it off to some endpoint. It is that endpoint that ISIS is trying to use.

What API should we be using and where is the documentation on how to use it?

Aside: As long as you have an unsecured endpoint, anyone in the world could be consuming it in the same way. So I would argue it is not entirely safe to assume that changes can be made without breakages.

[amesamoyers]

Hi all, I am taking a look at this issue right now.

To give some context, the codebase for the production nomenclature site is housed and deployed from the USGS gitlab: code.chs.usgs.gov/asc/dk8s/planetary-names -- this is still a Java site. This was moved to gitlab within the past year and has since undergone deployment changes but negligible (if any) functionality change. The link referenced above-- USGS-Astrogeology/PlanetaryNomenclature@c5ba81e is a commit in the flask re-write repository that we were housing on github (this has since also been moved to gitlab to gear up for deployment). There have been no production related changes to accomodate the flask python re-write, we have only begun to officially deploy that code to a dev site for testing. I'm currently troubleshooting this with IT. We haven't yet determined the underlying cause but I am actively working to resolve and will check back.

@jlaura @acpaquette I was not aware that external products call the endpoints from nomenclature in an api-driven way.
When we started re-writing nomenclature as a flask app we didn't approach it this way, although changes could be made to generate responses that are more conducive. The goal with the re-write is to end intermediate calls to nomenclature and have those go directly to the WFS server that sits on top of nomenclature's data, like what @thareUSGS is describing. Geoserver is the longterm WFS server but until it is production ready (and the flask re-write deployed) I'm down to troubleshoot this and figure out what is going on short term.

Thank you @lwellerastro for submitting--

[jlaura]

@amesamoyers @thareUSGS We are going to mark this as blocked. A few requests if possible:

1. Can we track all conversation on this issue here? Sounds like we have some other conversations going on that would be good to capture here so that all the stakeholders can see the progress.
2. If this is just a firewall issue, what is the timeline for having that fixed? Once fixed, I am still going to leave this as blocked because, without documentation, we have not actually solved the issue.
3. Can you let us know the timeline to having a documented, versioned API available to us? We have users that make regular use of the nomenclature tool and we want to maximize the stability for them. Since we are a consumer of the nomenclature service, that stability is dependent, in part, on the stability of your service.
4. Can we archive or remove the nomenclature site from GitHub? If that repository is not being used this will remove confusion in where to look should one want to see the code.

[amesamoyers]

@jlaura

1. I agree, I will @ people on the rest of the process here
2. @evindunn-usgs can you post back here what you find on AdvancedSearch() for networking or infrastructure related issues and we can explore further based on that.
3. @thareUSGS can you weigh in on us doing a documented, versioned API -- I'm happy to work on this. Based on it being Geoserver, much of it will be Geoserver CQL query statements but there would need to be additional documentation in terms of what layers and resources are available within that and we also need to figure out the timeframe for when that will be considered production ready.
4. I agree, we should get the https://github.com/USGS-Astrogeology/PlanetaryNomenclature repo archived. I don't have access to do this but if @USGSsakins or @jlaura has access (is there an admin group beyond repo owner?), will one of you archive?

I haven't been doing alot of documentation work yet, right now I am focusing on getting the flask re-write finished so we can deploy it, if there's any type of documentation aside from this I'd be happy to contribute that as well.

[jlaura]

@amesamoyers Sounds good!

@amesamoyers - Item 4: I have archived the repository.

@amesamoyers @thareUSGS What kind of timeline are we looking at as well? I would like to help our users know when they can expect to have this working again.

[evindunn]

@amesamoyers Network latency should only happen in dev, when your local machine is making calls to a database that's at ASC. When the site's live, it'll be at ASC too, so no latency. I think that's what you're asking?

[amesamoyers]

@evindunn question was for other networking issues related to Qview's ability to hit the AdvancedSearch() endpoint in Nomenclature, thank you for reaching out to USGS networking on this. I see the email that they increased the keepalive timeout, @acpaquette can you test to see if QView is getting expected responses from AdvancedSearch()?

[jlaura]

@acpaquette Let us know yesterday that the longer keep alive did not help. I believe he said that the failure time was identical to before the increase was put in place. @acpaquette Did I convey that at all correctly?

[evindunn]

@jlaura @acpaquette @amesamoyers I've done some testing and it's a networking issue that I hope to get resolved soon. I'm going to keep it vague here as it's a public setting.

[jlaura]

👍 @evindunn That sounds great and the vague update is an awesome way to keep all our users in the loop without having to be super specific when we can't. Thanks!

[evindunn]

@jlaura @acpaquette @amesamoyers Here's how I got it working:

#!/bin/bash

form_data=(
    'target=MOON'
    'is_positive_east=true'
    'is_0_360=false'
    'is_planetographic=false'
    'featureNameColumn=true'
    'targetColumn=true'
    'diameterColumn=true'
    'centerLatLonColumn=true'
    'coordSystemColumn=true'
    'approvalStatusColumn=true'
    'approvalDateColumn=true'
    'originColumn=true'
    'sort_column=name'
    'sort_asc=true'
    'displayType=CSV'
    'Search=Search'
)

form_data_joined=$(IFS='&'; echo "${form_data[*]}")

curl -siL -X POST -d "$form_data_joined" \
    -H 'Content-Type: application/x-www-form-urlencoded' \
    -H 'User-Agent: Mozilla/5.0 (X11; Linux i686; rv:6.0) Gecko/20100101 Firefox/6.0' \
    https://planetarynames.wr.usgs.gov/SearchResults

The biggest change is removing /nomenclature from the path and switching back to /SearchResults
https://planetarynames.wr.usgs.gov/SearchResults

[jlaura]

Fixed with #4030

[lwellerastro]

Perhaps this should be a new issue since this is an old post, but I get the following error while trying to use the nomenclature tool via isis7.0.0:

Not sure if this is an older, uncaught error or if this is new and due to the recent changes with our various web sites.
FYI at the very least.

[jlaura]

Sent at 9:35 via email to all internal ASC personnel.

Astro's public websites running in AWS Kubernetes (EKS) are having issues and are down. This is all our public web services: Pilot, Astropedia, Nomeclature, etc. Also Astro-budgets

We are investigating, Thanks