Extend service to retrieve values from HTTP request by XPATH (for example) path

[reddog]

Building upon #548, extend badge source by permitting specification by XPATH (for example) path.

This extends #820 where JSON from a HTTP request can be used as the button source This issues is to allow for non-JSON sources. In particular, XPATH path specifiers will allow for regular HTML web pages to be used as a button source.

[paulmelnikow]

Hi there! This sounds like a really powerful features and is a great suggestion.

Unfortunately parsing HTML is pretty resource intensive and our server resources are really limited. Until they expand significantly we won't be able to enable a feature like this on the production servers. However I'd be happy to have this support in the codebase under a feature toggle.

It's easy to host your own server using Now, Heroku, or another cloud platform.

Let me know if you'd like to do that!

[paulmelnikow]

Also, if you could help spread the word that we need to raise money, that would be helpful! We're asking for one-time donations of $10 from people who use and love Shields. https://opencollective.com/shields

[tooomm]

Let's start with xml first, as initially planned by @RedSparr0w: #820 (comment)

[reddog]

Thanks for your reply @paulmelnikow. A few follow up questions:

1. if it was something I (or someone else) could work on, what's your preferred method? A fork and pull request?

2. I don't know too much about the workings of Shields - is there some sort of technical architecture / process document I could use to get up to speed? eg I don't know if Shields goes off and generates a new button (from whatever source) on every request, or if there's some form of cache, etc.

3. assuming (and it is an assumption!) that there is some form of caching of created buttons in place, would a possible solution for xpath-source buttons be for these to be added to some sort of processing queue which is only actioned when there's free resources? Is that something that would work at all?

Happy to promote the $10 fund!

Can consider the self-hosted route if none of this works out.

@tooomm would you see XML sources being dealt with differently in some way that xpath wouldn't work for them? Maybe a separate issue for XML sources would be useful if a different approach is required?

[paulmelnikow]

Hi!

if it was something I (or someone else) could work on, what's your preferred method? A fork and pull request?

Yes, a fork and a pull request is good. If you have questions along the way, this issue is a great place to get answers.

I don't know too much about the workings of Shields - is there some sort of technical architecture / process document I could use to get up to speed? eg I don't know if Shields goes off and generates a new button (from whatever source) on every request, or if there's some form of cache, etc.

There is not, unfortunately. It's been asked for before in #114. The short answer is – it's very, very simple. A request comes in for a badge, we make a request to an underlying service, we render it as SVG, and we return it. We use caches for the service requests, and also for the text measuring, which is the slow aspect of the rendering. Github auth has some fancy stuff… though otherwise there's nothing too fancy. Do you have any particular questions?

assuming (and it is an assumption!) that there is some form of caching of created buttons in place, would a possible solution for xpath-source buttons be for these to be added to some sort of processing queue which is only actioned when there's free resources? Is that something that would work at all?

That sounds possible, though I'm not sure exactly how we'd go about that. Maybe we could force a bottleneck, so that only one of these parsing requests is going on per server at a time? That should mostly guarantee that the impact of this feature is limited to people who are using it, though of course not completely, since there are other things that will be running on the server at the time…

Sorry for the long lag here. Hope to be more responsive going forward!

[paulmelnikow]

I think a good way to solve this would be a new feature we're offering: the Endpoint badge. You can use it with tools like RunKit and Jupyter Kernel Gateway to implement the exact badge content and logic you want, while letting Shields take care of formatting, rendering, and caching.

If you decide to use the new feature, your feedback is much appreciated! Please feel free to comment on the Endpoint beta issue if you have feedback or questions about how to use it.