-
-
Notifications
You must be signed in to change notification settings - Fork 431
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Community Marketplace Add-on Service - initial contribution #2405
Conversation
A jar might not be sufficient since a lot of bindings have dependencies (on core-features or other bundles). Karaf features should be available as well. |
@ghys Great Job Yannick, already saw the new marketplece in the community. |
kar-files would be another option for more complex addons. |
I too saw the threads created in the forum and got excited. A leading question similar to J-N-K's but for rules. Based on the above it appears that only UI created rules will be supported (at least to start). That doesn't seem wholly unreasonable. However, is there any thought about rules/rule templates that have external dependencies too? For example, I've an implementation of the Time of Day all but ready to submit. In fact I've several implementations of Design Patterns. However, there is a lot of common code between them all and that common code is put into library files in $OH_CONF/automation/lib/javascript/community. Is there any thought made about whether/how to handle that? I can totally see that being a version two goal but without the ability to include something like that, building and maintaining rule templates like this will become quite a challenge. |
Yes and the UI widgets could have dependencies too, static assets like images or stylesheets. There's a system of "handlers" in this PR which was ported over the old Eclipse Marketplace service (https://github.com/openhab/openhab2-addons/blob/2.5.x/bundles/org.openhab.extensionservice.marketplace/src/main/java/org/openhab/extensionservice/marketplace/MarketplaceExtensionHandler.java); it works similarly: the service submits the Discourse post transformed into an This allowed to port over the existing handlers with little change - the Eclipse Marketplace only supported .jar bundles and rule templates in .json format, and I threw in the UI widget handler + YAML support instead of JSON because it was relatively straightforward. This already handles simple situations when someone just wants to share something easily. For more complex situations, there could be a handler detecting manifests in the post as mentioned above, that could deal with these complex use cases, but the spec & implementation of this could take time and as you've said it can wait. For the record, don't expect even any of this to land in 3.1. |
Hey @ghys, really awesome work, thank you so much! For the moderation, I think we could try it with letting people publish directly i.e. moderation "a posteriori". If this leads to much crap and problems, we can still change our approach on it. Nonetheless, using tags to identify published once can be helpful, so people could start drafting and only deciding to publish it once they are satisfied with it. For @rkoshak's question, I guess that KARs could be solution for it (that could be implemented in a later step). In KARs, we can package feature files, which in turn can package files for the file system or define dependencies to other artefacts that need to be installed with it. Two problems I see with KARs is that dependencies would need to be available in a Maven repo that we reference in the distro and that their creation is not as straight forward as zipping an archive with a few files only. But I guess whenever one wants to introduce dependency management, complexity naturally comes with it... That's a good argument to leave it out in step 1. What do you still see as a todo on this PR? The todos you list above seem to be "nice to haves" for a follow-up, but not necessarily mandatory for the first implementation, right? |
As long as the process is documented well enough for those who don't know maven from makefiles and/or provide tooling and scripts, I don't think putting them into kar files is too much to ask. Though in this scenario each kar file is self contained, or can they define external stuff as dependencies. For example, let's say I write a rule template in Python. Obviously that means the rule will require the Jython add-on as a requirement. Can I specify that requirement in the kar file somehow? If not it can be handled in the docs for the rule template I'm sure. Another example. Let's say I've written a utility library called timerMgr which makes it easier to create and interact with a bunch of timers in a rule (e.g. a one timer per Item situation). This isn't a rule unto itself but used by rules. So would I deploy this as it's own kar file and then in my Time of Day Rule just specify "this rule depends on timerMgr" or would I embed timerMgr into the Time of Day kar file and make Time of Day available as a self contained unit? I don't see anything wrong with either approach. The rules developer doesn't necessarily need to keep multiple copies of timerMgr in their source control system. The kar build process would just have to grab a copy of it when building the kar file for Time Of Day and any other rule that uses it. I'm just thinking about the thought process so I can start preparing my existing code and structuring new stuff I write. I don't mind being an alfa and beta tester for this even if it means making inline copies of my libraries into each rule for now. I'm sure I can come up with a script to make that easier to manage until version 2 and 3 when kar files (or whatever) becomes supported. :-D But it would be nice to plan now for what's coming down the line. |
Agreed, let's go with the tag then; we can either educate authors so they apply it themselves, this allows an eventual drafting phase before self-publishing (and even unpublishing if necessary); and if no pre-moderation leads to vandalism, abuse or poor quality entries we have the ability to change policy instantly and restrict self-publishing using the "tags are visible to anyone but only the following groups can use them" option shown above.
Mostly I submitted this draft PR as soon as I had something working to validate some of the design choices and possibly adjust them early, like:
|
I think the same, that's why I meant caching could be done at a later stage in case we see that we need it. As we roll it out slowly (first snapshots, then milestones and only at the end of 2021 as a release), I think we have enough time to check what load it causes in reality. Licenses are a good point, I think that is a must. We might even have closed source stuff, where people only provide a jar. Would hence be good to possibly have both, a license as a SPDX and a link to the license text.
Yes, I am for it. It does not add much to the distro and improves the UX and your possibilities in the UI.
Yes, see above.
Yes, an opt-out makes sense if people want to really have an offline system. |
I just had a look at how to implement this. Unfortunately, the "Add-on Management" service (and with it the |
Some bindings (e.g. SNMP, Network, ...) need global configuration. Dropping the JAR of such a binding in the addons folder starts the binding, but the configuration can only be adjusted manually on the karaf console via the configuration admin (which is nearly impossible to explain to the "normal" user) because the UI only shows addons that are installed as a feature. This should be avoided for marketplace addons. |
Too bad, but having a separate service configuration works. I was merely advocating for a consolidation of the configuration of similar services to avoid a proliferation of top-level menu entries in the UI with only a couple of options. Regional Settings and Ephemeris also come to mind - I remember suggesting this a while ago. But if it's not trivial for a service to source its configuration from another service, then so be it. This could be solved at the UI level too, I guess.
This has been reported before and thus was added to the page where you can choose the binding when adding a new thing (Settings > Things > + button). Configurable bindings will have a gears icon you can click to access the configuration. It's the only place where all bindings whatever their source are listed. To make a comparison with e.g. Android, the Settings > Add-ons part of the UI is the Play Store (so mostly used to browse and install/uninstall) while dropping a .jar in the addons folder is like sideloading an app from its .apk file. |
@ghys Interesting. I never noticed that. Thanks. |
Depends on openhab/openhab-core#2405. This replaces the current section of the UI for managing add-ons with a completely new "store-like" interface. External data, like logos and detailed descriptions for distribution add-ons are fetched from well-known locations on the website or GitHub to provide a "store" look & feel. The add-on store is orgnanized around 4 categories: bindings, automation, UI, and others, with a separate search tab. Each category is splitted into subsections. The availability of these sections depend on the add-on services that are registered. The sections currently supports add-ons provided by the "karaf" & "marketplace" services, as provided by openhab/openhab-core#2405. Signed-off-by: Yannick Schaus <[email protected]>
@ghys As you have now created openhab/openhab-webui#1158 that is waiting on this PR - will you soon remove its "draft" status, so that we can do a final review? As discussed above, I don't really see any todo that still needs to be added to this initial PR. |
Yes I still have minor quirks with it but it will be ready for prime time soon. |
Signed-off-by: Yannick Schaus <[email protected]>
Signed-off-by: Yannick Schaus <[email protected]>
Signed-off-by: Yannick Schaus <[email protected]>
Signed-off-by: Yannick Schaus <[email protected]>
Signed-off-by: Yannick Schaus <[email protected]>
Change content types Javadoc fixes Implement getAddId(URL) for marketplace (unused) Bump Maven version Signed-off-by: Yannick Schaus <[email protected]>
Change content type for bundles Signed-off-by: Yannick Schaus <[email protected]>
SAT failed with
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @ghys, your updates look awesome! I think this is now a pretty good solution for the add-on bundles as we also store the content of UI widgets and rule templates locally in USER_DATA and don't fetch newer versions of them after an openHAB update - so all types are now nicely aligned in the way they are handled.
Once the SAT issue is fixed, we should be good to merge!
Signed-off-by: Yannick Schaus <[email protected]>
Btw, I just noticed that with the setting of the "location id", you can use the marketplace add-on ID to address the related bundle in the Karaf console, which is kind of cool I guess:
|
Unfortunately - contrary to the statement in the first post - the changes to |
You're right, shouldn't be a problem to add overloads with the old signatures. Mistakes happen. |
I'm already preparing a PR for that.
is "unknown" ok for the contentType? |
I guess, or even the empty string or May I ask where is the old API used outside OHC? Because if it's third-party and you're relying on the current generic add-on management UI, it will definitely break with openhab/openhab-webui#1158 (but I will keep the current codebase in the repo and if all goes well it would be relatively easy for forks to revert to it, with a compilation flag or similar). |
Yes, it's 3rd party. It's something similar to the marketplace (manager for installing 3rd party addons). Unfortunately it can't be made obsolete (which I would prefer) by the marketplace because the marketplace only supports .jar, neither complete features (which I would prefer) nor .kar. If it'll break anyway, there is no need to re-add it. I'll just provide another version for OH 3.2. |
This could be solved with another |
Hello @ghys I have a small question. 🙂 Was there a reason for not already adding the other add-on types (io/misc, persistence, transform, voice)?: Lines 110 to 115 in 0d5b2d6
|
no real reason, the goal was to restore the functionality of the Eclipse Marketplace. Also to avoid having to create empty Discourse categories. currently you can infer the type and content type from the Discourse category alone:
But this logic obviously has to change now. Maybe put all Java based add-ons in one category (rename Bindings to Bundles & Features or similar?) and use mandatory tags ("transformation" , "kar") to derive the type and content type. |
Is it possible to have mandatory tags with "one-of-list"? So we can force to set one of |
Yes the content type is a bit difficult to handle with the categories. It is also likely that there will be Automation (script engines) and UI add-on JARs or KARs. Those don't fit the Bindings category. |
Possibly, check out the screenshot in the first post about tag groups, there's an option that does that apparently. |
Not sure if you can make tags mandatory, but at least we can prevent invalid combinations (like kar and jar for the same addon) with the tag groups. Nice. Looks like an easy way. If we agree on that, I can make the changes in the kar handler PR. |
Depends on openhab/openhab-core#2405. This replaces the current section of the UI for managing add-ons with a completely new "store-like" interface. External data, like logos and detailed descriptions for distribution add-ons are fetched from well-known locations on the website or GitHub to provide a "store" look & feel. The add-on store is orgnanized around 4 categories: bindings, automation, UI, and others, with a separate search tab. Each category is splitted into subsections. The availability of these sections depend on the add-on services that are registered. The sections currently supports add-ons provided by the "karaf" & "marketplace" services, as provided by openhab/openhab-core#2405. Other add-on services should be supported as well. Signed-off-by: Yannick Schaus <[email protected]>
Related to: * openhab/openhab-core#2405 * openhab/openhab-core#2491 Signed-off-by: Wouter Born <[email protected]>
Related to: * openhab/openhab-core#2405 * openhab/openhab-core#2491 Signed-off-by: Wouter Born <[email protected]>
…2405) Signed-off-by: Yannick Schaus <[email protected]> GitOrigin-RevId: 2663a3f
This is an initial attempt at reimplementing an add-on service as an alternative to the distribution (
KarafAddonService
), similar to the (sooner or later) discontinued Eclipse IoT Marketplace from OH2.It is inspired and implements discussions in #2070, #2058, openhab/openhab-addons#8448
and borrows code and concepts from:
The community marketplace sources its content from a special category on Discourse (https://community.openhab.org/c/marketplace/69) which is still hidden to the general public in order to avoid confusion (but if you're in the
staff
group you may have seen it already). If you intend to review this PR and don't happen to have access to https://community.openhab.org/c/marketplace/69, I can give you access and provide you with an API key to put in Settings > Community Marketplace, ask on Discourse via PM.The add-on service is able to list and import these types of add-ons:
Non-breaking API changes have been made to
Addon
,AddonService
andAddonResource
to add more info and allow to select the add-on service when managing add-ons. Among other things:Map<String, Object>
(for instance, the number of views and likes as reported by Discourse).verifiedAuthor
field allows a service to mark the author as "trustworthy" - hardcoded to true in theKarafAddonService
(the author name being "openHAB") while the marketplace service currently marks every author as "untrustworthy".A follow-up overhaul of the add-on management UI would for instance put a "verified" check icon (similarly to social network platforms) next to the author's name and will display trustworthy add-ons prominently.
Untrusted .jar add-ons might also get a warning that they might not have been properly reviewed, so the user knows installing them is at their own risk.
The "verified" icon and lack of warning would hopefully incentivize developers to still prefer contributing Java-based add-ons like bindings to the openHAB distribution rather than publish them in the marketplace (or have a dual stable version in the distribution + development versions in the marketplace).
NOTE: while these UI changes are needed, the marketplace service is currently functional without them.
/rest/addons/{addonId}
will also include the entire original post indetailedDescription
, which will allow displaying the entire description in the UI, as well as pre-extracted links or code blocks, for instance:localhost:8080/rest/addons/marketplace:123574?serviceId=marketplace
TODO:
caching to avoid requesting the Discourse API at every request to
/rest/addons?serviceId=marketplace
(and allow to force a refresh by usingAddonService::refreshSource()
)plan for when there will be a lot of entries
filtering based on maturity level (managed with tags)
moderation, there are several options:
published
tag (there's a "tag group" feature in Discourse that allows to restrict certain tags on topics to people in a certain group); the topic will be visible on the forum but will not appear in the UI before a curator adds the tags:I'm not sure how it works since my test posts skipped that review phase but I suppose it would appear in https://community.openhab.org/review for reviewers like reports
"Curators" are currently restricted to "staff" meaning admins and moderators as appearing in https://community.openhab.org/about but it could be expanded to: