Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can the official provide an interface(web api) to query the latest package information #1646

Open
cheapw opened this issue Jun 6, 2020 · 21 comments
Labels
Issue-Feature Complex enough to require an in depth planning process and actual budgeted, scheduled work.

Comments

@cheapw
Copy link

cheapw commented Jun 6, 2020

Can the official provide an interface(web api) to query the latest package information, then we can use the package info to build windows desktop Gui app to manager our software base on winget commands, that may be user friendly. If we could get the available package info easily, we can do a lot of amazing things.

@cheapw cheapw added the Issue-Feature Complex enough to require an in depth planning process and actual budgeted, scheduled work. label Jun 6, 2020
@ghost ghost added the Needs: Triage label Jun 6, 2020
@latere-a-latere
Copy link
Contributor

Something like this? microsoft/winget-cli#118

@SoftCreatR
Copy link
Contributor

SoftCreatR commented Jun 6, 2020

It's pretty easy. I'm doing this on https://www.winget.it for example. All you have to do is to query the GitHub API twice:

  1. https://api.github.com/repos/microsoft/winget-pkgs/git/trees/master

Here, you need to fetch the SHA hash for the path manifests.

  1. https://api.github.com/repos/microsoft/winget-pkgs/git/trees/XXXXXX?recursive=1

Replace XXXXXX with the SHA hash you've got from the 1st request.

--

That's all. You can now extract publisher, name and version from all paths of type blob.

@SoftCreatR
Copy link
Contributor

Alternatively (but not guaranteed to exist forever) you can use our cached package list, which is being refreshed every few minutes: https://www.winget.it/packages.json

@cheapw
Copy link
Author

cheapw commented Jun 7, 2020

Something like this? microsoft/winget-cli#118

Something like this? microsoft/winget-cli#118

Yes, that's a good idea

@SoftCreatR
Copy link
Contributor

@cheapw Well, do you really want to wait until there's something from M$? I mean... The approach explained above should already fit your needs.

@cheapw
Copy link
Author

cheapw commented Jun 7, 2020

Alternatively (but not guaranteed to exist forever) you can use our cached package list, which is being refreshed every few minutes: https://www.winget.it/packages.json

It's pretty easy. I'm doing this on https://www.winget.it for example. All you have to do is to query the GitHub API twice:

  1. https://api.github.com/repos/microsoft/winget-pkgs/git/trees/master

Here, you need to fetch the SHA hash for the path manifests.

  1. https://api.github.com/repos/microsoft/winget-pkgs/git/trees/XXXXXX?recursive=1

Replace XXXXXX with the SHA hash you've got from the 1st request.

--

That's all. You can now extract publisher, name and version from all paths of type blob.

Thanks! But query the github api directly may come across GitHub API limit with the increasing number of api request. It may need some backend cache mechanism, just like thist url (https://www.winget.it/packages.json)

@cheapw
Copy link
Author

cheapw commented Jun 7, 2020

@cheapw Well, do you really want to wait until there's something from M$? I mean... The approach explained above should already fit your needs.

No, I just want to write a personal app base on the winget-cli, not so urgent, I will keep an eye on the progress of this project, I will glad to see any new features and progress.

@SoftCreatR
Copy link
Contributor

It may need some backend cache mechanism, just like thist url (https://www.winget.it/packages.json)

The API is limited to 5k requests per hour, so generating/hosting your own (prepared) JSON might be a good idea.

@cheapw
Copy link
Author

cheapw commented Jun 7, 2020

The API is limited to 5k requests per hour, so generating/hosting your own (prepared) JSON might be a good idea.

Thanks, I will give it a try !

@SoftCreatR
Copy link
Contributor

As shown on https://www.winget.it, performing these two API requests (and cache their response, at least for 5 minutes) already provides nearly all informations you need. The only thing that is missing is the manifest data itself. However, (when needed) i would request them (without using the GH API) on demand and cache them too.

@cheapw
Copy link
Author

cheapw commented Jun 7, 2020

Yes, we can just use app's Id to install the app, and the version Field is to judge whether a new version is available based on the already cached version.

@SoftCreatR
Copy link
Contributor

FYI: I've rewritten the process that creates the package.json. The file now contains all manifest contents for maximum flexibility. Just in case, that anyone wants to use it: https://www.winget.it/packages.json

@SoftCreatR
Copy link
Contributor

SoftCreatR commented Jun 13, 2020

Here's another (more official) way: Download the signed MSIX package from the source server: https://winget.azureedge.net/cache/source.msix

This file contains an SQLite database, located in source.msix/Public/index.db. This SQLite database contains all required informations about manifests. It will be rebuilt upon merges in this repository, so whenever a change happens, this file should cover it.

Then, instead of downloading single manifest files using the GitHub API, download it from the source server, e.g. https://winget.azureedge.net/cache/manifests/AnsgarBecker/HeidiSQL/eca9-11.0.0.5919.yaml

To make sure, that you always use the latest, cached version and to prevent downloading the same database over and over again, perform a HEAD request against the msix file linked above and save either the Etag, or the Last-Modified timestamp from the response header.

Good luck :)

@denelon denelon added this to the Packages Backlog milestone Jun 16, 2020
@denelon
Copy link
Contributor

denelon commented Jun 16, 2020

When we finish the REST API work, we do expect to have an instance available with all of the packages from this repository.

@dominikgeimer
Copy link

When we finish the REST API work, we do expect to have an instance available with all of the packages from this repository.

Any update about the status?

@denelon
Copy link
Contributor

denelon commented Jan 6, 2021

We are building the REST API, and will provide reference documentation. Until the manifest changes for 1.0 are implemented, we will hold off on building a RESTful instance of the community package repository/source.

@ghost ghost added the No-Recent-Activity No activity has occurred on this work item for seven days. label Jan 13, 2021
@ghost

This comment has been minimized.

@denelon denelon removed the No-Recent-Activity No activity has occurred on this work item for seven days. label May 21, 2021
@yoshiask
Copy link

We are building the REST API, and will provide reference documentation. Until the manifest changes for 1.0 are implemented, we will hold off on building a RESTful instance of the community package repository/source.

It's been quite a while, are there any updates on this?

@denelon
Copy link
Contributor

denelon commented Jun 15, 2022

You can follow along at https://github.com/microsoft/winget-cli-restsource to see changes being made. We are standing up the infrastructure. Next steps include taking the output from the Windows Package Manager Community Repository here and passing merged manifests to the REST source.

@Trenly
Copy link
Contributor

Trenly commented Nov 5, 2022

@denelon - Can the team provide an update on the progress here?

@denelon
Copy link
Contributor

denelon commented Nov 14, 2022

We've got an instance up and running. We're working through some of the publishing challenges. We're not ready to offer any SLA guarantees so we've decided not to "officially" share the URL until the decision regarding support has been made. There has also been some discussion around the differences in the UI/UX with the WinGet client when hitting a REST source. We've also been discussing what it would look like to have the PreIndexed package and the REST source able to logically back each other up in the event of an outage (similar to what happened with the CDN problems we've recently encountered).

@denelon denelon removed this from the Backlog-Pipelines milestone Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Issue-Feature Complex enough to require an in depth planning process and actual budgeted, scheduled work.
Projects
None yet
Development

No branches or pull requests

7 participants