pip and Azure Pipelines

[pradyunsg]

/cc @pypa/pip-committers @brcrista

#5785 has become a little long. Plus, it feels weird to me to have a discussion about general things on a PR. :)

[pradyunsg]

I've gone ahead and renamed the Pipelines to remove the words "pip CI" from it; since that's redundant wherever it would show up. I imagine unless GitHub uses the exact name of the CI jobs somewhere, this should not cause any issues.

@brcrista Do we need to specifically setup anything more for the GitHub - Azure DevOps mapping? https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/github?view=vsts#map-github-organizations-to-azure-devops

[pradyunsg]

I didn't see any reason to have https://dev.azure.com/pypa/pip/_releaseDefinition?definitionId=1&_a=definition-pipeline on currently, so I've gone ahead and switched off the Continuous Deployment automation trigger that causes it to attempt to push to test.pypi.org (which was failing anyway). I've not changed any other configuration there AFAICT.

I imagine it being possible to use this setup for eventually solving #2313.

[brcrista]

@pradyunsg unfortunately I've learned that that document is just describing a principle for keeping things organized, not any kind of automatic setup. We're still trying to figure out what connecting GitHub permissions to Azure DevOps permissions should look like.

The 1:1 mapping of GitHub repos to Azure DevOps projects would be nice and easy, but from what I've seen, a lot of projects don't fit in that box.

How many users do you need to add? Let me know if you need help.

[pradyunsg]

How many users do you need to add?

I don't think we'd add any more than @pypa/pip-committers -- which is currently 7 people and in an ideal world, we'd be able to have a few extra seats for more people in the future as well. Is there a limit to the size of a team associated with a project on Azure DevOps; for OSS projects?

[brcrista]

Ok cool, I was just wondering whether it would be faster to add them manually or script it.

Give users the "Stakeholder" access level instead of "Basic" -- those are "free users" in the sense that they don't count towards any kind of licensing limit. On https://dev.azure.com/pypa/_settings/users you can keep track of all this:

For Azure Pipelines, there's no difference between "Stakeholder" and "Basic" users. (More so for the other services: Repos, Boards, etc.)

If someone shows up as "Visual Studio Subscriber," they're also free.

[pradyunsg]

Oh, to the pypa organization, I imagine as more PyPA projects adopt Azure DevOps, we'll add more people. :P

OTOH, how does a user become a "Stakeholder"? I'm not one -- is there anything needed other than someone actually going in the UI and making me one?

[brcrista]

There's a dropdown when you add a user, I had just added you as Basic ... let me get back to you with the right instructions here. I think if you just click the "Invite" button it will default you to Basic

[pradyunsg]

Sure thanks! ^.^

[lorenpaulsen]

Oh, to the pypa organization, I imagine as more PyPA projects adopt Azure DevOps, we'll add more people. :P

OTOH, how does a user become a "Stakeholder"? I'm not one -- is there anything needed other than someone actually going in the UI and making me one?

In order to add a user as a stakeholder, they must be added through the Users tab in the Organization Settings, rather than using the invite button on the Project Summary Page: https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/add-organization-users?view=vsts&tabs=new-nav#add-users-to-your-organization

[brcrista]

Looks like @dstufft can do this for pypa as a "Project Collection Administrator." I don't have that level of permission. You can see that group at https://dev.azure.com/pypa/_settings/security?_a=members

[brcrista]

Ok, somewhat forget what I said -- you can just click the "Invite" button and everything should be fine.

Basic and Stakeholder are treated the same OSS Pipelines. ("Invite" will add users as Basic while there is capacity and then start adding them as Stakeholders.)

The reason you see all these licensing concepts at the org level is that public and private projects could exist in the same org. A private project would need to keep track of this stuff, but public gets mostly everything for free (i.e. with Stakeholder).

[techalchemy]

@pradyunsg I can add you also if you still need it, just ping me

@davidstaheli has been helping us out / keeping us current on this front and sometimes he appears out of thin air and tells me what to do

w/r/t 'more than one repo per user' there's inherent friction between organizations and CI because you'd need to map permissions from github to azure at a per-project level or something, since membership in the org doesn't imply that you should control CI for all of its projects, but you should be able to control it for your own project

[pradyunsg]

The MacOS job seems flaky.

[brcrista]

Link to the build history for convenience:
https://dev.azure.com/pypa/pip/_build?definitionId=7&_a=summary

Looks like it's failed a few times on test_force_reinstall_with_same_version_specifier, and only on the Python 2.7 job. The first occurrence I could find from master was on Jan 20: https://dev.azure.com/pypa/pip/_build/results?buildId=3928

Do you have any context on how the code under test works? Seems suspicious that test_force_reinstall_with_no_version_specifier isn't failing as well.

[pradyunsg]

Thanks a lot for investigating this @brcrista! I'd mostly just noted this here since I didn't want to forget that this had to be addressed at some point.

I'm sorry, but I don't have the bandwidth to look into this in further detail at this time.

[cjerdonek]

Looks like it's failed a few times on test_force_reinstall_with_same_version_specifier(), and only on the Python 2.7 job.

This is what the intermittent failure looks like:

assert result2.files_updated, 'force-reinstall failed'
AssertionError: force-reinstall failed
assert {}
 +  where {} = <tests.lib.TestPipResult object at 0x10913cb90>.files_updated

My guess is that this is an issue with the scripttest library when running on Mac, specifically with how equality is computed with the files_updated dict:

https://github.com/pypa/scripttest/blob/01c7db96ff01de034bd7781a4ed28d74eb716d32/scripttest.py#L391-L394

You can even see that someone filed a similar issue four years ago in pip's tracker: #2580

Someone should probably file an issue in scripttest's tracker.

[pradyunsg]

Another thing that I don't have time to investigate right now.

#6249 has 26 "checks" from Azure DevOps. Does anyone know why Azure DevOps is showing step-wise information instead of the overall "Windows" "Linux" "MacOS" jobs we had until recently?

[pradyunsg]

26 "checks" from Azure DevOps. Does anyone know why Azure DevOps is showing step-wise information instead of the overall "Windows" "Linux" "MacOS" jobs we had until recently?

Pinging @zooba @brcrista @techalchemy @gaborbernat for help.

Could someone point me to how we could consolidate these checks to show up as just 3 on the GitHub side?

[brcrista]

@pradyunsg to be clear, Pipelines is posting a check for each job, and then a check for each pipeline that acts as an aggregate of all its jobs. Right now there's no option to toggle that.

The reason you used to get three is because we changed the behavior a few months back. Some people I've talked to want per-job checks and some people don't, so I've passed on the feedback that an option may be good here.

[pradyunsg]

Great! Thanks @brcrista!

[pradyunsg]

Closing this out, since we've moved to GitHub Actions now.