Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/devtools experiment #21

Merged
merged 86 commits into from
Feb 3, 2024
Merged

Feature/devtools experiment #21

merged 86 commits into from
Feb 3, 2024

Conversation

ms-lolo
Copy link
Collaborator

@ms-lolo ms-lolo commented Jan 11, 2024

  • starting in feb, all pipelines are required to specify their needed permissions, so i added ours
  • moved ci to a new set of 1es agents that run in the Immunomics R&D Infra sub for better isolation
  • switched to a new managed identity with limited permissions
  • redid devtools to remove all dependencies pointing into the component
  • removed immunodata dependencies from devtools
  • set up a new devtools flow with pipx
  • removed our use of docker images for building each component for now
  • started to experiment with devtools owning our ci logic in order to eventually have it generate ci pipelines
  • added component governance pipeline in ADO

some reasoning:
before this pr, we used the devtools component to install development packages into all our actual components. but this caused complexity with lots of unnecessary dependencies and made the packaging and releasing of the products difficult because of a "special" local dependency.

the new structure inverts the dependency so nothing depends on devtools, leaving the released product in a cleaner state without anything special about them. we use the new devtools component to create commands meant to help us keep this repo in a healthy state. for example, oneml-devtools generate-cg-manifest [path] generates a cgmanifest.json file in the component pointed to by the path argument. since this logic is not something we want to release as part of oneml packages, putting it in devtools means the other components have no relationship to it. i think of the devtools component as a place to put maintenance logic that is more complex than what we could do in small bash scripts.

Copy link

github-actions bot commented Jan 11, 2024

Test Results: oneml-pipelines

31 tests   31 ✅  0s ⏱️
 1 suites   0 💤
 1 files     0 ❌

Results for commit 578cdef.

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Jan 11, 2024

Test Results: oneml-habitats

0 tests   - 46   0 ✅  - 46   0s ⏱️ -4s
0 suites  -  1   0 💤 ± 0 
0 files    -  1   0 ❌ ± 0 

Results for commit b8dee7a. ± Comparison against base commit e77b52c.

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Jan 11, 2024

Test Results: oneml-processors

112 tests   112 ✅  4s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 578cdef.

♻️ This comment has been updated with latest results.

.gitignore Show resolved Hide resolved
- remove dependency to immunodata
- add click library for experimentation
- install oneml-pipelines so we can use services
- add setuptools because pycharm needs it
- register a oneml-devtools script with poetry
this method complicates the api quite a bit but i've never used it so i
would rather think about adding something when a use case comes up.
this allows this wheel to be installed on already-published versions of
oneml-pipelines. when this package is installed with pipx, it has to download
the dependencies from our ado registry.
this allows us to build some dev experience commands that we can share
across components.
@ms-lolo ms-lolo force-pushed the feature/devtools-experiment branch from 61111a9 to 7d8de28 Compare January 16, 2024 21:50
…ls-experiment

# Conflicts:
#	.azure-pipelines/ci/core.yaml
#	.azure-pipelines/ci/pr.yaml
#	.azure-pipelines/templates/jobs.application-ci.check-all.yaml
#	.azure-pipelines/templates/jobs.application-ci.docker.yaml
#	.azure-pipelines/templates/jobs.application-ci.wheel.yaml
#	.azure-pipelines/templates/jobs.deploy-app.yaml
#	.azure-pipelines/templates/jobs.deploy-jupyter-kernel.core.yaml
#	.azure-pipelines/templates/jobs.docs.yaml
#	.azure-pipelines/templates/jobs.list-experiment-configs.yaml
#	.azure-pipelines/templates/jobs.list-experiments.yaml
#	.github/workflows/ci.yaml
#	oneml-habitats/poetry.lock
#	oneml-habitats/pyproject.toml
.azure-pipelines/cg.yaml Outdated Show resolved Hide resolved
.github/workflows/main.yaml Outdated Show resolved Hide resolved
@ms-lolo
Copy link
Collaborator Author

ms-lolo commented Feb 2, 2024

change cg pipeline to fail on warnings

@ms-lolo ms-lolo merged commit 4a48a2d into main Feb 3, 2024
9 checks passed
@ms-lolo ms-lolo deleted the feature/devtools-experiment branch February 3, 2024 01:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants