Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor and new adapter components #44

Merged
merged 31 commits into from
Nov 21, 2024
Merged

Refactor and new adapter components #44

merged 31 commits into from
Nov 21, 2024

Conversation

SebastianScherer88
Copy link
Owner

@SebastianScherer88 SebastianScherer88 commented Nov 12, 2024 •
edited
Loading

Refactor and new Adapter components

  • Introducing cleaner directory and module structure for library and tests
  • Adding new Adapter components to make running pods outside of ArgoWorkflow context easier. Originally introduced when mistakenly thinking the torch ddp issues where due to the argo emissary context
  • Renaming decorators to as_... to better disambiguate them from their parent modules
  • Refactoring inline script runners and improving unit test coverage for those

@SebastianScherer88 SebastianScherer88 marked this pull request as draft November 12, 2024 18:08
@SebastianScherer88
started adding utility io functions to support moving argo workflow p…
1a2c938 
…arameters and artifacts in and out of the argo workflow context
@SebastianScherer88
started adding the custom torch script classes to help parametrize a …
271751b 
…more complex torch inline script runner to help build code snippets for both adapter and the main ddp component
@SebastianScherer88
wrapping adapter io utils in one class
2ba4f90
@SebastianScherer88
major restructuring of modules
ce2d029
@SebastianScherer88
finally fixed the broken tests. the build directory till had old outp…
a31f2bc 
…uts and mixed it with the new ones leading to a weird pip install that contained both old and new modules, leading to various different import paths and _pipeline_context instances. finally fixed and updated sdk make target to avoid it in the future
@SebastianScherer88
fixed import in test
f02441c
@SebastianScherer88
reconfiguring and renaming transformer pipeline to two nodes
7fa7af8
@SebastianScherer88
reverting name changes for component decorators
bcaf80b
@SebastianScherer88
moved the adapter io into the adapter module
2a95fcd
@SebastianScherer88
updated constants with adapter values
65f982f
@SebastianScherer88
minor clean up of base inline script runner
44c39e2
@SebastianScherer88
added component, decorator, script and inline script runner classes f…
d7da8d5 
…or adapter types
@SebastianScherer88
minor clean up of standard component
1faed3a
@SebastianScherer88
adding unit tests for adapter components. updated conftest
e7c1dea
@SebastianScherer88
converted the s3_prefix data to Artifact type bc Parameter are tied t…
37faa51 
…o the underlying function signature atm. added a load method to the IO class to allow populating the value attribute of an InputArtifact in particular. finished up and added working unit tests for both adapter components
@SebastianScherer88
cleaned up type hints in component decorators
c185744
@SebastianScherer88
started work on new ddp component
71c42c3
@SebastianScherer88
started work on new ddp component
ffa4cbb
@SebastianScherer88 SebastianScherer88 mentioned this pull request Nov 20, 2024
Merged
@SebastianScherer88
renamed component decorators to resolve import issues. readded torch …
b6b2b03 
…ddp component component in new refactored layout. refactored the base inline script runner to simplify torch ddp inline script runner. added scipy to test dependencies as the lightning integration test case failed on local.
@SebastianScherer88
restructured tests to mirror module directory structure
78c65f4
@SebastianScherer88
renaming torch_ddp decorator to as_torch_ddp and updating torch ddp i…
8b1dbaf 
…nline script runner.
@SebastianScherer88
fixed import in base inline script runner
9aaefda
@SebastianScherer88
added unit test for base inline script runner and torch ddp inline sc…
51360d5 
…ript runner
@SebastianScherer88
fixed bug in standard image name
adb7af3
@SebastianScherer88
fixed torch import handling
0b5af36
@SebastianScherer88
fixed import handling of torch dependencies again
8c142b7
@SebastianScherer88
renamed pipeline decorator to as_pipeline to be consistent with compo…
21aa5f4 
…nent decorators
@SebastianScherer88 SebastianScherer88 changed the title Re-designing the torch ddp component - fixing current broken implementation Refactor and new adapter components Nov 21, 2024
@SebastianScherer88 SebastianScherer88 marked this pull request as ready for review November 21, 2024 00:04
@SebastianScherer88 SebastianScherer88 merged commit 5a2c08f into master Nov 21, 2024
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@SebastianScherer88