[SDK] Get the correct TrainJob components using get_job() API

[andreyvelich]

What you would like to be added?

As we discussed, currently get_job() API can return multiple Pods for every TrainJob component, like initializer or trainer-node-0: #2324 (comment). That can happen when Pods are re-created based on Batch/Job restart policies.
Therefore, users can see unexpected logs while using the Kubeflow Training SDK.

We should improve this API to show the correct TrainJob components to users.
For example, when we list all of the Pods, we can select the most recently created Pod with the same role (e.g. dataset-initializer).

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreyvelich]

/remove-lifecycle stale
/good-first-issue

[google-oss-prow]

@andreyvelich:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/remove-lifecycle stale
/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Beihao-Zhou]

/assign

Hiiii! I'm new to the kubeflow community but I'm very interested in contributing to the community by starting with the good first issue! Let me know if there is anything specific I should be aware of before jumping into the implementation.

[andreyvelich]

Thank you for your interest @Beihao-Zhou!
Please try to follow the PR threads that mentioned in this PR, and explore the Kubeflow Trainer SDK.

[Beihao-Zhou]

Hi @andreyvelich, I'm currently trying to test the Python SDK locally and ran into a few setup issues. Could you share how you typically configure your environment for testing the SDK? Thanks in advance!!

[tenzen-y]

Hi @andreyvelich, I'm currently trying to test the Python SDK locally and ran into a few setup issues. Could you share how you typically configure your environment for testing the SDK? Thanks in advance!!

You can check this: https://www.kubeflow.org/docs/components/trainer/getting-started/

[Beihao-Zhou]

Hi @andreyvelich, I'm currently trying to test the Python SDK locally and ran into a few setup issues. Could you share how you typically configure your environment for testing the SDK? Thanks in advance!!

You can check this: https://www.kubeflow.org/docs/components/trainer/getting-started/

Thanks!! Just to confirm, so the usual way for now is running docker containers and run make <different_tests> right?

[Garvit-77]

Hello @Beihao-Zhou are you still working this issue ?

[Beihao-Zhou]

@Garvit-77 Sorry about the delay. I was busy these days and wouldn't be able to ship it recently. Feel free to work on it if you want!

[Garvit-77]

@Beihao-Zhou Surely I would be raising my pr shortly
/assign