Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restored beta warn checks #7282

Closed
wants to merge 3 commits into from

Conversation

vfdev-5
Copy link
Collaborator

@vfdev-5 vfdev-5 commented Feb 17, 2023

Context:

  • 4774fe3 broke warning check
  • We can use pytest --noconftest and test is passing again, but I decided to get rid of pytest due to other possible side-effects.

torchvision.disable_beta_transforms_warning()

from common_utils import CUDA_NOT_AVAILABLE_MSG, IN_FBCODE, IN_OSS_CI, IN_RE_WORKER, OSS_CI_GPU_NO_CUDA_MSG
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This prevents warnings from v2 imported in common_utils

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

@vfdev-5 vfdev-5 marked this pull request as ready for review February 17, 2023 15:49
Copy link
Collaborator

@pmeier pmeier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vfdev-5
Copy link
Collaborator Author

vfdev-5 commented Feb 20, 2023

Flaky test here (Unit-tests on Linux CPU / tests (3.9) / linux-job (pull_request)):

2023-02-20T09:32:45.8999693Z �[36m�[1m=========================== short test summary info ============================�[0m
2023-02-20T09:32:45.9000296Z �[31mFAILED�[0m test/test_transforms_v2_functional.py::�[1mTestKernels::test_scripted_vs_eager[cpu-rotate_bounding_box-19]�[0m - AssertionError: Tensor-likes are not close!
2023-02-20T09:32:45.9000550Z 
2023-02-20T09:32:45.9000639Z Mismatched elements: 1 / 24 (4.2%)
2023-02-20T09:32:45.9000978Z Greatest absolute difference: 1.3532526992321436e-06 at index (1, 0, 0) (up to 1e-06 allowed)
2023-02-20T09:32:45.9001360Z Greatest relative difference: 1.4774978678428505e-05 at index (1, 0, 0) (up to 1e-06 allowed)
2023-02-20T09:32:45.9001529Z 
2023-02-20T09:32:45.9001607Z The failure occurred for item [0]
2023-02-20T09:32:45.9001844Z Failure happened for the following parameters:
2023-02-20T09:32:45.9001988Z 
2023-02-20T09:32:45.9002078Z (tensor([[[ 0.0000, 10.4270,  9.6026, 17.1515],
2023-02-20T09:32:45.9002272Z          [ 0.0000, 12.8995, 16.0000, 22.1969],
2023-02-20T09:32:45.9002449Z          [ 0.0000, 11.3209, 11.8092, 21.0413]],
2023-02-20T09:32:45.9002562Z 
2023-02-20T09:32:45.9002643Z         [[ 0.0916,  6.0138, 16.0000, 18.8870],
2023-02-20T09:32:45.9002829Z          [ 2.2459,  9.2190, 13.4401, 13.7892],
2023-02-20T09:32:45.9003039Z          [16.0000,  8.7480, 16.0000, 12.8995]]], dtype=torch.float64), (29, 16))
2023-02-20T09:32:45.9003262Z (tensor([[[ 0.0000, 10.4270,  9.6026, 17.1515],
2023-02-20T09:32:45.9003450Z          [ 0.0000, 12.8995, 16.0000, 22.1969],
2023-02-20T09:32:45.9003626Z          [ 0.0000, 11.3209, 11.8092, 21.0413]],
2023-02-20T09:32:45.9003818Z 
2023-02-20T09:32:45.9003900Z         [[ 0.0916,  6.0138, 16.0000, 18.8870],
2023-02-20T09:32:45.9004084Z          [ 2.2459,  9.2190, 13.4401, 13.7892],
2023-02-20T09:32:45.9004304Z          [16.0000,  8.7480, 16.0000, 12.8995]]], dtype=torch.float64), (29, 16))
2023-02-20T09:32:45.9004557Z format=BoundingBoxFormat.XYXY
2023-02-20T09:32:45.9004769Z spatial_size=(29, 16)
2023-02-20T09:32:45.9004965Z angle=-87
2023-02-20T09:32:45.9265486Z �[31m= �[31m�[1m1 failed�[0m, �[32m73886 passed�[0m, �[33m73524 skipped�[0m, �[33m1629 xfailed�[0m, �[33m4605 warnings�[0m�[31m in 1463.79s (0:24:23)�[0m�[31m =�[0m
2023-02-20T09:33:03.5461688Z ##[error]Process completed with exit code 1.
2023-02-20T09:33:03.5488532Z Prepare all required actions

@NicolasHug
Copy link
Member

Instead of adding more ad-hoc checks on a separate test suite, I'd prefer going for a more robust solution which would allow to test more scenarios, as the one proposed in #7269. I'll try to open a PR today

@vfdev-5
Copy link
Collaborator Author

vfdev-5 commented Feb 20, 2023

I do not think it worth adding new code like https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/_testing.py#L747 in torchvision just to run some files like python check_file.py inside a single pytest test call.

If you point is about using again pytest framework instead of manually coding with warnings.catch_warnings(record=True) as w:, it is possible and can remove 1-2 lines. The point of removing pytest was to ensure no mixing with contest.py etc

@NicolasHug
Copy link
Member

NicolasHug commented Feb 20, 2023

We need a lot more checks than those written in this PR. Each of these additional checks would require a separate script to properly test it without risking side-effects. This is untractable.

The testing util I want to introduce doesn't have to be as complicated as the one in scikit-learn, we can trim it down for it to look more like https://github.com/pytorch/hub/blob/a8c0f7ccfde4b8ea6cda812f5f61dd82bdeeda65/test_run_python_code.py#L35-L41

@NicolasHug
Copy link
Member

I followed-up with #7288, LMK if it's OK with you @vfdev-5

@pmeier
Copy link
Collaborator

pmeier commented Feb 21, 2023

Closing this since it is superseded by #7288 and the latter was merged.

@pmeier pmeier closed this Feb 21, 2023
@vfdev-5 vfdev-5 deleted the restore-beta-warn-checks branch February 21, 2023 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants