Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve test performance #1953

Merged
merged 39 commits into from
Oct 21, 2024
Merged

Improve test performance #1953

merged 39 commits into from
Oct 21, 2024

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Oct 17, 2024 •
edited
Loading

Description

  • Improve milvus startup time bypassing milvus' own wait_for_started method which was causing an intermittent delay up to 40s
  • Consolidate tests test_kafka_source_commit and test_kafka_source_batch_pipe these two tests were 80% similar
  • Remove the num_records=100 parameter from test_kafka_source_batch_pipe
  • Reduce parameters for test test_http_server.py::test_simple_request lowering the total number of test combinations from 865 down to 145.
  • Update rag pipeline tests to share a common milvus collection, lowering test time from 45s down to 25s

Closes #1948

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

dagardner-nv and others added 30 commits October 15, 2024 14:32
@dagardner-nv
Mark slow tests as slow
4ebb4ff
@dagardner-nv
Replace deprecated config modifications
5573454 
ref:  https://docs.pydantic.dev/2.9/concepts/dataclasses/#dataclass-config
@dagardner-nv
Tell pytest not to look in the tests/_utils dir, avoids warnings abou…
5ee378e 
…t the TestDirectories class looking like a test class
@dagardner-nv
HuggingFaceEmbeddings is now in langchain_community
8000ec3
@dagardner-nv
Remove print
c98110a
@dagardner-nv
PyPDFLoader moved to langchain_community
a3e71b3
@dagardner-nv
Remove deprecated infer_datetime_format argument per https://pandas.p…
be4597a 
…ydata.org/pdeps/0004-consistent-to-datetime-parsing.html
@dagardner-nv
Specify weights_only=False to avoid a FutureWarning
10ad08a
@dagardner-nv
Remove deprecated extra = "forbid"
366f4ff
@dagardner-nv
validator has been replaced with field_validator
4915dac
@dagardner-nv
Fix redefined function name
552eceb
@dagardner-nv
Avoid warning about cudf.read_json wanting StringIO and now strings
bf95dbb
@dagardner-nv
Bump unittest timeout to 90 minutes
697e1f5
@dagardner-nv
Merge branch 'branch-24.10' into david-2410-warnings
51ec950
@dagardner-nv
Move import into fixture to abvoid invoking during test discovery
b178bbc
@dagardner-nv
Remove redundant langchain fixture impl
a13e25b
@dagardner-nv
Restructure the rwd_conf fixture such that the file is read from disk…
3ee6182 
… only once
@dagardner-nv
morpheus_sys_path should be session scope
fe7f2a0
@dagardner-nv
Consolidate dask and mlflow fixtures
ea6b154
@dagardner-nv
Consolidate langchain fixtures, fix conda command
1a1c194
@dagardner-nv
redefine fixtures as session fixtures
f591ea3
@dagardner-nv
Remove unused imports
b8301fd
@dagardner-nv
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
d3e70b9 
…avid-2410-warnings
@dagardner-nv
Revert "Remove deprecated extra = "forbid""
b664298 
This reverts commit 366f4ff.
@dagardner-nv
Use ConfigDict(extra='forbid') to replace nested config class
cad8c46
@dagardner-nv
Remove uneeded pylint rule disables
657438d
@dagardner-nv
Use model_config rather than the decorator
3dea6d0
@dagardner-nv
Drop the num_records=100 parameter
8583cb0 
consolidate the test_kafka_source_commit and test_kafka_source_batch_pipe tests as they were nearly identical
Improve test time from ~4min to 2:39
@dagardner-nv
Give the input and output topics the same postfix
a179cb2
@dagardner-nv
Consolidate the milvus_service fixture
6ff03fb
@dagardner-nv dagardner-nv added bug Something isn't working non-breaking Non-breaking change skip-ci Optionally Skip CI for this PR labels Oct 17, 2024
@dagardner-nv dagardner-nv self-assigned this Oct 17, 2024
@dagardner-nv dagardner-nv requested a review from a team as a code owner October 17, 2024 15:14
@dagardner-nv dagardner-nv marked this pull request as draft October 17, 2024 15:18
@dagardner-nv dagardner-nv removed the skip-ci Optionally Skip CI for this PR label Oct 17, 2024
@dagardner-nv dagardner-nv marked this pull request as ready for review October 17, 2024 22:50
@dagardner-nv
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 981314e into nv-morpheus:branch-24.10 Oct 21, 2024
11 of 12 checks passed
@dagardner-nv dagardner-nv deleted the david-slow-tests-1948 branch October 21, 2024 22:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

Assignees

@dagardner-nv dagardner-nv

Labels
bug Something isn't working non-breaking Non-breaking change
Projects
Morpheus Boards
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG]: CI Test stage is hitting the one hour limit
2 participants
@dagardner-nv @mdemoret-nv