Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] HeapAttackIT testLookupExplosionBigStringManyMatches failing #121465

Closed
elasticsearchmachine opened this issue Jan 31, 2025 · 7 comments · Fixed by #121565 or #121714
Closed

[CI] HeapAttackIT testLookupExplosionBigStringManyMatches failing #121465

elasticsearchmachine opened this issue Jan 31, 2025 · 7 comments · Fixed by #121565 or #121714
Assignees
@nik9000
Labels
:Analytics/ES|QL AKA ESQL low-risk An open issue or test failure that is a low risk to future releases Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Jan 31, 2025 •
edited
Loading

Build Scans:

Reproduction Line:

./gradlew ":test:external-modules:test-esql-heap-attack:javaRestTest" --tests "org.elasticsearch.xpack.esql.heap_attack.HeapAttackIT.testLookupExplosionBigStringManyMatches" -Dtests.seed=D59D0CC178364FFA -Dtests.locale=gl-Latn-ES -Dtests.timezone=Greenwich -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

junit.framework.AssertionFailedError: Expected exception ResponseException but no exception was thrown

Issue Reasons:

  • [8.x] 12 failures in test testLookupExplosionBigStringManyMatches (1.9% fail rate in 642 executions)
  • [8.x] 5 failures in step part1 (6.2% fail rate in 81 executions)
  • [8.x] 2 failures in step amazonlinux-2_platform-support-aws (12.5% fail rate in 16 executions)
  • [8.x] 5 failures in pipeline elasticsearch-intake (6.2% fail rate in 81 executions)
  • [8.x] 6 failures in pipeline elasticsearch-periodic-platform-support (37.5% fail rate in 16 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :Analytics/ES|QL AKA ESQL >test-failure Triaged test failures from CI labels Jan 31, 2025
elasticsearchmachine added a commit that referenced this issue Jan 31, 2025
@elasticsearchmachine
Mute org.elasticsearch.xpack.esql.heap_attack.HeapAttackIT testLookup…
09e1426 
…ExplosionBigStringManyMatches #121465
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 2 failures in test testLookupExplosionBigStringManyMatches (8.7% fail rate in 23 executions)

Build Scans:

@elasticsearchmachine elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) needs:risk Requires assignment of a risk label (low, medium, blocker) labels Jan 31, 2025
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-analytical-engine (Team:Analytics)

@alex-spies
Copy link
Contributor

@nik9000 , could you please take a look at this one? Maybe this test needs further tuning, in the topmost build scan I found:

[2025-02-03T17:17:09,206][ERROR][o.e.x.e.h.HeapAttackIT   ] [testLookupExplosionBigStringManyMatches] should have failed but got {took=2042, is_partial=false, columns=[{name=COUNT(string), type=long}], values=[[500]]}

@alex-spies alex-spies added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Feb 3, 2025
@nik9000 nik9000 mentioned this issue Feb 3, 2025
Merged
nik9000 added a commit to nik9000/elasticsearch that referenced this issue Feb 3, 2025
@nik9000
ESQL: Reenable heap attacks (elastic#121565)
cae0c97 
Reenables some heap attack tests, bumping their memory requirements to
try and force a failure on all CI machines. Previously some CI machines
weren't failing, invalidating the test on those machines.

Close elastic#121481 Close elastic#121465
@nik9000 nik9000 mentioned this issue Feb 3, 2025
Merged
nik9000 added a commit to nik9000/elasticsearch that referenced this issue Feb 3, 2025
@nik9000
ESQL: Reenable heap attacks (elastic#121565)
dcf9deb 
Reenables some heap attack tests, bumping their memory requirements to
try and force a failure on all CI machines. Previously some CI machines
weren't failing, invalidating the test on those machines.

Close elastic#121481 Close elastic#121465
@nik9000 nik9000 mentioned this issue Feb 3, 2025
Merged
nik9000 added a commit to nik9000/elasticsearch that referenced this issue Feb 3, 2025
@nik9000
ESQL: Reenable heap attacks (elastic#121565)
a8b9396 
Reenables some heap attack tests, bumping their memory requirements to
try and force a failure on all CI machines. Previously some CI machines
weren't failing, invalidating the test on those machines.

Close elastic#121481 Close elastic#121465
@nik9000 nik9000 mentioned this issue Feb 3, 2025
Merged
elasticsearchmachine added a commit that referenced this issue Feb 3, 2025
@elasticsearchmachine
Mute org.elasticsearch.xpack.esql.heap_attack.HeapAttackIT testLookup…
c0dcc9f 
…ExplosionBigStringManyMatches #121465
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch 8.x

Mute Reasons:

  • [8.x] 6 failures in test testLookupExplosionBigStringManyMatches (1.8% fail rate in 328 executions)
  • [8.x] 2 failures in step amazonlinux-2_platform-support-aws (20.0% fail rate in 10 executions)
  • [8.x] 4 failures in pipeline elasticsearch-periodic-platform-support (40.0% fail rate in 10 executions)

Build Scans:

alex-spies pushed a commit that referenced this issue Feb 4, 2025
@nik9000
ESQL: Reenable heap attacks (#121565) (#121576)
10cf628 
Reenables some heap attack tests, bumping their memory requirements to
try and force a failure on all CI machines. Previously some CI machines
weren't failing, invalidating the test on those machines.

Close #121481 Close #121465
elasticsearchmachine pushed a commit that referenced this issue Feb 4, 2025
@nik9000
ESQL: Reenable heap attacks (#121565) (#121573)
8f79947 
Reenables some heap attack tests, bumping their memory requirements to
try and force a failure on all CI machines. Previously some CI machines
weren't failing, invalidating the test on those machines.

Close #121481 Close #121465
elasticsearchmachine pushed a commit that referenced this issue Feb 4, 2025
@nik9000 @alex-spies
ESQL: Reenable heap attacks (#121565) (#121575)
410bb4f 
* ESQL: Reenable heap attacks (#121565)

Reenables some heap attack tests, bumping their memory requirements to
try and force a failure on all CI machines. Previously some CI machines
weren't failing, invalidating the test on those machines.

Close #121481 Close #121465

* Unmute

---------

Co-authored-by: Alexander Spies <[email protected]>
rjernst added a commit that referenced this issue Feb 4, 2025
@rjernst
re-mute HeapAttackIT.testLookupExplosionBigStringManyMatches
04c04cc 
relates #121465
@rjernst
Copy link
Member

rjernst commented Feb 4, 2025

I've re-muted this test because it is still failing, causing PR failures. See https://gradle-enterprise.elastic.co/s/3kezzxetjkumw/console-log/task/:test:external-modules:test-esql-heap-attack:javaRestTest?anchor=15&page=1

fzowl pushed a commit to voyage-ai/elasticsearch that referenced this issue Feb 4, 2025
@nik9000 @fzowl
ESQL: Reenable heap attacks (elastic#121565)
c045d82 
Reenables some heap attack tests, bumping their memory requirements to
try and force a failure on all CI machines. Previously some CI machines
weren't failing, invalidating the test on those machines.

Close elastic#121481 Close elastic#121465
fzowl pushed a commit to voyage-ai/elasticsearch that referenced this issue Feb 4, 2025
@rjernst @fzowl
re-mute HeapAttackIT.testLookupExplosionBigStringManyMatches
baa7472 
relates elastic#121465
@nik9000
Copy link
Member

nik9000 commented Feb 4, 2025

OK. I'm going to make the test scale it's memory usage higher and higher until it breaks. It should break in the first round, but it looks like some CI nodes are... fun! And if that still doesn't break I'll be quite surprised.

nik9000 added a commit to nik9000/elasticsearch that referenced this issue Feb 4, 2025
@nik9000
ESQL: Grow heap attack tests
f19193d 
This causes the ESQL heap attack tests to grow their memory usage if
they first don't cause a circuit breaking exception. It just tries again
with more data. That's slow, but it should stop this from failing quite
as much. And it'll give us even more information about failures.

Closes elastic#121465
@nik9000 nik9000 mentioned this issue Feb 4, 2025
Merged
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch 8.18

Mute Reasons:

  • [8.18] 9 failures in test testLookupExplosionBigStringManyMatches (1.8% fail rate in 506 executions)
  • [8.18] 2 failures in pipeline elasticsearch-periodic (14.3% fail rate in 14 executions)
  • [8.18] 4 failures in pipeline elasticsearch-periodic-platform-support (28.6% fail rate in 14 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Feb 5, 2025
@elasticsearchmachine
Mute org.elasticsearch.xpack.esql.heap_attack.HeapAttackIT testLookup…
09a7665 
…ExplosionBigStringManyMatches #121465
@nik9000 nik9000 closed this as completed in 85da28d Feb 5, 2025
nik9000 added a commit to nik9000/elasticsearch that referenced this issue Feb 5, 2025
@nik9000
ESQL: Grow heap attack tests (elastic#121714)
04cfc0b 
This causes the ESQL heap attack tests to grow their memory usage if
they first don't cause a circuit breaking exception. It just tries again
with more data. That's slow, but it should stop this from failing quite
as much. And it'll give us even more information about failures.

Closes elastic#121465
@nik9000 nik9000 mentioned this issue Feb 5, 2025
Merged
nik9000 added a commit to nik9000/elasticsearch that referenced this issue Feb 5, 2025
@nik9000
ESQL: Grow heap attack tests (elastic#121714)
ef91437 
This causes the ESQL heap attack tests to grow their memory usage if
they first don't cause a circuit breaking exception. It just tries again
with more data. That's slow, but it should stop this from failing quite
as much. And it'll give us even more information about failures.

Closes elastic#121465
@nik9000 nik9000 mentioned this issue Feb 5, 2025
Merged
nik9000 added a commit to nik9000/elasticsearch that referenced this issue Feb 5, 2025
@nik9000
ESQL: Grow heap attack tests (elastic#121714)
cb87e97 
This causes the ESQL heap attack tests to grow their memory usage if
they first don't cause a circuit breaking exception. It just tries again
with more data. That's slow, but it should stop this from failing quite
as much. And it'll give us even more information about failures.

Closes elastic#121465
@nik9000 nik9000 mentioned this issue Feb 5, 2025
Merged
nik9000 added a commit that referenced this issue Feb 5, 2025
@nik9000
ESQL: Grow heap attack tests (#121714) (#121830)
45f6c66 
This causes the ESQL heap attack tests to grow their memory usage if
they first don't cause a circuit breaking exception. It just tries again
with more data. That's slow, but it should stop this from failing quite
as much. And it'll give us even more information about failures.

Closes #121465
elasticsearchmachine pushed a commit that referenced this issue Feb 5, 2025
@nik9000
ESQL: Grow heap attack tests (#121714) (#121832)
fdadf2d 
This causes the ESQL heap attack tests to grow their memory usage if
they first don't cause a circuit breaking exception. It just tries again
with more data. That's slow, but it should stop this from failing quite
as much. And it'll give us even more information about failures.

Closes #121465
nik9000 added a commit that referenced this issue Feb 6, 2025
@nik9000
ESQL: Grow heap attack tests (#121714) (#121831)
ad92edf 
This causes the ESQL heap attack tests to grow their memory usage if
they first don't cause a circuit breaking exception. It just tries again
with more data. That's slow, but it should stop this from failing quite
as much. And it'll give us even more information about failures.

Closes #121465
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nik9000 nik9000

Labels
:Analytics/ES|QL AKA ESQL low-risk An open issue or test failure that is a low risk to future releases Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
4 participants
@nik9000 @rjernst @elasticsearchmachine @alex-spies