Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"the object has been modified" errors seen during allocation #3553

Open
Kalaiselvi84 opened this issue Dec 19, 2023 · 4 comments
Open

"the object has been modified" errors seen during allocation #3553

Kalaiselvi84 opened this issue Dec 19, 2023 · 4 comments
Labels
awaiting-maintainer Block issues from being stale/obsolete/closed kind/bug These are bugs.

Comments

@Kalaiselvi84
Copy link
Contributor

What happened:

The failed e2e test: TestGameServerAllocationDuringMultipleAllocationClients

What you expected to happen:

Pass all the time

How to reproduce it (as minimally and precisely as possible):

It fails sometimes in the CI

Anything else we need to know?:

VERBOSE:     gameserverallocation_test.go:1417: 
VERBOSE:         	Error Trace:	/go/src/agones.dev/agones/test/e2e/gameserverallocation_test.go:1417
VERBOSE:         	Error:      	Not equal: 
VERBOSE:         	            	expected: 100
VERBOSE:         	            	actual  : 99
VERBOSE:         	Test:       	TestGameServerAllocationDuringMultipleAllocationClients
VERBOSE: --- FAIL: TestGameServerAllocationDuringMultipleAllocationClients (38.74s)

Environment:
@Kalaiselvi84 Kalaiselvi84 added the kind/bug These are bugs. label Dec 19, 2023
@markmandel markmandel mentioned this issue Jan 10, 2024
Merged
@markmandel markmandel mentioned this issue Feb 29, 2024
Merged
@Kalaiselvi84 Kalaiselvi84 mentioned this issue Mar 15, 2024
Merged
@Kalaiselvi84 Kalaiselvi84 mentioned this issue Mar 26, 2024
Merged
@markmandel
Copy link
Collaborator

More and more!

https://console.cloud.google.com/cloud-build/builds/6322a904-b37c-4bcc-8e04-5e29a8315c21?project=258182270954

This is killing me.

@markmandel markmandel mentioned this issue Apr 8, 2024
Merged
@zmerlynn zmerlynn mentioned this issue Apr 10, 2024
Merged
@zmerlynn
Copy link
Collaborator

https://console.cloud.google.com/cloud-build/builds/6322a904-b37c-4bcc-8e04-5e29a8315c21?project=258182270954

could not completed gsa1 allocation : an error on the server ("error updating allocated gameserver: Operation cannot be fulfilled on gameservers.agones.dev \"preferred-m8hsh-mzrd8-g682d\": the object has been modified; please apply your changes to the latest version and try again

This is the same conflict @gongmax sees in perf tests, and I assume it's because two allocators try to allocate the same GameServer but fail (and it's somehow not retried by the allocator).

@zmerlynn zmerlynn changed the title Flake: TestGameServerAllocationDuringMultipleAllocationClients "the object has been modified" errors seen during allocation Apr 10, 2024
@zmerlynn
Copy link
Collaborator

This is also mentioned in the load test README:

agones/test/load/allocation/README.md

Lines 62 to 64 in c1320ee

(failed(client=3,allocation=43): rpc error: code = Unknown desc = error updating allocated gameserver: Operation cannot be fulfilled on gameservers.agones.dev "simple-game-server-mlljx-g9crp": the object has been modified; please apply your changes to the latest version and try again
(failed(client=2,allocation=47): rpc error: code = Unknown desc = error updating allocated gameserver: Operation cannot be fulfilled on gameservers.agones.dev "simple-game-server-mlljx-rxflv": the object has been modified; please apply your changes to the latest version and try again
(failed(client=7,allocation=45): rpc error: code = Unknown desc = error updating allocated gameserver: Operation cannot be fulfilled on gameservers.agones.dev "simple-game-server-mlljx-x4khw": the object has been modified; please apply your changes to the latest version and try again

zmerlynn pushed a commit that referenced this issue Apr 11, 2024
@Kalaiselvi84
GameServerAllocationDuringMultipleAllocationClients: Change expected …
65a997e 
…allocation count to 98

The allocator is generating conflicts: see #3553
zmerlynn added a commit to zmerlynn/agones that referenced this issue May 16, 2024
@zmerlynn
TestGameServerAllocationDuringMultipleAllocationClients: Readdress flake
8034463 
The actual bug is here:
  https://github.com/googleforgames/agones/blob/2023077aff4125b9d9822c5d20cd773d13138385/test/e2e/gameserverallocation_test.go#L1423

When we tried to accept errors in googleforgames#3750, we forgot to
change this t.Errorf to a simple log.

Along the way, some cleanup.

Towards googleforgames#3553
zmerlynn added a commit to zmerlynn/agones that referenced this issue May 16, 2024
@zmerlynn
TestGameServerAllocationDuringMultipleAllocationClients: Readdress flake
f688a5c 
The actual bug is here:
  https://github.com/googleforgames/agones/blob/2023077aff4125b9d9822c5d20cd773d13138385/test/e2e/gameserverallocation_test.go#L1423

When we tried to accept errors in googleforgames#3750, we forgot to
change this t.Errorf to a simple log.

Along the way:

* change the timing of the scale-down so it happens more probably in
the middle of the allocations, which was the intent. By my guess, 3s
might be past all allocations. Also add a log to see if I'm right.

* minor cleanup

Towards googleforgames#3553
@zmerlynn zmerlynn mentioned this issue May 16, 2024
Merged
zmerlynn added a commit to zmerlynn/agones that referenced this issue May 16, 2024
@zmerlynn
TestGameServerAllocationDuringMultipleAllocationClients: Readdress flake
31603f6 
The actual bug is here:
  https://github.com/googleforgames/agones/blob/2023077aff4125b9d9822c5d20cd773d13138385/test/e2e/gameserverallocation_test.go#L1423

When we tried to accept errors in googleforgames#3750, we forgot to
change this t.Errorf to a simple log.

Along the way:

* change the timing of the scale-down so it happens more probably in
the middle of the allocations, which was the intent. By my guess, 3s
might be past all allocations. Also add a log to see if I'm right.

* minor cleanup

Towards googleforgames#3553
markmandel pushed a commit that referenced this issue May 17, 2024
@zmerlynn
TestGameServerAllocationDuringMultipleAllocationClients: Readdress fl…
bff72b1 
…ake (#3831)

The actual bug is here:
  https://github.com/googleforgames/agones/blob/2023077aff4125b9d9822c5d20cd773d13138385/test/e2e/gameserverallocation_test.go#L1423

When we tried to accept errors in #3750, we forgot to
change this t.Errorf to a simple log.

Along the way:

* change the timing of the scale-down so it happens more probably in
the middle of the allocations, which was the intent. By my guess, 3s
might be past all allocations. Also add a log to see if I'm right.

* minor cleanup

Towards #3553
@github-actions GitHub Actions
Copy link

'This issue is marked as Stale due to inactivity for more than 30 days. To avoid being marked as 'stale' please add 'awaiting-maintainer' label or add a comment. Thank you for your contributions '

@github-actions github-actions bot added the stale Pending closure unless there is a strong objection. label Aug 15, 2024
@zmerlynn zmerlynn added awaiting-maintainer Block issues from being stale/obsolete/closed and removed stale Pending closure unless there is a strong objection. labels Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting-maintainer Block issues from being stale/obsolete/closed kind/bug These are bugs.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@markmandel @zmerlynn @Kalaiselvi84