Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memcached: fix stuck on processing GET request #94

Merged
merged 1 commit into from
Dec 28, 2021

Conversation

ligurio
Copy link
Member

@ligurio ligurio commented Dec 16, 2021

Summary: memcached stuck on processing request when number of
processed commands is exceeded of (batch_count + 1) and buffer has a
request with retrieval command ("get", "gets", "gat", and "gats").

Fixes #73

@ligurio ligurio marked this pull request as draft December 16, 2021 16:40
@ligurio ligurio force-pushed the ligurio/gh-73-binary-boundary branch from 9b3528a to da90b77 Compare December 18, 2021 10:18
@ligurio ligurio marked this pull request as ready for review December 18, 2021 10:21
@ligurio ligurio requested a review from Totktonada December 18, 2021 10:21
@ligurio ligurio force-pushed the ligurio/gh-73-binary-boundary branch 2 times, most recently from 3b86908 to f3a5b65 Compare December 20, 2021 12:45
@ligurio ligurio force-pushed the ligurio/gh-73-binary-boundary branch from f3a5b65 to 3cf6ed3 Compare December 27, 2021 12:49
Summary: memcached stuck on processing request when number of
processed commands is exceeded of (batch_count + 1) and buffer has a
request with retrieval command ("get", "gets", "gat", and "gats").

For each incoming connection memcached starts endless loop in function
memcached_loop() where it reads network data to buffer
(memcached_loop_read), parse commands from a buffer one by one
(parse_request), process commands (process_request) and write answers to
network socket (memcached_flush).

Imagine we have a sequence of requests equal to con->cfg->batch_count
(it is a constant value and equal to 20) + 1 with SETQ (SETQ is the same
as a SET command, but it is not required confirmation) commands and a
single GET command that should return a value set by one of the previous
SETQ command. Processing of SETQ commands finished successfully. After
processing the latest request with SETQ command we have a GET command in
a buffer and batch_count counter become equal to 20 (batch_count initial
value is 0). In such case control flow returns to start of the
read-parse-process-write loop where we stuck in memcached_loop_read()
and wait for next requests instead of replying to client with value of
requested key.

    rc = con->cb.process_request(con);
    ...
    if (rc == -1)
            memcached_loop_error(con);
    if (con->close_connection) {
            break;
    } else if (rc == 0 && ibuf_used(con->in) > 0 &&
               batch_count < con->cfg->batch_count) {
            goto next;
    }
    ...
    continue;

Fixes #73
@ligurio ligurio force-pushed the ligurio/gh-73-binary-boundary branch from 3cf6ed3 to d11732f Compare December 27, 2021 12:52
Copy link
Member

@Totktonada Totktonada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Feel free to proceed.

@ligurio
Copy link
Member Author

ligurio commented Dec 27, 2021

Seems a temporary network problems on GH side and memcached CI has failed several times in a row:

Download action repository 'actions/checkout@v2' (SHA:ec3a7ce113134d7a93b817d10a8272cb61118579)
Warning: Failed to download action 'https://api.github.com/repos/actions/checkout/tarball/ec3a7ce113134d7a93b817d10a8272cb61118579'. Error: The operation was canceled.
Warning: Back off 17.326 seconds before retry.
Download action repository 'tarantool/setup-tarantool@v1' (SHA:7112e91775b8733f1237f6884696cd62757e0eee)
Download action repository 'actions/setup-python@v2' (SHA:f38219332975fe8f9c04cca981d674bf22aea1d3)
Warning: Failed to download action 'https://api.github.com/repos/actions/setup-python/tarball/f38219332975fe8f9c04cca981d674bf22aea1d3'. Error: The operation was canceled.
Warning: Back off 28.993 seconds before retry.
Warning: Failed to download action 'https://api.github.com/repos/actions/setup-python/tarball/f38219332975fe8f9c04cca981d674bf22aea1d3'. Error: The operation was canceled.
Warning: Back off 20.315 seconds before retry.
Error: The operation was canceled.

https://www.githubstatus.com/history

Screenshot from 2021-12-27 17-51-25

@ligurio ligurio merged commit bd78d75 into master Dec 28, 2021
@ligurio ligurio deleted the ligurio/gh-73-binary-boundary branch December 28, 2021 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

test: binary/binary-boundary.test.py test gets stuck from time to time
2 participants