Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add test for reloading the module with external iteration state via Cartridge hotreload #54

Closed
akudiyar opened this issue May 12, 2021 · 2 comments · Fixed by #123
Closed

Add test for reloading the module with external iteration state via Cartridge hotreload #54

akudiyar opened this issue May 12, 2021 · 2 comments · Fixed by #123
Assignees
@oleg-jukovec
Labels
tests

Comments

@akudiyar
Copy link

Use case:

A user creates an expiration task, for which the expiration check callback or the iteration function saves some state into the globals or cartridge.vars (e.g. iterator state or the last tuple value, etc). In case of reloading the module in a Cartridge role, the user must have the ability to preserve the state for continuing the iteration with it after the reload.

AC:

New integration test added for the above scenario and is passing in the CI
@akudiyar akudiyar added the tests label May 12, 2021
@akudiyar akudiyar added this to the Q2 2021 milestone May 12, 2021
@ArtDu ArtDu mentioned this issue Jun 1, 2021
Closed
ArtDu added a commit that referenced this issue Jul 21, 2021
@ArtDu
Test for check hotreload
3b2a9ae 
The class of problems that this test catches is the expirationd
integration with cartridge. When hot-reloading roles
(that is, when call require("cartridge.roles").reload() ),
expirationd should exit gracefully.
All fibers should terminate and the task list should be cleared.

Also added cartridge rock to ci.
Bump luatest cause bug with before_suite.

Closes #54
ArtDu added a commit that referenced this issue Jul 21, 2021
@ArtDu
Test for check hotreload
d546b79 
The class of problems that this test catches is the expirationd
integration with cartridge. When hot-reloading roles
(that is, when call require("cartridge.roles").reload() ),
expirationd should exit gracefully.
All fibers should terminate and the task list should be cleared.

Also added cartridge rock to ci.
Bump luatest cause bug with before_suite.

Closes #54
ArtDu added a commit that referenced this issue Jul 22, 2021
@ArtDu
test: check hotreload
bca2c58 
The class of problems that this test catches is the expirationd
integration with cartridge. When hot-reloading roles
(that is, when call require("cartridge.roles").reload() ),
expirationd should exit gracefully.
All fibers should terminate and the task list should be cleared.

Also added cartridge rock to ci.
Bump luatest cause bug with before_suite.

Closes #54
ArtDu added a commit that referenced this issue Jul 24, 2021
@ArtDu
test: check hotreload
5575583 
The class of problems that this test catches is the expirationd
integration with cartridge. When hot-reloading roles
(that is, when call require("cartridge.roles").reload() ),
expirationd should exit gracefully.
All fibers should terminate and the task list should be cleared.

Also added cartridge rock to ci.
Bump luatest cause bug with before_suite.

Closes #54
ArtDu added a commit that referenced this issue Jul 26, 2021
@ArtDu
test: check hotreload
bfe3e9e 
The class of problems that this test catches is the expirationd
integration with cartridge. When hot-reloading roles
(that is, when call require("cartridge.roles").reload() ),
expirationd should exit gracefully.
All fibers should terminate and the task list should be cleared.

Also added cartridge rock to ci.
Bump luatest cause bug with before_suite.

Closes #54
@vrogach2020
Copy link

After some discussions I think we should split this issue to support some cartridge-aware features:

  • Create catridge role for expirationd. This provides a common way to
    use the module with cartridge and support standard role lifecycle (init, apply config, stop etc).

  • Configure tasks from YAML config

  • Support hot-reload of cartridge role

  • Save task state to continue iteration of large spaces

@Totktonada
Copy link
Member

Regarding storing the state of the iterator. I had a question. We can do the following:

  • Always start from beginning.
  • Save a state in runtime.
  • Persist the state in the database.

The second option is proposed, but it is not clear how this choice was made.

Now I got it: we can't just store a current key anymore, because there is ability to pass a custom iterator to expirationd. Okay so.

@Totktonada Totktonada added the 3sp label Jun 18, 2022
@Totktonada Totktonada removed this from the Q2 2021 milestone Jun 18, 2022
@oleg-jukovec oleg-jukovec self-assigned this Jul 14, 2022
oleg-jukovec added a commit that referenced this issue Jul 18, 2022
@oleg-jukovec
api: continue a task from a last tuple
d72acd4 
After the patch a task continues processing from a last tuple at
startup unless it was killed using `task:kill()` or
`expirationd.kill()`. We cannot provide a safe behavior for all user
cases, so it works only with default `start_key` and `iterate_with`
callbacks.

The behavior supports a cartridge hot-reload.

Closes #54
oleg-jukovec added a commit that referenced this issue Jul 18, 2022
@oleg-jukovec
api: continue a task from a last tuple
7af5f17 
After the patch a task continues processing from a last tuple at
startup unless it was killed using `task:kill()` or
`expirationd.kill()`. We cannot provide a safe behavior for all user
cases, so it works only with default `start_key` and `iterate_with`
callbacks.

The behavior supports a cartridge hot-reload.

Closes #54
oleg-jukovec added a commit that referenced this issue Jul 18, 2022
@oleg-jukovec
api: continue a task from a last tuple
e2e5893 
After the patch a task continues processing from a last tuple at
startup unless it was killed using `task:kill()` or
`expirationd.kill()`. We cannot provide a safe behavior for all user
cases, so it works only with default `start_key` and `iterate_with`
callbacks.

The behavior supports a cartridge hot-reload.

Closes #54
oleg-jukovec added a commit that referenced this issue Jul 18, 2022
@oleg-jukovec
api: continue a task from a last tuple
6b119b2 
After the patch a task continues processing from a last tuple at
startup unless it was killed using `task:kill()` or
`expirationd.kill()`. We cannot provide a safe behavior for all user
cases, so it works only with default `start_key` and `iterate_with`
callbacks.

The behavior supports a cartridge hot-reload.

Closes #54
@oleg-jukovec oleg-jukovec mentioned this issue Jul 18, 2022
Merged
oleg-jukovec added a commit that referenced this issue Jul 21, 2022
@oleg-jukovec
api: continue a task from a last tuple
e234430 
After the patch a task continues processing from a last tuple at
startup unless it was killed using `task:kill()` or
`expirationd.kill()`. We cannot provide a safe behavior for all user
cases, so it works only with default `start_key` and `iterate_with`
callbacks.

The behavior supports a cartridge hot-reload.

Closes #54
oleg-jukovec added a commit that referenced this issue Jul 21, 2022
@oleg-jukovec
api: continue a task from a last tuple
92c92b9 
After the patch a task continues processing from a last tuple at
startup unless it was killed using `task:kill()` or
`expirationd.kill()`. We cannot provide a safe behavior for all user
cases, so it works only with default `start_key` and `iterate_with`
callbacks.

The behavior supports a cartridge hot-reload.

Closes #54
oleg-jukovec added a commit that referenced this issue Jul 22, 2022
@oleg-jukovec
api: continue a task from a last tuple
c168329 
After the patch a task continues processing from a last tuple at
startup unless it was killed using `task:kill()` or
`expirationd.kill()`. We cannot provide a safe behavior for all user
cases, so it works only with default `start_key` and `iterate_with`
callbacks.

The behavior supports a cartridge hot-reload.

Closes #54
oleg-jukovec added a commit that referenced this issue Jul 22, 2022
@oleg-jukovec
api: continue a task from a last tuple
6879f60 
After the patch a task continues processing from a last tuple at
startup unless it was killed using `task:kill()` or
`expirationd.kill()`. We cannot provide a safe behavior for all user
cases, so it works only with default `start_key` and `iterate_with`
callbacks.

The behavior supports a cartridge hot-reload.

Closes #54
oleg-jukovec added a commit that referenced this issue Jul 22, 2022
@oleg-jukovec
api: continue a task from a last tuple
718bdf2 
After the patch a task continues processing from a last tuple at
startup unless it was killed using `task:kill()` or
`expirationd.kill()`. We cannot provide a safe behavior for all user
cases, so it works only with default `start_key` and `iterate_with`
callbacks.

The behavior supports a cartridge hot-reload.

Closes #54
oleg-jukovec added a commit that referenced this issue Jul 22, 2022
@oleg-jukovec
api: continue a task from a last tuple
cc7605a 
After the patch a task continues processing from a last tuple at
startup unless it was killed using `task:kill()` or
`expirationd.kill()`. We cannot provide a safe behavior for all user
cases, so it works only with default `start_key` and `iterate_with`
callbacks.

The behavior supports a cartridge hot-reload.

Closes #54
oleg-jukovec added a commit that referenced this issue Jul 27, 2022
@oleg-jukovec
api: continue a task from a last tuple
bd20ded 
After the patch a task continues processing from a last tuple at
startup unless it was killed using `task:kill()` or
`expirationd.kill()`. We cannot provide a safe behavior for all user
cases, so it works only with default `start_key` and `iterate_with`
callbacks.

The behavior supports a cartridge hot-reload.

Closes #54
oleg-jukovec added a commit that referenced this issue Aug 4, 2022
@oleg-jukovec
Release 1.3.0
167519d 
Overview

    This release adds a Tarantool Cartridge role for expirationd
    package and improves the default behavior.

Breaking changes

    None.

New features

    Continue a task from a last tuple (#54).

    Decrease tarantool-checks dependency from 3.1 to 2.1 (#124).

    Process a task on a writable space by default (#42).

    Wait until a space or an index is created (#68, #116).

    Tarantool Cartridge role (#107).

Bugfixes

    Fix build and installation of rpm/deb packages (#124).

    Do not restart work fiber if index does not exist (#64).

    Update changelogs for rpm/deb packages.

Testing

    Shuffle tests (#118).

    Fix test_mvcc_vinyl_tx_conflict (#104, #105).

    Fix flaky 'simple expires test' (#90).

Other

    Add GitHub Actions workflow with debug Tarantool build (#102).

    Add GitHub Actions workflow for deploying module packages to S3
    based repositories (#43).
@oleg-jukovec oleg-jukovec mentioned this issue Aug 4, 2022
Merged
oleg-jukovec added a commit that referenced this issue Aug 4, 2022
@oleg-jukovec
Release 1.3.0
0df103e 
Overview

    This release adds a Tarantool Cartridge role for expirationd
    package and improves the default behavior.

Breaking changes

    None.

New features

    Continue a task from a last tuple (#54).

    Decrease tarantool-checks dependency from 3.1 to 2.1 (#124).

    Process a task on a writable space by default (#42).

    Wait until a space or an index is created (#68, #116).

    Tarantool Cartridge role (#107).

Bugfixes

    Fix build and installation of rpm/deb packages (#124).

    Do not restart a work fiber if an index does not exist (#64).

    Update changelogs for rpm/deb packages.

Testing

    Shuffle tests (#118).

    Fix test_mvcc_vinyl_tx_conflict (#104, #105).

    Fix flaky 'simple expires test' (#90).

Other

    Add GitHub Actions workflow with debug Tarantool build (#102).

    Add GitHub Actions workflow for deploying module packages to S3
    based repositories (#43).
oleg-jukovec added a commit that referenced this issue Aug 5, 2022
@oleg-jukovec
Release 1.3.0
ee32a37 
Overview

    This release adds a Tarantool Cartridge role for expirationd
    package and improves the default behavior.

Breaking changes

    None.

Deprecated

    Obsolete functions: task_stats, kill_task, get_task, get_tasks, run_task,
    show_task_list.

New features

    Continue a task from a last tuple (#54).

    Decrease tarantool-checks dependency from 3.1 to 2.1 (#124).

    Process a task on a writable space by default (#42).

    Wait until a space or an index is created (#68, #116).

    Tarantool Cartridge role (#107).

Bugfixes

    Fix build and installation of rpm/deb packages (#124).

    Do not restart a work fiber if an index does not exist (#64).

    Update changelogs for rpm/deb packages.

Testing

    Shuffle tests (#118).

    Fix test_mvcc_vinyl_tx_conflict (#104, #105).

    Fix flaky 'simple expires test' (#90).

Other

    Add GitHub Actions workflow with debug Tarantool build (#102).

    Add GitHub Actions workflow for deploying module packages to S3
    based repositories (#43).
oleg-jukovec added a commit that referenced this issue Aug 11, 2022
@oleg-jukovec
Release 1.3.0
3d50c4d 
Overview

    This release adds a Tarantool Cartridge role for expirationd
    package and improves the default behavior.

Breaking changes

    None.

Deprecated

    Obsolete functions: task_stats, kill_task, get_task, get_tasks, run_task,
    show_task_list.

New features

    Continue a task from a last tuple (#54).

    Decrease tarantool-checks dependency from 3.1 to 2.1 (#124).

    Process a task on a writable space by default (#42).

    Wait until a space or an index is created (#68, #116).

    Tarantool Cartridge role (#107).

Bugfixes

    Fix build and installation of rpm/deb packages (#124).

    Do not restart a work fiber if an index does not exist (#64).

    Update changelogs for rpm/deb packages.

Testing

    Shuffle tests (#118).

    Fix test_mvcc_vinyl_tx_conflict (#104, #105).

    Fix flaky 'simple expires test' (#90).

Other

    expirationd.start() parameter `space_id` has been renamed to
    `space` (#112).

    Add GitHub Actions workflow with debug Tarantool build (#102).

    Add GitHub Actions workflow for deploying module packages to S3
    based repositories (#43).
oleg-jukovec added a commit that referenced this issue Aug 11, 2022
@oleg-jukovec
Release 1.3.0
4a7e381 
Overview

    This release adds a Tarantool Cartridge role for expirationd
    package and improves the default behavior.

Breaking changes

    None.

Deprecated

    Obsolete functions: task_stats, kill_task, get_task, get_tasks, run_task,
    show_task_list.

New features

    Continue a task from a last tuple (#54).

    Decrease tarantool-checks dependency from 3.1 to 2.1 (#124).

    Process a task on a writable space by default (#42).

    Wait until a space or an index is created (#68, #116).

    Tarantool Cartridge role (#107).

Bugfixes

    Fix build and installation of rpm/deb packages (#124).

    Do not restart a work fiber if an index does not exist (#64).

    Update changelogs for rpm/deb packages.

Testing

    Shuffle tests (#118).

    Fix test_mvcc_vinyl_tx_conflict (#104, #105).

    Fix flaky 'simple expires test' (#90).

Other

    expirationd.start() parameter `space_id` has been renamed to
    `space` (#112).

    Add GitHub Actions workflow with debug Tarantool build (#102).

    Add GitHub Actions workflow for deploying module packages to S3
    based repositories (#43).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@oleg-jukovec oleg-jukovec

Labels
tests
Projects
None yet
Milestone
No milestone
5 participants
@Totktonada @akudiyar @ArtDu @oleg-jukovec @vrogach2020