Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Persistent Queue v1 #5958

Closed
wants to merge 203 commits into from
Closed

Persistent Queue v1 #5958

wants to merge 203 commits into from

Conversation

colinsurprenant
Copy link
Contributor

This is a WIP level PR to start getting eyeballs and bootstrapping the review process. Please note the following:

  • tasks/issues tracking for the first iteration of PQ is in [meta] first persistence implementation release tracking #5638. There are still important details to look into so try and avoid reporting already tracked issues/tasks in this PR
  • by looking at the files changed by filing the PR I can already see a few files that should not be part of this PR, I will look into this shortly

First Iteration Design Goals:

  • prevent inflight data loss upon app/os/machine crash by aiming at at-least-once delivery semantic. note that at this point there is no change in the per-event processing/delivery failure where best-effort is done at the plugin level and can result in dropped event. This will be addressed in the future Dead Letter Queue feature, see Design: Events eviction and Dead Letter Queue #5283
  • provide backpressure handling within logstash using this variable length persistent queue. This will help avoid installing external intermediate message queues for the sole purpose of handling logstash backpressure.

Usage:

  • by default PQ is enabled and in the data/queue path. make sure it exists: $ mkdir -p data/queue
  • $ rm -f data/queue/* will clean the queue if needed when logstash is not running.
  • The following new config options are supported:
    • path.queue: which defaults todata/queue`
    • queue.type: which default to the disk based PQ but can also take the memory option to use the in-memory implementation of the PQ (that we use in tests), or synchronous which is the legacy in-memory queue.
    • queue.page_capacity which defaults to 250M
    • queue.queue_capacity which for now is a maximum number of events limit in the queue, by default there is no limit

colinsurprenant and others added 30 commits June 22, 2016 15:12
@colinsurprenant
java_persistence feature branch first commit from copy of colinsurpre…
a04576b 
…nant/acked_persistent_queue repo
@colinsurprenant
acking tests and fixes
07d5ecd
@colinsurprenant
extracted page acking state handling into its own PageState class
72a66fc
@colinsurprenant
added single element read
90fa3ca
@colinsurprenant
first queue implementation
f26e428
@colinsurprenant
fix acking and add reset
688ee49
@colinsurprenant
Queue tests
75bf5ac
@colinsurprenant
cleanups
0ba9183
WIP Translog, todo: TranslogWriter and TranslogReader
0b0a8a5
@colinsurprenant
renamed Metadata to VolatileMetadata
171cd45
@colinsurprenant
turned Metadata into interface with volatile impl, PageHandler is now…
ba03824 
… abstract with volatile impl
@colinsurprenant
extracted Queue interface
3a2ee45
@colinsurprenant
Merge remote-tracking branch 'upstream/feature/java_persistence' into…
8290248 
… feature/java_persistence
@colinsurprenant
renamed ackingState to pageState
076e91b
@colinsurprenant
renamed Metadata to QueueState
e8c4d0e
@colinsurprenant
renamed PageHandler to PagedQueue
2294593
@colinsurprenant
post renaming cleanups
b0adf3d
@colinsurprenant
now using int for page index, started moving more logic in QueueState
1e00a92
@colinsurprenant
renamed package ackedqueue to previous_ackedqueue
93ee0d1
@colinsurprenant
first draft class layout for new queue and persistence architecture
745145e
@colinsurprenant
most methods have implementation and/or pseudocode, identified todos
9a4811e
@colinsurprenant
added new Queueable interface
9eb2090
Checkpoint basic read and write
443015f
Merge commit '9eb20906f2835a82819e424c568c39bfd5779ba1' into feature/…
5228cc4 
…java_persistence

# Conflicts:
#	logstash-core/src/main/java/org/logstash/ackedqueue/Checkpoint.java
#	logstash-core/src/main/java/org/logstash/ackedqueue/HeadPage.java
#	logstash-core/src/main/java/org/logstash/ackedqueue/Queue.java
@colinsurprenant
document page read/write concerns inconsistency
5cd8559
@colinsurprenant
Merge remote-tracking branch 'upstream/feature/java_persistence' into…
2a2fbf8 
… feature/java_persistence
@colinsurprenant
page IO delegation first iteration
b871de9
@colinsurprenant
elementClass getter
ce86af0
@colinsurprenant
added temp simple StringElement and ElementFactory to work around des…
8d92b7b 
…erialization+interface problem
implement MemoryElementIOStream and tests.
7f32fc1
@colinsurprenant colinsurprenant mentioned this pull request Oct 18, 2016
Closed
jsvd and others added 15 commits October 19, 2016 03:54
@jsvd
in agent specs don't rely on complete queue drain
fe485a3 
with persistence we no longer wait for total queue drain before
terminating the pipeline. Some of the specs for the Agent class use the
count of events created by the generator plugin to ensure the metric
counters are correct.
This commit changes the assertion strategy to not depend on exact
values, but instead on their properties.

Fixes #6084
@colinsurprenant
remove commented require
7e7b413
@colinsurprenant
remove commented require
832e3a0
@colinsurprenant
moved /MemoryPageIOStream into wip package
9fbc160
@colinsurprenant
Merge remote-tracking branch 'upstream/feature/java_persistence' into…
74a5f5e 
… feature/java_persistence
@colinsurprenant
added comment
2466ded
@colinsurprenant
removed commented code added comment
b8f4d0d
@colinsurprenant
removed commented code
47503a4
@colinsurprenant
license comment on top
0097b65
@colinsurprenant
removed glob import
339b7e5
@jordansissel
Byte values are whole units, not floating point values (no fractions …
3b7b64e 
…of a byte). This fixes a flakey test
@jordansissel
Skip socket test due to path length test failure that only affects th…
2cd7286 
…e test itself on OSX.
@colinsurprenant
fixed empty head pages creationm, reworked and DRYed Queue.open
7c57699
@colinsurprenant
Merge remote-tracking branch 'upstream/feature/java_persistence' into…
27de296 
… feature/java_persistence
@colinsurprenant
Merge branch 'master' into feature/java_persistence
73d4fe6
talevy
talevy reviewed Oct 27, 2016
View reviewed changes
logstash-core/src/main/java/org/logstash/common/io/wip/MemoryPageIOStream.java
@@ -0,0 +1,279 @@
package org.logstash.common.io.wip;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this package?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as discussed this is a temporary package to hold the unfinished MemoryPageIOStream.java so it can benefit any code refactor etc.

@colinsurprenant colinsurprenant changed the title [WIP] Persistent Queue Persistent Queue v1 Oct 28, 2016
@suyograo
Copy link
Contributor

suyograo commented Nov 2, 2016

Closing this as it is already merged.

@suyograo suyograo closed this Nov 2, 2016
@original-brownbear original-brownbear mentioned this pull request May 9, 2017
Closed
@suyograo suyograo deleted the feature/java_persistence branch July 24, 2017 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@guyboertje guyboertje guyboertje left review comments

@talevy talevy talevy left review comments

@suyograo suyograo suyograo left review comments

@jordansissel jordansissel jordansissel approved these changes

Assignees
No one assigned
Labels
enhancement v5.1.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@colinsurprenant @jordansissel @talevy @suyograo @guyboertje @jsvd