Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add close and clean harvesting #70

Merged
merged 6 commits into from
Jan 5, 2017
Merged

Add close and clean harvesting #70

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
dca54f0
Paramatrize prospectors variables for filebeat 5
Jan 3, 2017
2858b47
remove an extra blank
Jan 3, 2017
3e11de6
Check some harvesting parameters are booleans and link dependencies b…
Jan 5, 2017
8cb4237
check boolean values
Jan 5, 2017
5ea868d
print a message on client side when filebeat is missconfigured
Jan 5, 2017
e3e3f4d
deleting notify since it has to be done by filebeat service these kin…
Jan 5, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions manifests/prospector.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,13 @@
$backoff = '1s',
$max_backoff = '10s',
$backoff_factor = 2,
$close_inactive = '5m',
$close_renamed = false,
$close_removed = true,
$close_eof = false,
$clean_inactive = 0,
$clean_removed = true,
$close_timeout = 0,
$force_close_files = false,
$include_lines = [],
$exclude_lines = [],
Expand All @@ -25,12 +32,17 @@

validate_hash($fields, $multiline)
validate_array($paths, $exclude_files, $include_lines, $exclude_lines, $tags)
validate_bool($tail_files, $close_renamed, $close_removed, $close_eof, $clean_removed)

$prospector_template = $filebeat::real_version ? {
'1' => 'prospector1.yml.erb',
default => 'prospector5.yml.erb',
}

if $close_removed == false and $clean_removed == true {
notify('If you set close_removed to false, does not make sense to set clean_removed to true. Review the official Filebeat documentation')
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how much commentary to add about configuration issues - there are a number of conflicting configuration parameters that can be set (e.g. ignore_older has to be greater than close_inactive).

Checking and warning for every conflict would be difficult to maintain, so I'm inclined to leave them out entirely.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough. Let's skip these kind of conflicts

}

case $::kernel {
'Linux' : {
$filebeat_path = $filebeat::real_version ? {
Expand Down
20 changes: 13 additions & 7 deletions templates/prospector5.yml.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,33 +121,39 @@ filebeat:
# Close inactive closes the file handler after the predefined period.
# The period starts when the last line of the file was, not the file ModTime.
# Time strings like 2h (2 hours), 5m (5 minutes) can be used.
#close_inactive: 5m
<%- if @close_inactive -%>
close_inactive: <%= @close_inactive %>
<%- end -%>

# Close renamed closes a file handler when the file is renamed or rotated.
# Note: Potential data loss. Make sure to read and understand the docs for this option.
#close_renamed: false
close_renamed: <%= @close_renamed %>

# When enabling this option, a file handler is closed immediately in case a file can't be found
# any more. In case the file shows up again later, harvesting will continue at the last known position
# after scan_frequency.
#close_removed: true
close_removed: <%= @close_removed %>
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If close_removed is false then this will be excluded, defaulting to true.


# Closes the file handler as soon as the harvesters reaches the end of the file.
# By default this option is disabled.
# Note: Potential data loss. Make sure to read and understand the docs for this option.
#close_eof: false
close_eof: <%= @close_eof %>

### State options

# Files for the modification data is older then clean_inactive the state from the registry is removed
# By default this is disabled.
#clean_inactive: 0
<%- if @clean_inactive -%>
clean_inactive: <%= @clean_inactive %>
<%- end -%>

# Removes the state for file which cannot be found on disk anymore immediately
#clean_removed: true
clean_removed: <%= @clean_removed %>
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic for setting this will work the same way that close_removed works, so we should probably improve that a bit here.


# Close timeout closes the harvester after the predefined time.
# This is independent if the harvester did finish reading the file or not.
# By default this option is disabled.
# Note: Potential data loss. Make sure to read and understand the docs for this option.
#close_timeout: 0
<%- if @close_timeout -%>
close_timeout: <%= @close_timeout %>
<%- end -%>