Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(gcp_stackdriver_logs sink): refactor to new style #18335

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
a16de09
First iteration, unit tests pass.
neuronull Aug 9, 2023
969653d
Extract common Service functionality for HTTP based stream sinks.
neuronull Aug 9, 2023
5dc4921
docs touchup
neuronull Aug 9, 2023
10fef82
Touch ups
neuronull Aug 9, 2023
d320b10
spell checker
neuronull Aug 9, 2023
a496821
fix rust doc
neuronull Aug 10, 2023
44c99b3
hopefully fix encoding regression
neuronull Aug 10, 2023
4aa3eca
Try item sized batching
neuronull Aug 14, 2023
68ff83c
Refactor a bit
neuronull Aug 15, 2023
98c3113
cleanup
neuronull Aug 15, 2023
152de8a
cleanup
neuronull Aug 15, 2023
1e696d4
doc clean up
neuronull Aug 16, 2023
06381de
extract common service code for re use in other HTTP sinks
neuronull Aug 16, 2023
24a28f7
Merge branch 'master' into neuronull/sink_newstyle_refactor_http
neuronull Aug 16, 2023
9efe546
feeback sw
neuronull Aug 17, 2023
c22f4d3
Merge branch 'master' into neuronull/sink_newstyle_refactor_http
neuronull Aug 17, 2023
2ea6181
duplicate
neuronull Aug 17, 2023
2bfe0d0
feedback ds
neuronull Aug 18, 2023
b2fac79
feedback ds
neuronull Aug 18, 2023
5316602
clippy
neuronull Aug 18, 2023
38969d2
unit tests pass
neuronull Aug 21, 2023
2fe1f05
First iteration, unit tests pass.
neuronull Aug 9, 2023
fe2e4a4
Extract common Service functionality for HTTP based stream sinks.
neuronull Aug 9, 2023
3fc23a1
docs touchup
neuronull Aug 9, 2023
fcc1851
Touch ups
neuronull Aug 9, 2023
a521b2f
spell checker
neuronull Aug 9, 2023
b99b9a7
fix rust doc
neuronull Aug 10, 2023
d9c6acc
hopefully fix encoding regression
neuronull Aug 10, 2023
ebf8e77
Try item sized batching
neuronull Aug 14, 2023
8f8d2a2
Refactor a bit
neuronull Aug 15, 2023
8f91d17
cleanup
neuronull Aug 15, 2023
527a919
cleanup
neuronull Aug 15, 2023
89d3909
doc clean up
neuronull Aug 16, 2023
dfe2335
extract common service code for re use in other HTTP sinks
neuronull Aug 16, 2023
43ae60f
feeback sw
neuronull Aug 17, 2023
7c3e56d
duplicate
neuronull Aug 17, 2023
d4337f8
feedback ds
neuronull Aug 18, 2023
a66a924
feedback ds
neuronull Aug 18, 2023
bc6ec0d
clippy
neuronull Aug 18, 2023
b58e48f
unit tests pass
neuronull Aug 21, 2023
17aacf4
Merge branch 'neuronull/sink_newstyle_refactor_gcp_stackdriver_logs' …
neuronull Aug 21, 2023
e6ddb44
cleanup
neuronull Aug 21, 2023
517c692
use the common batcher
neuronull Aug 21, 2023
20c9199
Merge branch 'master' into neuronull/sink_newstyle_refactor_gcp_stack…
neuronull Aug 21, 2023
4933945
merge conflicts
neuronull Aug 21, 2023
6571e8d
clippy
neuronull Aug 21, 2023
71c5c03
feedback ds
neuronull Aug 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
use the common batcher
  • Loading branch information
neuronull committed Aug 21, 2023
commit 517c6926554a2247afc34beace5150d2e13fc961
12 changes: 9 additions & 3 deletions src/sinks/gcp/stackdriver/logs/sink.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
//! Implementation of the `gcp_stackdriver_logs` sink.

use crate::sinks::{prelude::*, util::http::HttpRequest};
use crate::sinks::{
prelude::*,
util::http::{HttpJsonBatchSizer, HttpRequest},
};

use super::request_builder::StackdriverLogsRequestBuilder;

Expand Down Expand Up @@ -33,8 +36,11 @@ where
async fn run_inner(self: Box<Self>, input: BoxStream<'_, Event>) -> Result<(), ()> {
let service = ServiceBuilder::new().service(self.service);
input
// Batch the input stream with byte size calculation
.batched(self.batch_settings.into_byte_size_config())
// Batch the input stream with size calculation based on the estimated encoded json size
.batched(
self.batch_settings
.into_item_size_config(HttpJsonBatchSizer),
)
// Build requests with no concurrency limit.
.request_builder(None, self.request_builder)
// Filter out any errors that occurred in the request building.
Expand Down
2 changes: 1 addition & 1 deletion src/sinks/util/http.rs
Original file line number Diff line number Diff line change
Expand Up @@ -690,7 +690,7 @@ pub fn http_response_retry_logic() -> HttpStatusRetryLogic<

/// Uses the estimated json encoded size to determine batch sizing.
#[derive(Default)]
pub(super) struct HttpJsonBatchSizer;
pub struct HttpJsonBatchSizer;

impl ItemBatchSize<Event> for HttpJsonBatchSizer {
fn size(&self, item: &Event) -> usize {
Expand Down