Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase splunkObservability and splunkPlatform timeouts to 20s #438

Closed
wants to merge 1 commit into from
Closed

Conversation

jvoravong
Copy link
Contributor

@jvoravong jvoravong commented Apr 26, 2022

Related Issue: signalfx/splunk-otel-collector#1454

  • Added timeout configuration to splunkObservability.
  • Increased splunkObservability.timeout from 5s or 10s to 20s.
  • Increase splunkPlatform.timeout from 10s to 20s.

@jvoravong jvoravong changed the title Increase splunkObservability and splunkPlatform timeout to 20s Increase splunkObservability and splunkPlatform timeouts to 20s Apr 26, 2022
@jvoravong jvoravong marked this pull request as ready for review April 26, 2022 17:25
@jvoravong jvoravong requested review from a team as code owners April 26, 2022 17:25
@@ -146,6 +146,9 @@
"profilingEnabled": {
"description": "Whether profiling data is enabled for Splunk Observability",
"type": "boolean"
},
"timeout": {
"type": "string"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mind adding a description?

@@ -108,6 +108,9 @@ splunkObservability:
# If you don't use AlwaysOn Profiling for Splunk APM, you can disable it.
profilingEnabled: false

# HTTP timeout for sending data.
timeout: 20s
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

20s is pretty high, so maybe we should keep the default of 10 with the new value until/unless we receive reports of it being exceeded (wasn't the linked issue just for exceeded platform timeouts)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Ryan. 20s for HTTP connection is too high. We shouldn't change it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your input.
I rarely see the same context error in my k8s clusters that upload data to Splunk Observability, it probably only happens when backend or middle issues are present.
I'm okay with not adding a splunkObservability.timeout config, I thought it might be useful for Observability users.

Copy link
Contributor

@dmitryax dmitryax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's reasonable option to have. We should merge open-telemetry/opentelemetry-collector-contrib#6803 and expose full set of http settings instead. 10s timeout is already very high. I doubt that users really need it

Copy link
Contributor

@dmitryax dmitryax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to block this PR for now. But I'm open to accept it if someone can address my concerns

@dmitryax
Copy link
Contributor

I just realized that all the http settings that will be added in open-telemetry/opentelemetry-collector-contrib#6803 are squashed, so this config option won't break it. Probably we can add timeout as done in this PR if it's really needed, but I'm apposed to bumping the defaults unless we are confident that this is needed

@jvoravong
Copy link
Contributor Author

I'm going to close this PR for now since we should take a different approach to addressing the related issue.

@jvoravong jvoravong closed this Apr 27, 2022
@jvoravong jvoravong deleted the 1454 branch April 27, 2022 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants