otel: add syslog-ng <-> syslog-ng communication via OTLP #4564
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alltilla
force-pushed
the
syslog-ng-otlp
branch
6 times, most recently
from
131f46d to
785a319
Compare
alltilla
force-pushed
the
syslog-ng-otlp
branch
3 times, most recently
from
69f66a1 to
8575307
Compare
alltilla
force-pushed
the
syslog-ng-otlp
branch
3 times, most recently
from
5188a35 to
a1bc6e6
Compare
Signed-off-by: Attila Szakacs <[email protected]>
We never do anything seqid related, the LogTemplateEvalOptions is only needed for the ValuePairs walk. The value 11 before was a copy paste error, and it caused no problems, because it is not used. Signed-off-by: Attila Szakacs <[email protected]>
Signed-off-by: Attila Szakacs <[email protected]>
Signed-off-by: Attila Szakacs <[email protected]>
So we can access them from descendant classes. Signed-off-by: Attila Szakacs <[email protected]>
It makes it easier to derive from the worker. Signed-off-by: Attila Szakacs <[email protected]>
Signed-off-by: Attila Szakacs <[email protected]>
This makes it easier to derive. Signed-off-by: Attila Szakacs <[email protected]>
Signed-off-by: Attila Szakacs <[email protected]>
This will be useful when parsing the syslog-ng-otel() messages. Signed-off-by: Attila Szakacs <[email protected]>
Signed-off-by: Attila Szakacs <[email protected]>
Signed-off-by: Attila Szakacs <[email protected]>
For the sake of symmetry. Signed-off-by: Attila Szakacs <[email protected]>
Signed-off-by: Attila Szakacs <[email protected]>
alltilla
force-pushed
the
syslog-ng-otlp
branch
from
a1bc6e6 to
78e3e9d
Compare
|
Signed-off-by: Attila Szakacs <[email protected]>
Signed-off-by: Attila Szakacs <[email protected]>
alltilla
force-pushed
the
syslog-ng-otlp
branch
from
78e3e9d to
42d1e10
Compare
|
|
Build FAILURE |
|
@kira-syslogng retest this please |
MrAnno
approved these changes
Aug 21, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
syslog-ng-otlp()source and destination helps to transfer the internal representation of a log message between syslog-ng instances. In contrary to thesyslog-ng()(ewmm()) drivers,syslog-ng-otlp()does not transfer the messages on simple TCP connections, but uses the OpenTelemetry protocol to do so.It is easily scalable (
workers()option), uses built-in application layer acknowledgement, out of the box supports google service authentication (ADC or ALTS), and gives the possibility of better load balancing.The performance is currently similar to
ewmm()(OTLP is ~30% quicker) but there is a source side limitation, which will be optimized. We measured 200-300% performance improvement with a PoC optimized code using multiple threads, so stay tuned.Note: The
syslog-ng-otlp()source is only an alias to theopentelemetry()source. This is useful for not needing to open different ports for the syslog-ng messages and other OpenTelemetry messages. The syslog-ng messages are marked with a@syslog-ngscope name and the current syslog-ng version as the scope version. Both sources will handle the incomingsyslog-ng messages as syslog-ng messages, and all other messages as simple OpenTelemetry messages.
Signed-off-by: Attila Szakacs [email protected]