Add filter configuration to log appender

[wallezhang]

Is your feature request related to a problem? Please describe.
Agent will instrument all log appenders so that all logs will be exported to collector. I just want to export logs on specified appenders.

Describe the solution you'd like
I think we can add configuration to filter exporting logs by appender name or log level. Configurations maybe like otel.instrumentation.log.enabled-appender-names.

[rahuldimri]

Can I work on this request ?

[mateuszrzeszutek]

Let me fill in some of the requirements here.

I think we could introduce a set of new configuration properties for this:

• For javaagent instrumentations, something like otel.instrumentation.<instrumentation-name>.experimental.logger-name-pattern, where <instrumentation-name> is one of the instrumentation names. This will make it similar to the other logging instrumentation settings.
• For library instrumentations, a new configuration setting on appender/builder.
• The value of this property would need to be a valid java.util.regex.Pattern.
• If this property is not set log events are not filtered at all.
• If this property is set, log events that do not match this pattern will not emit any telemetry.

The following instrumentations need to be updated:

• JUL
• JBoss LogManager
• Log4j 1
• Log4j 2, both javaagent and library instrumentations
• Logback, both javaagent and library instrumentations

[mateuszrzeszutek]

@rahuldimri are you still up for this? Let me know if you still want to work on this story, I'll assign you.

[rahuldimri]

Yes let me give a try !!

[trask]

I was assuming this issue was about "appender name", so that you could configure one appender with the particular configuration that you want, and then ask the javaagent to pick up only logs that flow through that one appender?

[mateuszrzeszutek]

I was assuming this issue was about "appender name", so that you could configure one appender with the particular configuration that you want, and then ask the javaagent to pick up only logs that flow through that one appender?

Indeed, I somehow missed that and got fixated on logger name instead 🙈

In that case, this feature will only concern javaagent logging instrumentations; when you're using library appenders you can already configure them however you want.

How about a list-type configuration property namedotel.instrumentation.<instrumentation-name>.experimental.include-appenders (or perhaps ...exclude-appenders to match several ...exclude-methods configs we already have)?

[wallezhang]

I was assuming this issue was about "appender name", so that you could configure one appender with the particular configuration that you want, and then ask the javaagent to pick up only logs that flow through that one appender?

Indeed, I somehow missed that and got fixated on logger name instead 🙈

In that case, this feature will only concern javaagent logging instrumentations; when you're using library appenders you can already configure them however you want.

How about a list-type configuration property namedotel.instrumentation.<instrumentation-name>.experimental.include-appenders (or perhaps ...exclude-appenders to match several ...exclude-methods configs we already have)?

Can we use the same configuration in javaagent instrumentation?

[JessHolle]

So is the thought here to instead automatically add/apply OpenTelemetryAppender wherever specific other appenders are applied? Or to instrument those appenders code and check appender name rather than instrumenting loggers?

[trask]

Or to instrument those appenders code and check appender name rather than instrumenting loggers?

I'm thinking something like this could be nice.

Then as a configuration property, you could set the appender name that you want the javaagent to capture from, and if that configuration property is set, the javaagent could suppress the general instrumentation and only instrument that specific appender.

[JessHolle]

Or to instrument those appenders code and check appender name rather than instrumenting loggers?

I'm thinking something like this could be nice.

Then as a configuration property, you could set the appender name that you want the javaagent to capture from, and if that configuration property is set, the javaagent could suppress the general instrumentation and only instrument that specific appender.

I'd think a set of appender names would be ideal -- a Set.contains() check should be cheap enough and allow maximum flexibility, e.g. to capture all of the data the original app routed to several different log files.

[xiangtianyu]

Is there any progress on this issue recently?

[mateuszrzeszutek]

Hey @xiangtianyu ,
No, no changes.

[xiangtianyu]

Hey @xiangtianyu , No, no changes.

Is there any plan for this? Or is there any discussion about this?

[mateuszrzeszutek]

Is there any plan for this? Or is there any discussion about this?

I believe none of the maintainers of this project have cycles to handle that.

[trask]

I'd suggest having the configuration be a "map", where key is logger name/prefix and value is min level to capture for that logger name/prefix

[wallezhang]

I'd suggest having the configuration be a "map", where key is logger name/prefix and value is min level to capture for that logger name/prefix

I want to try this issue. Please assign to me.

[trask]

This is being worked on at the specification level now through a combination of

• Add scope configuration properties opentelemetry-configuration#140
• OTEP: Logger.Enabled opentelemetry-specification#4290

(and some followups will be needed still)