Reduce stack logging

[gregwebs]

Feature Request

Make it possible to reduce the logging of stack traces

Problem

After upgrading TiDB, it is now logging lots of stack traces from warnings. This can make for a lot more data in the log collection pipeline.

Alternatives

Don't log warnings. This does fix my issue, but it creates another one!

Solutions

• Only log stack traces at the error level. Note that PD is now only logging them at the Fatal level
• Allow the user to configure what level stack traces are logged at

Solutions to reduce stack trace output

• Only log the first line of the stack trace (at the warning level)
• Reduce the directory path that gets logged (currently it includes /home/jenkins/workspace/release_tidb_2.1-ga/)

Releasing

It would be great

• to see this change in a 2.1 release since it does seem like a change during the previous 2.1 releases
• for PD and TiDB have uniform behavior for this

[nolouch]

PTAL @lysu

[gregwebs]

@nolouch @lysu can we get pd and TiDB to agree on the behavior here?

My suggestion is

• always log stack traces for panics
• Don't ever log stack traces for warnings
• Don't log stack traces for errors

Errors should have a way to trace an error. in PD I tried to start creating operational traces rather than stack traces. That is, one should be able to understand what operations are happening that lead up to the error. The stack trace is occasionally useful, but usually less useful than an operational trace

The error message itself or unique error codes (this is something we need!) are usually good enough to locate the source position of the error

[gregwebs]

Actually, I am changing my mind: for now given our lack of error and operation codes, it may be best to log stack traces for errors for now. But I still think they should not be logged by default for warnings.