vdk-ingest-http: reduce verbosity of ingestion logs

[antoniivanov]

Logging each payload at INFO is extremely verbose, we can have thousands
and thousands of payloads being sent. So it also slows down teh
ingestion.

Signed-off-by: Antoni Ivanov [email protected]

[ivakoleva]

ltgm;

On the logging topic, we have ~50 vdk built-in/internal loggers; maybe we may consider a plugin for (text-)configurable logging level per logger. The use case is, a data analyst would like to increase vdk.internal.builtin_plugins.ingestion logging when developing on local machine. So when configured, the vdk config will be stored, and applied to each data job local run.

[antoniivanov]

ltgm;

On the logging topic, we have ~50 vdk built-in/internal loggers; maybe we may consider a plugin for (text-)configurable logging level per logger. The use case is, a data analyst would like to increase vdk.internal.builtin_plugins.ingestion logging when developing on local machine. So when configured, the vdk config will be stored, and applied to each data job local run.

Yes. I've thought about this. we currently wehave LOG_LEVEL_VDK config -

versatile-data-kit/projects/vdk-core/src/vdk/internal/builtin_plugins/config/vdk_config.py

Line 64 in 1469b93

config_builder.add(

But maybe we can make it more genric -e.g LOG_LEVEL=vdk.api=INFO,vdk.internal=WARNING
(Similar to how in spring you can set logging.level.com.domain=INFO