control-service: add executions logging api #259
Conversation
If a user installs a control serivce/versatile data kit by default they do not have access to execution logs. This is likely to be huge issue for adoptoin. Requiring that users integrate with tools like fluentd for logging or having them expose kubectl can be pretty hard. All APIs that support some kind of task execution naturally need to expose logs interface to users. Examples: - Lambda (using aws cloudwatch) - Data Bricks Jobs (API returns link to spark ui/logs) - airflow (rest api) - OpenFaas (has faas-cli logs) Initially it would be used - in vdk-server workflow - instead of having users use kubectl, let them use vdk execute --logs and we can remove any mention of kind/kubectl. - in vdk-heartbeat - root cause of vdk-heartbeat test can be very complex. After the test finishes it cleans after itself. And this causes all logs of that execution to disappear making it very hard to see why the data job started by the test has failed. Currently this is added as only experimental feature. There's some issues that need to be decided before it can be promoted to stable (will be noted as TODOs in the code) Signed-off-by: Antoni Ivanov <[email protected]>
...es_control_service/src/main/java/com/vmware/taurus/datajobs/DataJobsExecutionController.java
Outdated
Show resolved
Hide resolved
|
As far as I understand this API will behave as |
When/if there's a need for it. This will be experimental for some time and we'd see what feedback we get. Streaming would be a bit more complicated to implement currently. And |
…c/main/java/com/vmware/taurus/datajobs/DataJobsExecutionController.java
Agreed. I found some simple implementations of Streaming API that could be useful - https://technicalsand.com/streaming-data-spring-boot-restful-web-service/ |
Cool. Thanks. I was googling around and did not find anything interesting. I see I need to work on my google skills. |
If a user installs a control serivce/versatile data kit by default they
do not have access to execution logs. This is likely to be huge issue
for adoptoin. Requiring that users integrate with tools like fluentd for
logging or having them expose kubectl can be pretty hard.
All APIs that support some kind of
task execution naturally need to expose logs interface to users.
Examples:
Initially it would be used
them use vdk execute --logs and we can remove any mention of
kind/kubectl.
complex. After the test finishes it cleans after itself. And this causes
all logs of that execution to disappear making it very hard to see why
the data job started by the test has failed.
Currently this is added as only experimental feature. There's some
issues that need to be decided before it can be promoted to stable (will
be noted as TODOs in the code)
Signed-off-by: Antoni Ivanov [email protected]