-
Notifications
You must be signed in to change notification settings - Fork 59
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
control-service: passing data job arguments through execute API (#267)
why: As a user I want to be able to pass extra arguments to my data job executions when running the data job manually through the execute API, like I can when running the data job locally. This change enables users to pass arguments to data job executions through the control-service API for starting data job executions. what: Added functionality to the KubernetesService class. We add the extra arguments to the job container's command list. testing: added unit tests, ci/cd, manual testing - with debugger made sure that the command flow of added functionality is correct. Signed-off-by: Momchil Zhivkov [email protected]
- Loading branch information
Momchil Z
authored
Sep 28, 2021
1 parent
a79362f
commit ac7bd62
Showing
13 changed files
with
325 additions
and
45 deletions.
There are no files selected for viewing
23 changes: 23 additions & 0 deletions
23
...lines_control_service/src/main/java/com/vmware/taurus/exception/JsonDissectException.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| /* | ||
| * Copyright 2021 VMware, Inc. | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| package com.vmware.taurus.exception; | ||
|
|
||
| import org.springframework.http.HttpStatus; | ||
|
|
||
| public class JsonDissectException extends DomainError implements UserFacingError { | ||
| public JsonDissectException(Throwable cause) { | ||
| super("Control service failed to parse provided arguments to a valid JSON string.", | ||
| String.format("The internal parser threw an exception because: %s", cause.getMessage()), | ||
| "The requested system call will not complete.", | ||
| "Inspect the cause and re-try the call with arguments that can be parsed to JSON.", | ||
| cause); | ||
| } | ||
|
|
||
| @Override | ||
| public HttpStatus getHttpStatus() { | ||
| return HttpStatus.BAD_REQUEST; | ||
| } | ||
| } |
23 changes: 23 additions & 0 deletions
23
...l_service/src/main/java/com/vmware/taurus/exception/KubernetesJobDefinitionException.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| /* | ||
| * Copyright 2021 VMware, Inc. | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| package com.vmware.taurus.exception; | ||
|
|
||
| import org.springframework.http.HttpStatus; | ||
|
|
||
| public class KubernetesJobDefinitionException extends SystemError implements UserFacingError { | ||
| public KubernetesJobDefinitionException(String jobName) { | ||
| super(String.format("A configuration error with the current kubernetes job: '%s' definition was found.", jobName), | ||
| "Likely due to recent changes in the internal implementation.", | ||
| "The current call will not be processed.", | ||
| "Please open a new GitHub issue with the details of this error.", | ||
| null); | ||
| } | ||
|
|
||
| @Override | ||
| public HttpStatus getHttpStatus() { | ||
| return HttpStatus.INTERNAL_SERVER_ERROR; | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
58 changes: 58 additions & 0 deletions
58
...es_control_service/src/main/java/com/vmware/taurus/service/deploy/JobCommandProvider.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| /* | ||
| * Copyright 2021 VMware, Inc. | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| package com.vmware.taurus.service.deploy; | ||
|
|
||
| import com.fasterxml.jackson.core.JsonProcessingException; | ||
| import com.fasterxml.jackson.databind.ObjectMapper; | ||
| import org.springframework.stereotype.Component; | ||
|
|
||
| import java.util.List; | ||
| import java.util.Map; | ||
|
|
||
| /** | ||
| * This class provides the command list for the K8S' data job | ||
| * container. The command list executes the data job, by calling | ||
| * vdk run ... | ||
| */ | ||
| @Component | ||
| public class JobCommandProvider { | ||
| private List<String> command; | ||
|
|
||
| public JobCommandProvider() { | ||
|
|
||
| this.command = List.of( | ||
| "/bin/bash", | ||
| "-c", | ||
| "export PYTHONPATH=/usr/local/lib/python3.7/site-packages:/vdk/site-packages/ && /vdk/vdk run" | ||
| ); | ||
| } | ||
|
|
||
| private String getJobNameArgument(String jobName) { | ||
| return String.format(" ./%s", jobName); | ||
| } | ||
|
|
||
| private String getExtraArguments(String arguments) { | ||
| return String.format(" --arguments '%s'", arguments); | ||
| } | ||
|
|
||
| public List<String> getJobCommand(String jobName) { | ||
|
|
||
| return List.of( | ||
| command.get(0), | ||
| command.get(1), | ||
| command.get(2) + getJobNameArgument(jobName) | ||
| ); | ||
| } | ||
|
|
||
| public List<String> getJobCommand(String jobName, Map<String, Object> extraArguments) throws JsonProcessingException { | ||
| var arguments = new ObjectMapper().writeValueAsString(extraArguments); | ||
| return List.of( | ||
| command.get(0), | ||
| command.get(1), | ||
| command.get(2) + getJobNameArgument(jobName) + getExtraArguments(arguments) | ||
| ); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.