vdk-core: errors occurred and the state (handled or not) context missing #1182
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
User code may trigger a SDK feature in a way that raises a user error. In such a use case, the error could be handled by the cautious user, in runtime. The VDK platform should be aware if user errors are handled in runtime or not, so that affects the data job termination status. The old errors.BLAMEES (experimental data structure for keeping track of responsibles for errors priorly logged) should be refactored: * a resolvable context abstraction introduced * it is memory-managed (singleton, reasoning documented) * populated by the 3 user-facing `errors.log_*()` methods - in favour of a method designed to populate ErrorMessage entries * keeps track if the actual exception, along with all properties like who is the actual respobsible persona (user or platform, that was free-form str before), resolved by a data job step or not, etc. Testing Done: added a functional test covering the `simple-query-failed-handled` scenario, added singleton verification for the resolvable context; ci/cd Signed-off-by: ivakoleva <[email protected]>
projects/vdk-core/src/vdk/internal/builtin_plugins/termination_message/writer.py
Outdated
Show resolved
Hide resolved
User code may trigger a SDK feature in a way that raises a user error. In such a use case, the error could be handled by the cautious user, in runtime. The VDK platform should be aware if user errors are handled in runtime or not, so that affects the data job termination status. The old errors.BLAMEES (experimental data structure for keeping track of responsibles for errors priorly logged) should be refactored: * a resolvable context abstraction introduced * it is memory-managed (singleton, reasoning documented) * populated by the 3 user-facing `errors.log_*()` methods - in favour of a method designed to populate ErrorMessage entries * keeps track if the actual exception, along with all properties like who is the actual respobsible persona (user or platform, that was free-form str before), resolved by a data job step or not, etc. Testing Done: added a functional test covering the `simple-query-failed-handled` scenario, added singleton verification for the resolvable context; ci/cd Signed-off-by: ivakoleva <[email protected]>
User code may trigger a SDK feature in a way that raises a user error. In such a use case, the error could be handled by the cautious user, in runtime. The VDK platform should be aware if user errors are handled in runtime or not, so that affects the data job termination status. The old errors.BLAMEES (experimental data structure for keeping track of responsibles for errors priorly logged) should be refactored: * a resolvable context abstraction introduced * it is memory-managed (singleton, reasoning documented) * populated by the 3 user-facing `errors.log_*()` methods - in favour of a method designed to populate ErrorMessage entries * keeps track if the actual exception, along with all properties like who is the actual respobsible persona (user or platform, that was free-form str before), resolved by a data job step or not, etc. Testing Done: added a functional test covering the `simple-query-failed-handled` scenario, added singleton verification for the resolvable context; ci/cd Signed-off-by: ivakoleva <[email protected]>
murphp15
reviewed
Sep 26, 2022
projects/vdk-core/src/vdk/internal/builtin_plugins/termination_message/writer.py
Outdated
Show resolved
Hide resolved
antoniivanov
requested changes
Sep 26, 2022
projects/vdk-core/src/vdk/internal/builtin_plugins/run/file_based_step.py
Outdated
Show resolved
Hide resolved
|
I think it would be more efficient to have a quick talk in zoom. So schedule something or ping me on slack to sync face 2 face (or in these times zoom to zoom). |
what: Added the ability to cancel the remaining job/template steps through the job_input interface. why: This feature is required by users in certain cases e.g: if a data job depends on processing data from a source which has indicated no new entries since last run, then we can skip the execution. testing: Tested locally with data job/templates and added functional tests. Signed-off-by: Momchil Zhivkov [email protected]
updates: - [github.com/asottile/reorder_python_imports: v3.8.2 → v3.8.3](asottile/reorder-python-imports@v3.8.2...v3.8.3) - [github.com/asottile/pyupgrade: v2.38.0 → v2.38.2](asottile/pyupgrade@v2.38.0...v2.38.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* vdk-properties-fs: new plugin for local FS properties storage We need a simplistic plugin, that allows a developer or presenter to quickly store secrets on the local FS during a dev/demo session, that does not require a prerequisite of the entire Control Service installed and running. Added a FS properties client, that uses a customizable dir and filename on the local file system to store the properties key-values. It does support multiple teams and data jobs, the properties of those are nested under a dedicated key for deduplication. JSON format is used, enabling nested data structures. Testing Done: added `test_fs_properties_plugin.py`, that covers multiple data jobs using matching keys, and default/customized directory and filename for file storage on the local FS. Signed-off-by: ivakoleva <[email protected]> Signed-off-by: ivakoleva <[email protected]> Co-authored-by: Gabriel Georgiev <[email protected]>
closes #1177. Signed-off-by: murphp15 [email protected] Signed-off-by: murphp15 [email protected]
what: The template arguments validator now makes use of the skip_remaining_steps() method when an empty source view is returned. why: Users of the template requested that empty source view stopped resulting in a User Error, after a discussion it was agreed that this is the correct and desired behaviour. testing: this change depends on #1188 Edited failing regression tests. Signed-off-by: Momchil Zhivkov [email protected]
Occasionally, vdk may indicate that a query execution took 10 or 20 minutes, even though the DB back shows that the query was actually executes in seconds. This results in the DB backend issuing a timeout error and a job failure. Because it happens sporadically, it is hard to reproduce and debug locally. This change adds a log message just before the result of a query is fetched, and should help better understand at which step the query execution hanged. Testing done: Documentation change. Signed-off-by: Andon Andonov <[email protected]> Signed-off-by: Andon Andonov <[email protected]>
User code may trigger a SDK feature in a way that raises a user error. In such a use case, the error could be handled by the cautious user, in runtime. The VDK platform should be aware if user errors are handled in runtime or not, so that affects the data job termination status. The old errors.BLAMEES (experimental data structure for keeping track of responsibles for errors priorly logged) should be refactored: * a resolvable context abstraction introduced * it is memory-managed (singleton, reasoning documented) * populated by the 3 user-facing `errors.log_*()` methods - in favour of a method designed to populate ErrorMessage entries * keeps track if the actual exception, along with all properties like who is the actual respobsible persona (user or platform, that was free-form str before), resolved by a data job step or not, etc. Testing Done: added a functional test covering the `simple-query-failed-handled` scenario, added singleton verification for the resolvable context; ci/cd Signed-off-by: ivakoleva <[email protected]>
antoniivanov
approved these changes
Sep 29, 2022
Address review comments Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
force-pushed
the
person/ikoleva/resolvable-context
branch
from
65e772c to
9db8381
Compare
antoniivanov
pushed a commit
that referenced
this pull request
Oct 4, 2022
Re submit the change - #1182 - again. The only difference is that I renamed _get_eception_message to get_exception_message as not to break backwords compatability. I will merge this after -- User code may trigger a SDK feature in a way that raises a user error. In such a use case, the error could be handled by the cautious user, in runtime. The VDK platform should be aware if user errors are handled in runtime or not, so that affects the data job termination status. The old errors.BLAMEES (experimental data structure for keeping track of responsibles for errors priorly logged) should be refactored: * a resolvable context abstraction introduced * it is memory-managed (singleton, reasoning documented) * populated by the 3 user-facing `errors.log_*()` methods - in favour of a method designed to populate ErrorMessage entries * keeps track if the actual exception, along with all properties like who is the actual respobsible persona (user or platform, that was free-form str before), resolved by a data job step or not, etc. Testing Done: added a functional test covering the `simple-query-failed-handled` scenario, added singleton verification for the resolvable context; ci/cd Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
pushed a commit
that referenced
this pull request
Oct 6, 2022
Re submit the change - #1182 - again. The only difference is that I renamed _get_eception_message to get_exception_message as not to break backwords compatability. I will merge this after -- User code may trigger a SDK feature in a way that raises a user error. In such a use case, the error could be handled by the cautious user, in runtime. The VDK platform should be aware if user errors are handled in runtime or not, so that affects the data job termination status. The old errors.BLAMEES (experimental data structure for keeping track of responsibles for errors priorly logged) should be refactored: * a resolvable context abstraction introduced * it is memory-managed (singleton, reasoning documented) * populated by the 3 user-facing `errors.log_*()` methods - in favour of a method designed to populate ErrorMessage entries * keeps track if the actual exception, along with all properties like who is the actual respobsible persona (user or platform, that was free-form str before), resolved by a data job step or not, etc. Testing Done: added a functional test covering the `simple-query-failed-handled` scenario, added singleton verification for the resolvable context; ci/cd Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
pushed a commit
that referenced
this pull request
Oct 6, 2022
Re submit the change - #1182 - again. The only difference is that I renamed _get_eception_message to get_exception_message as not to break backwords compatability. I will merge this after -- User code may trigger a SDK feature in a way that raises a user error. In such a use case, the error could be handled by the cautious user, in runtime. The VDK platform should be aware if user errors are handled in runtime or not, so that affects the data job termination status. The old errors.BLAMEES (experimental data structure for keeping track of responsibles for errors priorly logged) should be refactored: * a resolvable context abstraction introduced * it is memory-managed (singleton, reasoning documented) * populated by the 3 user-facing `errors.log_*()` methods - in favour of a method designed to populate ErrorMessage entries * keeps track if the actual exception, along with all properties like who is the actual respobsible persona (user or platform, that was free-form str before), resolved by a data job step or not, etc. Testing Done: added a functional test covering the `simple-query-failed-handled` scenario, added singleton verification for the resolvable context; ci/cd Signed-off-by: Antoni Ivanov <[email protected]> vdk-core: test Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
pushed a commit
that referenced
this pull request
Oct 6, 2022
Re submit the change - #1182 - again. The only difference is that I renamed _get_eception_message to get_exception_message as not to break backwords compatability. I will merge this after -- User code may trigger a SDK feature in a way that raises a user error. In such a use case, the error could be handled by the cautious user, in runtime. The VDK platform should be aware if user errors are handled in runtime or not, so that affects the data job termination status. The old errors.BLAMEES (experimental data structure for keeping track of responsibles for errors priorly logged) should be refactored: * a resolvable context abstraction introduced * it is memory-managed (singleton, reasoning documented) * populated by the 3 user-facing `errors.log_*()` methods - in favour of a method designed to populate ErrorMessage entries * keeps track if the actual exception, along with all properties like who is the actual respobsible persona (user or platform, that was free-form str before), resolved by a data job step or not, etc. Testing Done: added a functional test covering the `simple-query-failed-handled` scenario, added singleton verification for the resolvable context; ci/cd Signed-off-by: Antoni Ivanov <[email protected]> vdk-core: test Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
pushed a commit
that referenced
this pull request
Oct 10, 2022
Re submit the change - #1182 - again. The only difference is that I renamed _get_eception_message to get_exception_message as not to break backwords compatability. I will merge this after -- User code may trigger a SDK feature in a way that raises a user error. In such a use case, the error could be handled by the cautious user, in runtime. The VDK platform should be aware if user errors are handled in runtime or not, so that affects the data job termination status. The old errors.BLAMEES (experimental data structure for keeping track of responsibles for errors priorly logged) should be refactored: * a resolvable context abstraction introduced * it is memory-managed (singleton, reasoning documented) * populated by the 3 user-facing `errors.log_*()` methods - in favour of a method designed to populate ErrorMessage entries * keeps track if the actual exception, along with all properties like who is the actual respobsible persona (user or platform, that was free-form str before), resolved by a data job step or not, etc. Testing Done: added a functional test covering the `simple-query-failed-handled` scenario, added singleton verification for the resolvable context; ci/cd Signed-off-by: Antoni Ivanov <[email protected]> vdk-core: test Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
added a commit
that referenced
this pull request
Oct 10, 2022
…ing (#1212) Re submit the change - #1182 - again. The only difference is that I renamed _get_eception_message to get_exception_message as not to break backwords compatability. I will merge this after -- User code may trigger a SDK feature in a way that raises a user error. In such a use case, the error could be handled by the cautious user, in runtime. The VDK platform should be aware if user errors are handled in runtime or not, so that affects the data job termination status. The old errors.BLAMEES (experimental data structure for keeping track of responsibles for errors priorly logged) should be refactored: * a resolvable context abstraction introduced * it is memory-managed (singleton, reasoning documented) * populated by the 3 user-facing `errors.log_*()` methods - in favour of a method designed to populate ErrorMessage entries * keeps track if the actual exception, along with all properties like who is the actual respobsible persona (user or platform, that was free-form str before), resolved by a data job step or not, etc. Testing Done: added a functional test covering the `simple-query-failed-handled` scenario, added singleton verification for the resolvable context; ci/cd Signed-off-by: Antoni Ivanov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
User code may trigger a SDK feature in a way that raises a user error. In such a use case, the error could be handled by the cautious user, in runtime. The VDK platform should be aware if user errors are handled in runtime or not, so that affects the data job termination status.
The old errors.BLAMEES (experimental data structure for keeping track of responsibles for errors priorly logged) should be refactored:
errors.log_*()methods - in favour of a method designed to populate ErrorMessage entriesTesting Done: added a functional test covering the
simple-query-failed-handledscenario, added singleton verificationfor the resolvable context; ci/cd
Signed-off-by: ivakoleva [email protected]