vdk-jupyter: add iPython extension #1482

duyguHsnHsn · 2023-01-06T10:02:56Z

What:
Add VDK iPython extension which using the StandaloneDataJob class loads job_input variable to a notebook. The job_input is created for the data job that the user specifies for example with giving the path to the job's parent directory.
You can see the README file to see how the extension is used.

Why: It is an essential feature for the Jupyter Integration

Tests: unit tests are introduced

Signed-off-by: Duygu Hasan [email protected]

for more information, see https://pre-commit.ci

antoniivanov · 2023-01-06T10:17:02Z

essential feature for the Jupyter Integration

This says nothing really. I hope you can see that. Why is it essential? Who is the user persona and how they'd use it and why it would be useful for them.

murphp15 · 2023-01-06T10:17:24Z

projects/vdk-plugins/vdk-ipython-ext/README.md

+To use the extension it must be firstly installed with pip as a python package.
+Then to load the extension in Jupyter the user should use:
+```
+%reload_ext vdk_ipython_ext


would it not look a bit nicer if it was just %reload_ext vdk

In this way I get some errors with the imports and I thing it would be good to have a little difference since it is not loading the whole vdk but only a small part of it

murphp15 · 2023-01-06T10:18:44Z

projects/vdk-plugins/vdk-ipython-ext/README.md

+
+The CI/CD is separated in two stages, a build stage and a release stage.
+The build stage is made up of a few jobs, all which inherit from the same
+job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).


you are missing 3.11 here

antoniivanov · 2023-01-06T10:19:42Z

projects/vdk-plugins/vdk-ipython-ext/setup.py

+    # Define entry point called "vdk.plugin.run" with name of plugin and module to act as entry point.
+    entry_points={"vdk.plugin.run": ["ipython-ext = vdk_ipython_ext"]},
+    classifiers=[
+        "Development Status :: 4 - Beta",


It should be pre-alpha or alpha at most at the current stage - https://github.com/vmware/versatile-data-kit/wiki/Coding-Standard#development-status

Bumps [graphql-java-extended-scalars](https://github.com/graphql-java/graphql-java-extended-scalars) from 19.1 to 20.0. - [Release notes](https://github.com/graphql-java/graphql-java-extended-scalars/releases) - [Commits](graphql-java/graphql-java-extended-scalars@19.1...20.0) --- updated-dependencies: - dependency-name: com.graphql-java:graphql-java-extended-scalars dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [byte-buddy](https://github.com/raphw/byte-buddy) from 1.12.19 to 1.12.20. - [Release notes](https://github.com/raphw/byte-buddy/releases) - [Changelog](https://github.com/raphw/byte-buddy/blob/master/release-notes.md) - [Commits](raphw/byte-buddy@byte-buddy-1.12.19...byte-buddy-1.12.20) --- updated-dependencies: - dependency-name: net.bytebuddy:byte-buddy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Antoni Ivanov <[email protected]>

Bumps [nimbus-jose-jwt](https://bitbucket.org/connect2id/nimbus-jose-jwt) from 9.25.6 to 9.27. - [Changelog](https://bitbucket.org/connect2id/nimbus-jose-jwt/src/master/CHANGELOG.txt) - [Commits](https://bitbucket.org/connect2id/nimbus-jose-jwt/branches/compare/9.27..9.25.6) --- updated-dependencies: - dependency-name: com.nimbusds:nimbus-jose-jwt dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: ivakoleva <[email protected]>

…trol-service/projects/pipelines_control_service (#1475) * build(deps): bump mockito-core Bumps [mockito-core](https://github.com/mockito/mockito) from 4.10.0 to 4.11.0. - [Release notes](https://github.com/mockito/mockito/releases) - [Commits](mockito/mockito@v4.10.0...v4.11.0) --- updated-dependencies: - dependency-name: org.mockito:mockito-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Antoni Ivanov <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* vdk-jobs-troubleshooting: Add thread-dump utility This change adds the implementation of a thread-dump utility to the vdk-jobs-troubleshooting plugin. The utility uses an http server, through which an administrator is able to force a stacktrace dump of all threads used by the python process of the data job. The server is bound to a port on the localhost, so to get the stacktrace, one needs to be attached to the data job pod. Testing Done: Added unit tests for the utility registry, and tested the plugin itself locally by running a simple data job and examining the execution logs. Signed-off-by: Andon Andonov <[email protected]> * vdk-jobs-troubleshooting: Add functional test Add functional test. Minor code refactoring. Signed-off-by: Dako Dakov <[email protected]> * vdk-jobs-troubleshooting: Address review feedback Add documentation. Remove redundant Optional type. Signed-off-by: Dako Dakov <[email protected]> * vdk-jobs-troubleshooting: remove unused import Remove unused import. Signed-off-by: Dako Dakov <[email protected]> * vdk-jobs-troubleshooting: remove unused import Remove unused import. Signed-off-by: Dako Dakov <[email protected]> * vdk-jobs-troubleshooting: Address review feedback Remove redundancies. Signed-off-by: Dako Dakov <[email protected]> Signed-off-by: Andon Andonov <[email protected]> Signed-off-by: Dako Dakov <[email protected]> Co-authored-by: ddakov <[email protected]>

Release the plugin. Signed-off-by: Dako Dakov <[email protected]> Signed-off-by: Dako Dakov <[email protected]>

Bumps [nimbus-jose-jwt](https://bitbucket.org/connect2id/nimbus-jose-jwt) from 9.27 to 9.28. - [Changelog](https://bitbucket.org/connect2id/nimbus-jose-jwt/src/master/CHANGELOG.txt) - [Commits](https://bitbucket.org/connect2id/nimbus-jose-jwt/branches/compare/9.28..9.27) --- updated-dependencies: - dependency-name: com.nimbusds:nimbus-jose-jwt dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Antoni Ivanov <[email protected]>

… README

for more information, see https://pre-commit.ci

duyguHsnHsn · 2023-01-06T12:08:49Z

Closing this PR bc it is a duplicate of #1483
and deleting the branch

duyguHsnHsn added 3 commits January 5, 2023 14:37

Add ipython extension

9e04bfe

changes on loading path in reload_job function

c1715a9

Add tests and edit README and requirements

9f4aded

vmwclabot added the cla-not-required label Jan 6, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

a210a86

for more information, see https://pre-commit.ci

murphp15 reviewed Jan 6, 2023

View reviewed changes

antoniivanov reviewed Jan 6, 2023

View reviewed changes

antoniivanov approved these changes Jan 6, 2023

View reviewed changes

dependabot bot and others added 9 commits January 6, 2023 13:20

vdk-jobs-troubleshooting: release the plugin (#1481)

440e245

Release the plugin. Signed-off-by: Dako Dakov <[email protected]> Signed-off-by: Dako Dakov <[email protected]>

Change development status, delete unnedded testjobs, small changes on…

50531c8

… README

[pre-commit.ci] auto fixes from pre-commit.com hooks

4b58c65

for more information, see https://pre-commit.ci

duyguHsnHsn closed this Jan 6, 2023

duyguHsnHsn deleted the person/hduygu/vdk-ipython-ext branch January 6, 2023 12:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vdk-jupyter: add iPython extension #1482

vdk-jupyter: add iPython extension #1482

duyguHsnHsn commented Jan 6, 2023

antoniivanov commented Jan 6, 2023 •

edited

Loading

murphp15 Jan 6, 2023

duyguHsnHsn Jan 6, 2023

murphp15 Jan 6, 2023

antoniivanov Jan 6, 2023

duyguHsnHsn commented Jan 6, 2023

vdk-jupyter: add iPython extension #1482

vdk-jupyter: add iPython extension #1482

Conversation

duyguHsnHsn commented Jan 6, 2023

antoniivanov commented Jan 6, 2023 • edited Loading

murphp15 Jan 6, 2023

Choose a reason for hiding this comment

duyguHsnHsn Jan 6, 2023

Choose a reason for hiding this comment

murphp15 Jan 6, 2023

Choose a reason for hiding this comment

antoniivanov Jan 6, 2023

Choose a reason for hiding this comment

duyguHsnHsn commented Jan 6, 2023

antoniivanov commented Jan 6, 2023 •

edited

Loading