Transpile (optional) TypeScript, support imports (package and relative) and schema #28

krassowski · 2022-01-02T03:36:28Z

This is a follow up to #26 based on the discussion held in there. It is also a first step towards #25 and it relevant to #5. The ultimate goal is to make all the self-contained examples from jupyterlab/extension-examples just work without modification, including:

menus from schema
examples with multiple typescript files

but excluding importing styles (ultimately we could handle those too).

In short, this PR uses TypeScript Compiler API to transpile the code into CommonJS understood by browsers (without type checking!) and TypeScript AST Transformer API with minor modifications:

require() calls are replaced by async require()
exports are defined as an object on top of the module and returned in the body of new async function which serves the purpose of module executor

The require function is provided to the executor and depending on the module it returns:

if the imported value comes from a known package (a hard-coded, auto-generated list), which is available at runtime, the value will be passed at runtime
the loaded module from another file retrieved with contents API; circular imports are not supported
otherwise, requirejs is used; to avoid breaking the flow an await is used (the module loading is now done in n asynchronous function to accommodate that); as before, this will only work for AMD modules - and no further transpilation nor transformation is applied to the code retrieved by requirejs. To allow retrieval without explicitly passing the CDN URL, a new setting is added with a default pointing to https://cdn.jsdelivr.net/npm.
the expression given by export default is returned at the end, which gives us the plugin object

The proposed changes are backwards compatible - if no export is found, the old loading mechanism will be used instead.

The code can, but does not have to, be typed. Users who scrupulously annotate variables with types will be rewarded by better completer suggestions (at least on Binder where LSP is automatically installed).

Punchlist:

propose deactivate function and command unregisteration to allow hot reloading and disabling of already loaded plugins (Dynamic extensions reloading: IPlugin.deactivate, Application.registerPlugin and Application.deactivatePlugin proposal lumino#278)
consider systemjs as discussed in Support other loader mechanisms? #1?
- I considered it, and it seems very promising; I can look into this in the future but the bank holidays ended, so I won't have time to investigate it this month (happy to review if someone else picks up),
add submodule for testing extension-examples?
allow to hide the plugin playground launcher icon via setting
ask user to confirm before first require() call to a CDN that they like the URL that is configured and accept the responsibility for loading arbitrary packages from web? This could be a setting allowCDN with values user-has-not-decided, insecure-always (always, but mention insecure to highlight the dangers), never, ~~only-trusted-packages~~ (see There should be a way to verify subresource integrity #30)

Follow up items:

maybe randomise the name of the import function in the transpiled code to avoid users using it as if it was an API?
integrate SystemJS in the resolver as loader for ES6 modules (and CSS, and other resources)
add tests:
- jest: transformer is easily testable with unit tests, but it might be replaced in a subsequent PR by a babel-based transformer if we integrate with SystemJS so I don't feel like writing it now
- aspiration: run the same exact visual test as in extension-examples on those example; this is a big task and I would prefer to do this in a subsequent PR
support SRI: There should be a way to verify subresource integrity #30

Additionally:

plugins with the same ID can now be loaded multiple times
the addition of LSP is carried over from Add LSP to the Binder image #26
error handling was added to display useful error messages to users when any of the steps: loading, imports, or activation fail
once JupyterLab 3.3 gets released, the file editor will get a button to run the code as an extension
an icon to create and open a new TypeScript file is now displayed in the launcher

and when pressed, it creates and opens a pre-populated minimal hello word plugin.

krassowski · 2022-01-02T03:38:31Z

👈 Launch a binder notebook on branch krassowski:transpile-tokens

…r-widgets as a special case

src/index.ts

and reformat the main module into a class

jtpio · 2022-01-04T08:10:35Z

Thanks @krassowski this is looking great!

add submodule for testing extension-exmaples?

Or we pull the repo when creating the Binder, in postBuild? An alternative would be to use the released version of jupyterlab-plugin-playground on the Binder of the examples repo.

…migration

while we are left with non-pretty replacements of `require` with `await require`, maybe those could be performed in an `after` transformer in the future. The advantage of this approach is in lower maintenance burden (less to rewrite on each TypeScript major version bump) and a working implementation of exports, over quite complex custom transformer which was not fully functional for some edge cases. The disadvantage is that the transpiled code looks less like the original making debugging harder for plugin users.

krassowski · 2022-01-09T16:22:42Z

This is now ready for review (and hopefully a merge).

There was a major change in 0599fbc (followed up by cleanup in c1b011b): transpilation module target is now CommonJS as it turned out to be much easier to work with for full exports support (which is required for support of multiple files) - we just need to add await in front of require, define exports object and return it, instead of manually extracting this information. The net change is -321 LoC, and it this should be much more maintainable now.

Also, with newly added multi-file and schema support most examples in extension-examples now work out of the box. There are ~~two~~ three loaders which are missing:

CSS styles
SVG strings
react (tsx files)

the first can be easily added in a follow up PR; the tsc compiler also supports react but it might require some more changes (I don't know yet).

Ideally we would require a `.d.ts`, but for a dynamic playground I think it is fine to accept svg based on extension

krassowski · 2022-01-09T20:05:42Z

I added support for React/tsx as as it was just a simple switch in tsc config. I don't plan to work on adding any more features. I tested most of the extensions and they just work :)

There is a corner case of some extension examples using default import for React. It works fine if React is imported using:

import * as React from 'react';

but fails when it is imported using a default import as in:

import React from 'react';

Some extension examples do the former (e.g. kernel-messaging) and some the latter (e.g. custom-log-console). This comes down to allowSyntheticDefaultImports/esModuleInterop and can be easily fixed by exposing the namespace as default when default is not present (so re-implementing synthetic default).

Addresses jupyterlab#28 (comment)

krassowski · 2022-01-09T20:33:12Z

There is a corner case of some extension examples using default import for React

Fixed in 580c723.

meeseeksmachine · 2022-01-27T10:15:15Z

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/what-is-the-2022-way-to-display-javascript-in-a-python-notebook-in-jupyter-lab/12318/5

echarles

Great work @krassowski . This PR works for me (tested also with a few examples from https://github.com/jupyterlab/extension-examples). I have left a few comments.

src/index.ts

echarles · 2022-01-27T13:34:58Z

src/index.ts

+    let result;
+    try {
+      result = await pluginLoader.load(code, path);
+    } catch (error) {


Would be good to show a dialog in case of success.

Maybe. Ideally this would not be a dialog (which could be disruptive) but a notification. I guess we could defer until the notifications are in core.

echarles · 2022-01-27T13:35:21Z

src/index.ts

-    await app.activatePlugin(plugin.id);
+    // Unregister plugin if already registered.
+    if (this.app.hasPlugin(plugin.id)) {
+      delete (this.app as any)._pluginMap[plugin.id];


The leaf/dependent plugins are not unloaded (see jupyterlab/lumino#278 (comment)). I guess this is ok for now

echarles · 2022-01-27T13:36:16Z

src/modules.ts

@@ -0,0 +1,107 @@
+import * as jupyter_widgets_base from '@jupyter-widgets/base';


Could this list of predefined modules be dynamic (maybe a feature for later?)

src/dialogs.tsx

echarles · 2022-01-27T13:39:00Z

src/index.ts

+      } catch (e) {
+        showDialog({
+          title: `Plugin autostart failed: ${(e as Error).message}`,
+          body: formatErrorWithResult(e, result)


My VS Code complains with Argument of type 'unknown' is not assignable to parameter of type 'Error' . However, no issue when building from the CLI.

Maybe VS Code is using a different or more version of typescript?

You may have useUnknownInCatchVariables enabled (or strict). It is not enabled in main JupyterLab repo, but we could enable it here.

README.md

Co-authored-by: Eric Charles <[email protected]>

jtpio · 2022-02-16T08:47:47Z

Looks like there is a small conflict to fix in yarn.lock, and a couple of inline comments.

Otherwise it works great on Binder, and we should consider getting this in soon!

Co-authored-by: Jeremy Tuloup <[email protected]>

krassowski · 2022-03-23T19:28:34Z

I would say ready to be merged.

jtpio

Really nice, thanks!

jtpio · 2022-03-24T07:47:03Z

Let's get this in an iterate 🚀

jtpio · 2022-03-24T08:00:09Z

Starting a new release with this change.

krassowski added 2 commits January 2, 2022 02:28

Implement transpilation with import transformation

5ae0222

Update the list of packages

cad665e

krassowski added 3 commits January 2, 2022 12:04

Fix typos

4fc3bad

Catch and display activation errors, handle unknown tokens and jupyte…

2d53c90

…r-widgets as a special case

Host imports to main module, handle import errors

dce0d5f

krassowski commented Jan 2, 2022

View reviewed changes

src/index.ts Outdated Show resolved Hide resolved

Improve export transformer and error handling

5f519e2

krassowski mentioned this pull request Jan 2, 2022

Integrate with jupyterlab/extension-examples #25

Open

krassowski added 6 commits January 2, 2022 17:38

Prepare for aliased import (import-as) handling

3f1b003

Split up transpiler, resolver and loader

fc507fc

Check for existence of module properties

32a40d2

Warn, but do not raise because types will be undefined.

93cc1fa

Implement visit import aliasing and add missing transpiler

8f5be22

Support default imports, recognise type imports

1131f84

This was referenced Jan 2, 2022

Support other loader mechanisms? #1

Open

Dynamic extensions reloading: IPlugin.deactivate, Application.registerPlugin and Application.deactivatePlugin proposal jupyterlab/lumino#278

Closed

krassowski added 5 commits January 3, 2022 17:35

Clean up comments

81d4211

Use local copy of require.js, do not expose it in global window

ef7c8a5

and reformat the main module into a class

Add webpack config with workaround for microsoft/TypeScript#39436

b07d1eb

Update MANIFEST.in

9788f27

Update the fallback warning text

aa85c07

krassowski marked this pull request as ready for review January 4, 2022 03:57

jtpio added the enhancement New feature or request label Jan 4, 2022

jtpio mentioned this pull request Jan 4, 2022

Release with the Jupyter Releaser #29

Closed

krassowski added 4 commits January 5, 2022 18:36

Pass requirejs to the object-based plugins; document deprecation and …

81f0921

…migration

Correct default imports

abc6335

Allow to hide the icon in launcher

370fd89

Use more expressive setting name

dd7b6fd

krassowski added 4 commits January 9, 2022 14:14

Expose @jupyterlab/outputarea

bbd00cb

Fix require.js loading (requires the iframe to stay in DOM!)

fbc5b39

Implement schema loading

23ea3cf

krassowski changed the title ~~Transpile (optional) TypeScript, support tokens and package imports~~ Transpile (optional) TypeScript, support imports (package and relative) and schema Jan 9, 2022

krassowski added 2 commits January 9, 2022 16:10

Implement proper after transformers

c1b011b

Update README re support for relative imports/schema/styles/svg

e3baee9

Update setting name in README, remove irrelevant line

6527789

krassowski mentioned this pull request Jan 9, 2022

There should be a way to verify subresource integrity #30

Open

krassowski added 2 commits January 9, 2022 18:52

Add React and .tsx support

8a2ea13

Support SVG imports as a special case

ef20b38

Ideally we would require a `.d.ts`, but for a dynamic playground I think it is fine to accept svg based on extension

Implement default proxy for known modules without default export

580c723

Addresses jupyterlab#28 (comment)

echarles reviewed Jan 27, 2022

View reviewed changes

jtpio reviewed Feb 16, 2022

View reviewed changes

README.md Outdated Show resolved Hide resolved

jtpio reviewed Feb 16, 2022

View reviewed changes

README.md Outdated Show resolved Hide resolved

Update src/dialogs.tsx

040d344

Co-authored-by: Eric Charles <[email protected]>

krassowski and others added 3 commits March 23, 2022 18:46

Apply suggestions from code review

62d2308

Co-authored-by: Jeremy Tuloup <[email protected]>

Merge branch 'master' into transpile-tokens

0eb3c53

Lint with prettier

ad012b1

jtpio approved these changes Mar 24, 2022

View reviewed changes

jtpio merged commit d292779 into jupyterlab:master Mar 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transpile (optional) TypeScript, support imports (package and relative) and schema #28

Transpile (optional) TypeScript, support imports (package and relative) and schema #28

krassowski commented Jan 2, 2022 •

edited

Loading

krassowski commented Jan 2, 2022

jtpio commented Jan 4, 2022

krassowski commented Jan 9, 2022 •

edited

Loading

krassowski commented Jan 9, 2022

krassowski commented Jan 9, 2022

meeseeksmachine commented Jan 27, 2022

echarles left a comment

echarles Jan 27, 2022

krassowski Mar 23, 2022

echarles Jan 27, 2022

echarles Jan 27, 2022

echarles Jan 27, 2022 •

edited

Loading

jtpio Feb 16, 2022

krassowski Mar 23, 2022

jtpio commented Feb 16, 2022

krassowski commented Mar 23, 2022

jtpio left a comment

jtpio commented Mar 24, 2022

jtpio commented Mar 24, 2022

		@@ -0,0 +1,107 @@
		import * as jupyter_widgets_base from '@jupyter-widgets/base';

Transpile (optional) TypeScript, support imports (package and relative) and schema #28

Transpile (optional) TypeScript, support imports (package and relative) and schema #28

Conversation

krassowski commented Jan 2, 2022 • edited Loading

krassowski commented Jan 2, 2022

jtpio commented Jan 4, 2022

krassowski commented Jan 9, 2022 • edited Loading

krassowski commented Jan 9, 2022

krassowski commented Jan 9, 2022

meeseeksmachine commented Jan 27, 2022

echarles left a comment

Choose a reason for hiding this comment

echarles Jan 27, 2022

Choose a reason for hiding this comment

krassowski Mar 23, 2022

Choose a reason for hiding this comment

echarles Jan 27, 2022

Choose a reason for hiding this comment

echarles Jan 27, 2022

Choose a reason for hiding this comment

echarles Jan 27, 2022 • edited Loading

Choose a reason for hiding this comment

jtpio Feb 16, 2022

Choose a reason for hiding this comment

krassowski Mar 23, 2022

Choose a reason for hiding this comment

jtpio commented Feb 16, 2022

krassowski commented Mar 23, 2022

jtpio left a comment

Choose a reason for hiding this comment

jtpio commented Mar 24, 2022

jtpio commented Mar 24, 2022

krassowski commented Jan 2, 2022 •

edited

Loading

krassowski commented Jan 9, 2022 •

edited

Loading

echarles Jan 27, 2022 •

edited

Loading