Reflecting on the last year of deno_doc

[lucacasonato]

It is time to reflect on the last year of deno_doc: what is good, what is bad, and what needs improvements. The main pain point I currently see is this: the doc.deno.land website needs to do a lot of post processing on the deno doc --json output to get all the features it wants. I have written down all the features I want in deno doc directly as concrete goals below:

Goals

The high level goal of deno_doc is to take a JS / JS+tsdoc / TS source file, and extract all information from this file that is relevant for documentation of the module, and output it in a structured and easy to consume format. Specifically:

• The input should have to be no more than the root module, and all of it's (transitive) dependencies. All inputted modules consist of source code, a media type, and a fully resolved specifier.
• The output should be structured data that describes the signatures of all functions / types / interfaces declared / exported in that file.
  • Each component in this structured data should contain a reference to the module it is defined it, so a user can be deep linked there.
  • The structured data should not encode the actual implementation of code in a module.
  • The structured data should expose doc comments on declarations of fields of those as raw markdown, but without the jsdoc / tsdoc parameters (they should be exposed as structured data).
  • The structured data should be flat. There should be no hierarchical structure for individual modules or TypeScript namespaces. Nesting is fine for things like fields of classes / interfaces.
• The output of the structured data should be deterministic for a given input.
• The output format should require no extra computation to be processed into HTML with signature syntax highlighting, node deep linking (also across modules), or cleaning of documentation comments.
• The output format must be JSON serializable.
• deno_doc should support compilation to arm64 and x64 on Windows, macOS and Linux, and compilation to WASM.
• Module loading should be pluggable.
• Partial module graphs should be supported (ie let me generate an output for a module, even if some of its dependencies are unavailable).

[lucacasonato]

We should also consider direct HTML generation in Rust from the structured data. This should be a stretch goal for after we have this structured data though.

[vovacodes]

deno_doc should support compilation to arm64 and x64 on Windows, macOS and Linux, and compilation to WASM.

@lucacasonato I played with compiling deno_doc into wasm here: https://github.com/vovacodes/esmdoc it looks like it works just fine, see the deno test for the output wasm: https://github.com/vovacodes/esmdoc/blob/main/wasm_tests/tests.ts

I'm happy to bring it back into deno_doc if you guys interested.
There is a couple of decision to be made there though, like:

• What should be the public interface. So far I implemented just the function that returns raw JSON AST from parse_with_reexports.
• Should we rather return wasm_bindgen "shell" objects that lookup directly into the wasm memory rather, so we can skip potentially expensive serialization into JSON?
• Do we also need a function that returns html or do we want to let people generate their own markup based on the AST?

[kitsonk]

• The input should have to be no more than the root module, and all of it's (transitive) dependencies. All inputted modules consist of source code, a media type, and a fully resolved specifier.

We have integrated deno_graph now, I believe this is an effective implementation of this.

• The output should be structured data that describes the signatures of all functions / types / interfaces declared / exported in that file.

Could you elaborate more on from the DocNode[] what you expect? When I have consumed the doc nodes, I transform it into a collection where each kind appears in its own area, but that is only useful when doing certain things. When you are trying to find a node by name, and are uncertain what its kind is, or you just want to "render" one node, this can be quite hard to have it be structured.

• Each component in this structured data should contain a reference to the module it is defined it, so a user can be deep linked there.

I have proposed #140. Outside of TsTypeRef, I am not sure what else needs to be there. Especially when it comes to interfaces and namespace, they are "open ended" and don't have a definitive declaration point, so applying this everywhere could be problematic. Actually need to think about that in reference to #140 now.

• The structured data should not encode the actual implementation of code in a module.

Could you clarify this point more? I assume you mean that data should be the derived/inferred "type" information?

• The structured data should expose doc comments on declarations of fields of those as raw markdown, but without the jsdoc / tsdoc parameters (they should be exposed as structured data).

I agree with this, post processing of the parameters is yucky. Opened #141 with a proposal.

• The structured data should be flat. There should be no hierarchical structure for individual modules or TypeScript namespaces. Nesting is fine for things like fields of classes / interfaces.

I am sort of not 100% sure on this one. I guess flattening namespaces would be ok, though I don't specifically think of them as problematic if we solve the reference issue.

• The output format should require no extra computation to be processed into HTML with signature syntax highlighting, node deep linking (also across modules), or cleaning of documentation comments.

I generally agree with this, up to the point where it is still the domain of the printer to structure a "link". Like how to resolve to a global symbol is something the printer should really deal with, I agree "all" of the information needed should be there.

• deno_doc should support compilation to arm64 and x64 on Windows, macOS and Linux, and compilation to WASM.

The WASM support is merged and exposes a JavaScript API and is published on deno.land/x/

• Module loading should be pluggable.

This has been addressed via deno_graph integration, which supports a pluggable module loader.

• Partial module graphs should be supported (ie let me generate an output for a module, even if some of its dependencies are unavailable).

Agreed.