Add token preprocessor hook and URL query parameters #33

wrapperup · 2024-02-21T03:19:23Z

This adds a new lower-level hook called tokenPreprocessors to Environment that lets plugins transform a template's tokens into other tokens. It's like tags, except it isn't limited to a tag, and you output tokens.

Also adds support for URL query parameters. The sanitized path is passed to the loader, and the full path (with query strings) is passed to the token preprocessor, which unlocks a few capabilities.

Motivation

One use for this is to add template fragments for the htmx library (https://htmx.org/essays/template-fragments/). Basically, you can mark up portions of the template that can be rendered individually. For example:

Template Fragments Example Plugin

function fragmentPreprocessor(
  env: Environment,
  tokens: Token[],
  path?: string
): Token[] | undefined {
  if (!path) {
    return;
  }

  const splitPath = path.split("#");
    
  if (splitPath.length !== 2) {
    return;
  }

  const [_, fragment] = splitPath;

  let indexStart = 0;
  while (indexStart < tokens.length) {
    const token = tokens[indexStart];
    if (token[0] === "tag" && token[1] === "fragment " + fragment) {
      // Skip the fragment tag
      indexStart++;
      break;
    }
    indexStart++;
  }

  if (indexStart >= tokens.length) {
    throw new Error("Couldn't find fragment tag");
  }

  let indexEnd = indexStart;

  while (indexEnd < tokens.length) {
    const token = tokens[indexEnd];
    if (token[0] === "tag" && token[1] === "/fragment") {
      const fragment = tokens.slice(indexStart, indexEnd);
      return fragment;
    }
    indexEnd++;
  }

  throw new Error("Couldn't find end of fragment tag");
}

function fragmentTag(
  env: Environment,
  code: string,
  output: string,
  tokens: Token[],
): string | undefined {
  const match = code?.match(/^fragment (.+)$/);

  if (!match) {
    return;
  }

  const compiled: string[] = [];
  const compiledFilters = env.compileFilters(tokens, output);

  compiled.push(...env.compileTokens(tokens, output, ["/fragment"]));
  compiled.push(`${output} = ${compiledFilters};`);

  if (tokens.length && (tokens[0][0] !== "tag" || tokens[0][1] !== "/fragment")) {
    throw new Error(`Missing closing tag for fragment: ${code}`);
  }

  tokens.shift();

  return compiled.join("\n");
}


export function fragmentPlugin() {
  return (env: Environment) => {
    env.tokenPreprocessors.push(fragmentPreprocessor);
    env.tags.push(fragmentTag);
  }
}

Lets you write

<body>
  <div hx-target="this">
  {{ fragment button }}
    <button hx-get="/data">Refresh Data: {{ data }}</button>
  {{ /fragment }}
  </div>
</body>

and if I add this "#button" query parameter,

const frag = await env.load("my-template.vto#button");

my template now renders just the button:

<button hx-get="/data">Refresh Data: 10</button>

Design questions

Does having the user-defined "context" object make sense here? Alternatively, passing the filepath down to plugins and supporting URL query parameters in loaders would work too (and trivializes caching).
Any preferred name for "preprocessor" (or other types)? jinja2 calls their token processor hook "filter_tokens" for example.

Thanks.

TODO

Implement with cache
Document

oscarotero · 2024-02-21T16:00:39Z

If I understand correctly, you want to run preprocessors to modify the tags before the compilation in order to transform, filter or generate more tags.
I'm not sure about your use case (template fragments). It's easier with exports and imports tags.

And what you are proposing is the ability to load a template multiple times with different options

// Load with a fragment
const frag = await env.load("my-template.vto", undefined, { fragment: "button" });

// Load with another fragment
const frag = await env.load("my-template.vto", undefined, { fragment: "other" });

I don't think it's a good idea, it makes the cache invalidation more complicated, and it's confusing if you're importing the same file multiple times in your templates with different configurations.

The preprocessor idea could be useful for some use cases, but I don't like the CompileContext.
I think if it's implemented, it should be applied only once, after the template tokenization and without dynamic options.

wrapperup · 2024-02-21T17:35:10Z

If I understand correctly, you want to run preprocessors to modify the tags before the compilation in order to transform, filter or generate more tags. I'm not sure about your use case (template fragments). It's easier with exports and imports tags.

Right, though this particular use case it's a lot more convenient to do it this way. Otherwise, you'd need to have some additional runtime conditions or split it off into a new file.

And what you are proposing is the ability to load a template multiple times with different options

I don't think it's a good idea, it makes the cache invalidation more complicated, and it's confusing if you're importing the same file multiple times in your templates with different configurations.

The preprocessor idea could be useful for some use cases, but I don't like the CompileContext. I think if it's implemented, it should be applied only once, after the template tokenization and without dynamic options.

Agreed, not really a fan of context either. It definitely complicates caching, so instead I think supporting query parameters in the path like so:

const frag = await env.load("my-template.vto#fragment");

Would be a better approach, and can it be cached easily since it's just part of the path. Older versions of Nunjucks supported something like this (but that got lost in translation a few years ago unfortunately). However, the file loader would need to support it and trim off the query params. If you like, we could split that off into another PR, since preprocessing by itself is already pretty useful.

oscarotero · 2024-02-21T18:14:39Z

Okay. I think this can be split in two different steps:

Step 1: Allow query params and fragments in the file path. /template.vto?name=value#foo. Vento should clean this part before passing the path to the file loader (for backward compatibility).
Step 2: Implement the preprocessor system, which would be a function accepting 3 variables. the environment instance, the array of tokens and the complete filename (with query params and fragments).

what do you think?

wrapperup · 2024-02-21T18:59:51Z

Okay. I think this can be split in two different steps:

Step 1: Allow query params and fragments in the file path. /template.vto?name=value#foo.
Vento should clean this part before passing the path to the file loader (for backward compatibility).

Step 2: Implement the preprocessor system, which would be a function accepting 3 variables.
the environment instance, the array of tokens and the complete filename (with query params and fragments).

what do you think?

Yep, perfect. I'll try that.

wrapperup · 2024-02-21T19:35:51Z

I also changed the name of the hook to "tokenPreprocessors" to make it a bit more specific, and to leave room for "preprocessors" in the future (ie. minify HTML directly from source, which isn't easy to do in a token preprocessor I'd imagine).

oscarotero · 2024-02-22T00:18:28Z

It looks great. Thank you!

wrapperup changed the title ~~Add token preprocessor hooks and plugin context~~ Add token preprocessor hook and plugin context Feb 21, 2024

add token preprocessor hook, trim out query strings in load path

6a76854

wrapperup force-pushed the preprocessor-and-context branch from 95d8d40 to 6a76854 Compare February 21, 2024 19:27

wrapperup changed the title ~~Add token preprocessor hook and plugin context~~ Add token preprocessor hook and URL query parameters Feb 21, 2024

wrapperup marked this pull request as ready for review February 21, 2024 19:33

oscarotero merged commit 627b42f into ventojs:main Feb 22, 2024
1 check failed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add token preprocessor hook and URL query parameters #33

Add token preprocessor hook and URL query parameters #33

wrapperup commented Feb 21, 2024 •

edited

Loading

oscarotero commented Feb 21, 2024

wrapperup commented Feb 21, 2024 •

edited

Loading

oscarotero commented Feb 21, 2024

wrapperup commented Feb 21, 2024 •

edited

Loading

wrapperup commented Feb 21, 2024

oscarotero commented Feb 22, 2024

Add token preprocessor hook and URL query parameters #33

Add token preprocessor hook and URL query parameters #33

Conversation

wrapperup commented Feb 21, 2024 • edited Loading

Motivation

Design questions

TODO

oscarotero commented Feb 21, 2024

wrapperup commented Feb 21, 2024 • edited Loading

oscarotero commented Feb 21, 2024

wrapperup commented Feb 21, 2024 • edited Loading

wrapperup commented Feb 21, 2024

oscarotero commented Feb 22, 2024

wrapperup commented Feb 21, 2024 •

edited

Loading

wrapperup commented Feb 21, 2024 •

edited

Loading

wrapperup commented Feb 21, 2024 •

edited

Loading