Jekyll header as requested by #22

[TeXhackse]

Since I currently don't have that much free time I just cleaned it up a bit but it still lacks some adjustments.

First of all I currently cannot get it working directly from the LaTeX compiler but only via the cli-mode. I will have a look at this after finishing todays work.

Beside this it currently needs a blank line before the jekyll block, which should also be adjusted. Maybe it's even usefull to only allow it at the beginning of the a markdown block. Since you are more familiar with the package structure feel free to adjust. Otherwise I will do my best to continue with thsi.

[Witiko]

@TeXhackse Thank you for your work! I have added a review with some notes for future work. Generally, I am willing to accept the PR after we've added support for at least continued lines, sequences and mappings. The goal is not to implement the entire YAML spec, but at least to be compatible with what Pandoc can parse.

[TeXhackse]

thanks, this helps a lot! I will take care of this and continue :)

[Witiko]

Please, do let me know if I can help in some way.

[TeXhackse]

@Witiko Thanks, I'd do so. sadly had to struggle with some health issues, but getting better, already started with some things. Only thing I am not quite sure how to resolve is how to only allow the Renderer to be applied to the begin of the Markdown input. If you got a hint on that it might be helpful, otherwise I finish the other parts first.

[Witiko]

I would put the corresponding parser at the beginning of what the Blocks non-terminal symbol expands to (after larsers.Blank^0):

markdown/markdown.dtx

Line 16730 in 5b1625f

Blocks = larsers.Blank^0 * parsers.Block^-1

This way, the Jekyll header parser can only be entered at the beginning of a Markdown document.

[TeXhackse]

ah thanks :)

[Witiko]

ah thanks :)

There is one additional piece to the puzzle: Although it is possible that blockquotes, footnotes, and definition list items can be described by a PEG grammar, we cheat a little and we parse them recursively using the larsers.parse_toplevel_blocks method. Therefore, if we want to prohibit Jekyll data at the beginning of blockquotes, footnotes, and definition list items, we will need to set blocks_toplevel_t.JekyllData = fail here:

markdown/markdown.dtx

Line 16988 in c319b6b

local blocks_toplevel_t = util.table_copy(syntax)

[TeXhackse]

@Witiko I tried to return to my piece of this (sorry, my health is staying unstable… ), But I don't completly understand your last comment. Is there a way to explain it differently? I'd really love to finish that one.

[Witiko]

@Witiko I tried to return to my piece of this (sorry, my health is staying unstable… ), But I don't completly understand your last comment. Is there a way to explain it differently? I'd really love to finish that one.

Some parts of markdown are parsed recursively (such as blockquotes) rather than as part of a PE grammar. That is what I described as cheating.

Interestingly, it's unknown whether there exists a PE grammar for markdown without cheating. That is also why there are implementations and natural language specifications for markdown but no formal grammar specifications: markdown is notoriously difficult to parse.

Therefore, unless we want to allow Jekyll front matter at the beginning of blockquotes and other elements, we will need to disable it in the recursive invocations, as described in #77 (comment).

[TeXhackse]

Alright, I added that adjustment, and tested on current state of rebase on main. So… as far as my understanding works this should be resolved … But please let me know if there are some other secrets which I did not yet understand.

[TeXhackse]

Oh cool :) concerning the hooks. I will try to find some time during the weekend to have a look at, but probably not before sunday, has been a hard week.

[Witiko]

However, maybe the automatic \AddToHook{begindocument/end}{\maketitle} should be optional, I think its a bit surprising or needs customization before being useful.

@krono Customization is expected. The users can freely alter the definition of the markdown/jekyllData key-values as well as the jekyllDataBegin and jekyllDataEnd token renderers, which we already do to typeset the documentation. In the extreme case, the users can just scrap our LaTeX3 implementation and come up with their own on top of the low-level TeX token renderer macros.

[krono]

@krono Customization is expected. The users can freely alter the definition of the markdown/jekyllData key-values as well as the jekyllDataBegin and jekyllDataEnd token renderers. This will be documented in the LaTeX interface part of the technical documentation, which I have yet to write. In the extreme case, the users can just scrap our LaTeX3 implementation down and come up with their own on top of the low-level TeX token renderer macros.

I understand that; however, the markdown docu is already quite extensive (and also very detailed and good at that!), and the customization feels like one step beond the normal user.

So something like this would be helpful:

“If you use jekyll frontmatter, in particular with author and title keys, the title will be typeset automatically via \maketitle. This can be disable with the markdownForbFnorbActualNAme switch”

:)

[Witiko]

So something like this would be helpful:

“If you use jekyll frontmatter, in particular with author and title keys, the title will be typeset automatically via \maketitle. This can be disable with the markdownForbFnorbActualNAme switch”

@krono In the original implementation, @TeXhackse had a boolean key called maketitle, which was true by default and could be disabled by typing maketitle: false in YAML. Perhaps we should keep (and document) that key. 🙂

[krono]

In the original implementation, @TeXhackse had a boolean key called maketitle, which was true by default and could be disabled by typing maketitle: false in YAML. Perhaps we should keep that key.

That sounds very interesting!
Mabye that shifts the responsibility to the author of the markdown, who might not be the same person editing the LaTeX.
I would be fond of:

\markdownSetup{
  jekyllDataAutoMaketitle = true, % or false...
}

or something…

[Witiko]

Maybe that shifts the responsibility to the author of the markdown, who might not be the same person editing the LaTeX.

@krono The LaTeX person can arbitrarily change the key-values and therefore disable the maketitle key if they please. However, you make a good point that perhaps maketitle: true is something that does not belong to a Markdown document, since it is not related to the content but to the design.

I would be fond of:

\markdownSetup{
  jekyllDataAutoMaketitle = true, % or false...
}

or something…

At the moment, you would do the following: \markdownSetup { renderers = { jekyllDataEnd = {} } }.

[krono]

[…]

That sounds very interesting!
Maybe that shifts the responsibility to the author of the markdown, who might not be the same person editing the LaTeX.

The LaTeX person can arbitrarily change the key-values and therefore disable the maketitle key if they please.

Yes; but see below.

However, you make a good point that perhaps maketitle: true is something that does not belong to the Markdown document, since it is not related to the content but to the design.

exactly :)

[…]
At the moment, you would do the following: \markdownSetup { renderers = { jekyllDataEnd = {} } }.

I understand that. What I mean, that not every markdown-LaTeX user might want to understand the ramification of the renderers= parts, and perhaps they just want to avoid things magically appearing.

On the otherhand, it would be confusing, if you had

\markdownSetup { 
  renderers = { jekyllDataEnd = {} },
  jekyllDataAutoMaketitle = true
}

people might expect the \maketitle to appear anyway…
hmm :)

[Witiko]

I understand that. What I mean, that not every markdown-LaTeX user might want to understand the ramification of the renderers= parts, and perhaps they just want to avoid things magically appearing.

@krono Here is my take: If the user includes YAML metadata, they expect something to happen. At the moment, the package assumes that in 90% cases, that something will be the typesetting of the metadata on the title page. Of course, the defaults will never please everybody, so the remaining 10% can either reach for Markdown themes (if they don't want to program) or they can redefine the default renderers.

On the otherhand, it would be confusing, if you had

\markdownSetup { 
  renderers = { jekyllDataEnd = {} },
  jekyllDataAutoMaketitle = true
}

people might expect the \maketitle to appear anyway…
hmm :)

The Markdown options are quite low-level and typically map either to the Lua options (enabling different syntax extensions), or to the token renderers (changing how things look). In contrast, options such as jekyllDataAutoMaketitle that alter the default renderers would be quite high-level. Mixing low-level and high-level has the potential to produce weird results as you have shown.

[krono]

Ok, that makes sense…

[Witiko]

Oh cool :) concerning the hooks.

@TeXhackse Realistically, we may need to stay compatible with at least TeX Live 2020 to support Overleaf.
In 998b8a0, I added a fallback for TeX Live < 2021:

\@ifl@t@r
  \fmtversion
  { 2020-10-01 }
  { \AddToHook{begindocument/end}{\maketitle} }
  {
    \ifx\@onlypreamble\@notprerr
      % We are in the document
      \maketitle
    \else
      % We are in the preamble
      \RequirePackage{etoolbox}
      \AfterEndPreamble{\maketitle}
    \fi
  }

[Witiko]

We have fixed api7/lua-tinyyaml#10 in api7/lua-tinyyaml#11 and are waiting for the merge.

[Witiko]

@TeXhackse Congratulations, what a journey!

[TeXhackse]

@Witiko Thank you a lot. And I am sorry I could not do as much as I would have liked to.

[Witiko]

It's ok, you did lots: the package is 20k lines of not-exactly-contributor-friendly code. Thank you and @krono for kickstarting this feature. I learned a little LaTeX3 along the way, contributed to api7/lua-tinyyaml, and we have a first-ever YAML (subset) parser for TeX as a result. Not bad!