More flexible citation key syntax #6026

[Aver1y]

This implements the {citekey} syntax and also allows escaping using \ both } and \ inside {} and any illegal character in the existing syntax.

This does not address all of #6026. I still want to implement the [@citekey]: citekey' syntax.

TODO:

• Update documentation to reflect new behaviour of citekey parser regarding newlines.
• Add tests?

[jgm]

I'm not sure about the idea of allowing escapes in regular citations (without the {...}).
This changes current behavior in a way that might have some bad consequences.
For example, currently @doe\-1999 will create a citation to doe followed by the string -1999, whereas @doe-1999 will create a citation to doe-1999. I think that if we have the braces syntax, we don't need the escapes. And in braces, only braces and \\ need to be escapable. That keeps things simple.

[Aver1y]

Changed.

[Aver1y]

As for newlines, I think it would make sense to allow newlines only if not followed by a blankline (there's a parser for this

I think you mean spnl? That one is in Readers/Markdown.hs, so not available here unless we move it.

[Aver1y]

I wondered if it would be more consistent to parse any occurrence of multiple spaces as a single space. The way it is now I would think it may be especially confusing with trailing spaces? If one wants multiple spaces they can still escape them.

[jgm]

Yes, we could collapse consecutive spaces, that might make sense.

[brainchild0]

I suggest an alternative approach for handling squiggly braces within a key, which is to parse nested brace or parenthesis pairs. Thus @{a{b}c(d)e[f]g<h>i} would correspond to key a{b}c(d)e[f]g<h>i, because only the final brace matches the first. Such rules would handle all cases but a few pathological ones, which I believe are currently only hypothetical. It also would avoid user difficulty related to escaping sequences, which may be long and often would be inserted by cut-and-paste.

If a later need arises to support the remaining cases, then I would suggest doing so through a slightly more explicit form, such as @{"pathological\}case"}. Perhaps this support might be postponed.

[Aver1y]

@brainchild0 I see the point for matching braces. It is probably more common for braces to appear paired than on their own. For other parenthesis I don't see the benefit.

[brainchild0]

Yes, you may be right that in this case matching other braces is unnecessary, as long as they are protected from the processing that would occur in a regular context. Thus the first ] in [@{protect]me}] should probably not close the [. However, fully applying the same rules for all brace types creates a convention that robustly generalizes to other scenarios, which later may be useful considering the many separate meanings given to only a few brace types during processing depending on context. It also reduces ambiguity because the above form may then be considered illegal and caused to generate a warning.

[dhimmel]

An additional benefit of this pull request might be that it allows a way to terminate a citekey without space or punctuation. Specifically, what will @{happen}here? Am I correct that there prior to this PR there is no way to have a citekey happen immediately followed by here? And that this PR enables this?

Where this would come in handy is for filter like pandoc-fignos by @tomduck. Currently, to have a reference like "Figure 1A" with pandoc-fignos, the user has to write Figure {@fig:id}A. I think this PR would enable Figure @{fig:id}A without any special support by the filter.

[dhimmel]

Looks like 3f09f53 added this feature.

[jgm]

Oh, I'm sorry -- I'd completely forgotten about this PR!
Note that in my implementation, I don't allow backslash escapes at all. Braces are allowed, but only if matched.