[deprecated] MathML Accessibility Annotations

[dginev]

Opening a very early pre-alpha pull request, tracking #1303 .

So far I added a minimally working implementation on one example, and will be adapting from there - also taking in feedback and brainstorming. Tests will fail because I haven't added a customization switch yet, so the annotations are always added, hence existing presentation MathML tests don't match up.

Things to note from this naive first stab:

• when we have an XMApp with an operator token with a known "meaning" attribute, the correspondence is pretty easy/direct
• when we have an mrow, it's basically rewriting the immediate XMApp tree as an attribute.
• when we have another wrapper pmml non-terminal (such as msqrt) for now I assumed the operator token will not be in the presentation, and the meaning should be added as a literal (in this case "square-root").
• when we have no meaning on the XMApp operator token, I just omit annotating at all. So far this is most (all?) msup/msub variants.
• We seem to have mrows around integration, which lets me pretend it's just an integral symbol applied on its integrand. Not ideal but easy enough.
• I have not looked at XMDuals explicitly yet, that's next
• side-remark: I am thinking of renaming the attributes to data-semantic and data-arg for a quick demo this week, so that I can use them with the existing browser stack without any turmoil, staying within the HTML spec.

The (somewhat arbitrary) example in question is:

latexmlc --whatsin=math --whatsout=math --format=html 'literal:\int \sqrt{x^2+c_i} dx'

producing the MathML (at time of opening the PR, definitely subject to change):

<math id="p1.m1" class="ltx_Math" alttext="\int\sqrt{x^{2}+c_{i}}dx" display="inline">
  <mrow semantic="@op(@1)">
    <mo arg="op" largeop="true" semantic="integral" symmetric="true">∫</mo>
    <mrow arg="1" semantic="@op(@1,@2)">
      <msqrt arg="1" semantic="square-root(@1)">
        <mrow arg="1" semantic="@op(@1,@2)">
          <msup arg="1">
            <mi>x</mi>
            <mn semantic="2">2</mn>
          </msup>
          <mo arg="op" semantic="plus">+</mo>
          <msub arg="2">
            <mi>c</mi>
            <mi>i</mi>
          </msub>
        </mrow>
      </msqrt>
      <mo arg="op" semantic="times">⁢</mo>
      <mrow arg="2" semantic="@op(@1)">
        <mo arg="op" rspace="0pt" semantic="differential-d">𝑑</mo>
        <mi arg="1">x</mi>
      </mrow>
    </mrow>
  </mrow>
</math>

[dginev]

Ok, added a basic dual example, namely x,y,z:

latexmlc --whatsin=math --whatsout=math --format=html 'literal:x,y,z'

<math id="p1.m1" class="ltx_Math" alttext="x,y,z" display="inline">
  <mrow semantic="list(@1,@2,@3)">
    <mi arg="1">x</mi>
    <mo>,</mo>
    <mi arg="2">y</mi>
    <mo>,</mo>
    <mi arg="3">z</mi>
  </mrow>
</math>

Feedback welcome.

[dginev]

Getting a bit better with the duals logic, here's a "stacked" dual that now works for x<0,y<1,z<2:

<math id="p1.m1" class="ltx_Math" alttext="x&lt;0,y&lt;1,z&lt;2" display="inline">
  <mrow semantic="formulae(@1,@2)">
    <mrow arg="1" semantic="@op(@1,@2)">
      <mi arg="1">x</mi>
      <mo arg="op" semantic="less-than">&lt;</mo>
      <mn arg="2">0</mn>
    </mrow>
    <mo>,</mo>
    <mrow arg="2" semantic="formulae(@1,@2)">
      <mrow arg="1" semantic="@op(@1,@2)">
        <mi arg="1">y</mi>
        <mo arg="op" semantic="less-than">&lt;</mo>
        <mn arg="2" semantic="1">1</mn>
      </mrow>
      <mo>,</mo>
      <mrow arg="2" semantic="@op(@1,@2)">
        <mi arg="1">z</mi>
        <mo arg="op" semantic="less-than">&lt;</mo>
        <mn arg="2" semantic="2">2</mn>
      </mrow>
    </mrow>
  </mrow>
</math>

[dginev]

PR is getting better, as I progressively understand the associateNode code better and get to ask for the right bits of data to answer the trickier annotation questions.

I can now do nested duals (in simple cases I've tested, hopefully more), e.g. \lceil{(1+x)}\rceil:

<math id="p1.m1" class="ltx_Math" alttext="\lceil{(1+x)}\rceil" display="inline">
  <mrow data-semantic="ceiling(@1)">
    <mo stretchy="false">⌈</mo>
    <mrow data-arg="1" data-semantic="@op(@1,@2)">
      <mo data-arg="1" stretchy="false">(</mo>
      <mrow data-arg="1">
        <mn data-arg="1" data-semantic="1">1</mn>
        <mo data-arg="op" data-semantic="plus">+</mo>
        <mi data-arg="2">x</mi>
      </mrow>
      <mo data-arg="1" stretchy="false">)</mo>
    </mrow>
    <mo stretchy="false">⌉</mo>
  </mrow>
</math>

Or duals in other expressions, e.g. 2(a+b)

<math id="p1.m1" class="ltx_Math" alttext="2(a+b)" display="inline">
  <mrow data-semantic="@op(@1,@2)">
    <mn data-arg="1" data-semantic="2">2</mn>
    <mo data-arg="op" data-semantic="times">⁢</mo>
    <mrow data-arg="2" data-semantic="@op(@1,@2)">
      <mo stretchy="false">(</mo>
      <mrow>
        <mi data-arg="1">a</mi>
        <mo data-arg="op" data-semantic="plus">+</mo>
        <mi data-arg="2">b</mi>
      </mrow>
      <mo stretchy="false">)</mo>
    </mrow>
  </mrow>
</math>

Note: I'm using the data-* attribute names at this point for the purpose of a web-compliant demo, but these remain subject to change by the W3C group.

[brucemiller]

Cool! But in your next to the last example, there are several stray data-arg="1" (on parens and mrow) that shouldn't be there. I haven't dug into why, though.

[dginev]

Alright, I think the code can be cleaned even further, but now the logic is starting to have healthy roots. Principles:

• fragid is a primary source of information for arg annotations, always obey it
• extra care to never double-annotate presentational bits of duals
• tokens and apps have obvious mappings in the standard cases, use them
• spot the cases where the operator XMTok "disappears" from presentation (such as multirelation, square root) and add literals to the semantic attributes. This is the piece of logic that may still need thinking.

Fixed all my dual examples, including binomial, now producing:

  <mrow data-semantic="binomial(@1,@2)">
    <mo>(</mo>
    <mfrac linethickness="0pt">
      <mi data-arg="1">n</mi>
      <mi data-arg="2">m</mi>
    </mfrac>
    <mo>)</mo>
  </mrow>

and the \lceil{(1+x)}\rceil now producing:

<mrow data-semantic="ceiling(@1)">
    <mo stretchy="false">⌈</mo>
    <mrow data-arg="1" data-semantic="@op(@1,@2)">
      <mo stretchy="false">(</mo>
      <mrow>
        <mn data-arg="1" data-semantic="1">1</mn>
        <mo data-arg="op" data-semantic="plus">+</mo>
        <mi data-arg="2">x</mi>
      </mrow>
      <mo stretchy="false">)</mo>
    </mrow>
    <mo stretchy="false">⌉</mo>
  </mrow>

and I can now also properly represent multirelation (though I had to add it as a special case, since I don't know the general principle), namely:

  <mrow data-semantic="multirelation(@1,@2,@3,@4,@5)">
    <mi data-arg="1">x</mi>
    <mo data-arg="2" data-semantic="less-than">&lt;</mo>
    <mi data-arg="3">y</mi>
    <mo data-arg="4" data-semantic="less-than">&lt;</mo>
    <mi data-arg="5">z</mi>
  </mrow>

[brucemiller]

A side comment (probably a separate issue, if it is): does latexmlc --whatsout=math intentionally not run the rewrites? Using \lxDeclare has no effect with your command above.

[dginev]

I've reoriented the foundation to closely monitor fragids, and duals first, handling the simpler cases only later (if none of the complex ones apply). It's still slightly bumpy (i.e. some duals trick my logic), but it's slimming down the code, and starting to feel targeted... Maybe another couple of polishes and it gets there.

[dginev]

Hi @brucemiller , I tried the --whatsout=math and it behaves as expected. Same for --whatsin=math. Here's an example:

latexmlc --preload=latexml.sty --whatsout=math --whatsin=math 'literal:\lxDeclare[role=FUNCTION]{$f$}f(x)'

produces:

<Math xmlns="http://dlmf.nist.gov/LaTeXML" mode="inline" xml:id="p1.m1" tex="f(x)" text="f@(x)" fragid="p1.m1">
  <XMath>
    <XMDual>
      <XMApp>
        <XMRef idref="p1.m1.1"/>
        <XMRef idref="p1.m1.2"/>
      </XMApp>
      <XMApp>
        <XMTok role="FUNCTION" xml:id="p1.m1.1" font="italic" fragid="p1.m1.1">f</XMTok>
        <XMWrap>
          <XMTok role="OPEN" stretchy="false">(</XMTok>
          <XMTok role="UNKNOWN" xml:id="p1.m1.2" font="italic" fragid="p1.m1.2">x</XMTok>
          <XMTok role="CLOSE" stretchy="false">)</XMTok>
        </XMWrap>
      </XMApp>
    </XMDual>
  </XMath>
</Math>

I tried this both in the master branch and in the temporary state of this PR, works normal with both. Could you give me a full invocation that failed for you?

[dginev]

Alright, I just pushed another iteration of improvement and elbow grease. The accessibility subroutine is now ~90 lines of code with lots of comments and whitespaces, which starts feeling manageable. I may be ready for a really deep stacking of multiple duals, as a stress test.

Also, and I'll follow up about this separately via email, the tiny semantic macro example for \integral is now operational:

latexmlc --preload=latexml.sty --preload=calculus.sty --whatsin=math --whatsout=math --format=html 'literal:\lxDeclare[role=FUNCTION]{$f$}\integral{f(x)}{x}'

created:

<math id="p1.m1" class="ltx_Math" alttext="\integral{f(x)}{x}" display="inline">
  <mrow data-semantic="integral(#1,#2)">
    <mo largeop="true" symmetric="true">∫</mo>
    <mrow>
      <mrow data-arg="1" data-semantic="#op(#1)">
        <mi data-arg="op">f</mi>
        <mo>⁡</mo>
        <mrow>
          <mo stretchy="false">(</mo>
          <mi data-arg="1">x</mi>
          <mo stretchy="false">)</mo>
        </mrow>
      </mrow>
      <mo>⁢</mo>
      <mrow>
        <mo rspace="0pt">𝑑</mo>
        <mi data-arg="2">x</mi>
      </mrow>
    </mrow>
  </mrow>
</math>

[dginev]

One thing I know still needs cleaning up is the cases where I add annotations on unsemantic pMML, such as:

  <msup data-semantic="m:msup(#1,#2)">
    <mi data-arg="1">x</mi>
    <mn data-arg="2" data-semantic="2">2</mn>
  </msup>

I thought this was a nice crutch to make explicit what latexml considered the internal applicative structure, but it had a sad side-effect on binomials, which are now polluted with it:

 <mrow data-semantic="binomial(#1,#2)">
    <mo>(</mo>
    <mfrac data-semantic="m:mfrac(#1,#2)" linethickness="0pt">
      <mi data-arg="1">n</mi>
      <mi data-arg="2">m</mi>
    </mfrac>
    <mo>)</mo>
  </mrow>

It's almost "not even wrong", but it's just noisy, and I should silence it.

Edit: now fixed, alsod disabled annotating number tokens. So for x^2 and \binom{n}{m} we get the plainer:

  <msup>
    <mi>x</mi>
    <mn>2</mn>
  </msup>

and

  <mrow data-semantic="binomial(#1,#2)">
    <mo>(</mo>
    <mfrac linethickness="0pt">
      <mi data-arg="1">n</mi>
      <mi data-arg="2">m</mi>
    </mfrac>
    <mo>)</mo>
  </mrow>

[dginev]

I have started to break ground towards minimal semantic markup today, though very partially so -- mostly preparing the showcase page for later glory. I took one approach to doing \power.

However, I could only figure out how to do it by modding MathParser ,since both afterConstruct=> and DefMathRewrite happen too early, before the script token is brought into the world.

You can check the current state at the showcase page, look for the single-word "power" example in the selection dropdown:
https://dginev.github.io/tiny-mathml-a11y-demo/

[dginev]

I have the adjoint, transpose and 2nd derivative examples now working with (hopefully) minimal semantic annotations. Calling it a night.

[brucemiller]

Try

DefMath('\power{}{}', "{#1^{#2}}", meaning => 'power',
  reversion              => '#1^{#2}',
  hide_content_reversion => 1);

[dginev]

It works too it seems! Yours creates the markup:

  <XMath>
    <XMDual>
      <XMApp>
        <XMTok meaning="power" role="UNKNOWN"/>
        <XMRef idref="p1.m1.1"/>
        <XMRef idref="p1.m1.2"/>
      </XMApp>
      <XMApp>
        <XMTok role="SUPERSCRIPTOP" scriptpos="post2"/>
        <XMTok role="UNKNOWN" xml:id="p1.m1.1" font="italic" fragid="p1.m1.1">x</XMTok>
        <XMTok meaning="2" role="NUMBER" xml:id="p1.m1.2" fontsize="70%" fragid="p1.m1.2">2</XMTok>
      </XMApp>
    </XMDual>
  </XMath>

my \meaning take does:

  <XMath>
    <XMApp>
      <XMTok meaning="power" role="SUPERSCRIPTOP" scriptpos="post2"/>
      <XMTok role="UNKNOWN" font="italic">x</XMTok>
      <XMTok meaning="2" role="NUMBER" fontsize="70%">2</XMTok>
    </XMApp>
  </XMath>

which improves the tree but loses some attributes. I wonder if I should be modding DefMath instead of creating more magical DefConstructors...

[brucemiller]

Don't know about "improves", depends on what you like :> Your \meaning probably works in most simple cases, but (you imply) needs extra mods. DefMath is exactly that magical DefConstructor that already knows about the arguments, so it tends to put the attributes (meaning) in the right place. With the side benefit that it creates an XMDual! :>

[dginev]

Is this really down to personal preference in these two examples? I thought it was an XMath design goal to capture the meaning attribute of terminal tokens where they are clear. Would you say it's personal preference if the meaning="2" on the number token sits in a dual, as in:

<XMDual>
  <XMTok meaning="2" role="UNKNOWN"/>
  <XMTok role="NUMBER" xml:id="p1.m1.2" fontsize="70%" fragid="p1.m1.2">2</XMTok>
</XMDual>

Such tokens are also not cross-referenced, so the connection is less direct than it would be with an XMRef. I was approaching things with the mindset that the duals ought to only come into play where the tree correspondence between presentation and content breaks down, if we're to think of some notion of "optimal" XMath.

[dginev]

To put it simply, I'm fishing for a DefMath form which first instantiates the XMath tree as usual, then picks the operating token and assigns it a meaning. Maybe a new option for DefMath that insists on marking up in-place... (or an existing one, I'm still getting the hang of the advanced DefMath uses)

[brucemiller]

I think I've had this debate with myself at least as many times as I've had it with you :>

There's Taste, and there's the Possible, and then there are the Traps :> DefMath tries hard to do the right thing, and has been through many iterations of traps, so you want to think twice before reinventing it. OTOH, it's a bit conservative. So, for example, the \diffd case decided on a dual because it saw \mathrm and got suspicious. A better definition would be

DefMath('\diffd', 'd',
  meaning => 'differential', role => 'DIFFOP',
  font    => { shape => 'upright' });

which avoids the dual.

Next up is the "clever" way that role is used that allows a simple triple to represent both the presentation and content trees in many, but not all, cases. Alas, again DefMath isn't quite smart enough to construct a sample and figure out that the presentation & content trees are isomorphic, and thus avoid a dual. Basically, it sees that there are arguments and punts to a dual to be sure. Perhaps that's a further enhancement? Or lacking that, an extra option that asserts what kind of construction should be used?

The trap with a simple version, like \meaning, is that you grow to like it and start using it all over and then realize it's doing weird things, so it grows and handles various special cases until it ends up being... DefMath!

Another trap, and something I struggled with a few commits back, was wanting to immediately optimize some special cases that should produce something more canonical, but realized that the remaining machinery needed to handle the awkward markup as well as the simpler. So I put off that optimization for a bit.

So, maybe we could make \power cleaner, but not till after you make the annotation work with the bad markup, first :>

[dginev]

Thanks, I quite like this poetic overview of Taste, The Possible and Traps :> It even feels close to a documentation/writeup heading, just needs a third t-word. "Tools, Taste and Traps"?

You've successfully convinced me I should be using DefMath! But your comments in DefMath had convinced me that you've had my problems before and have been hitting similar walls namely:

# OK, this is getting a bit out-of-hand; I can't, myself, predict whether XMDual gets involved!

So I'll try to abolish my DefConstructor games with \meaning, and instead try to use DefMath exclusively, making it "less conservative" where needed - as you suggest. I think I only have a couple of new cases that I'm interested in (not making a dual when a script has a clear meaning but using a compact XMApp, maybe a general rewrite rule that compacts redundant duals into apps...).

Since the DefMath interface allows for open-ended key=>val options, I can just hide whatever I need behind a new option name, keep the defaults nice and conservative :> Thanks for the great feedback!

[dginev]

Today I discovered LaTeXML::Core::Document::pruneXMDuals, and I think this is exactly what I was looking to extend. Expect updates...

[dginev]

Small updates:

• \fnderive{f}{2} , \fnderive{f}{n+y} now generates literal ('') and symbolic ((n+y)) presentational notations for derivative scripts, while achieving the a11y annotations per the semantics-mini spec.
• I succeeded in hiding the a11y upgrade behind a processing instruction in the XML, now only enabled when a11ymark.sty is loaded. Which meant the default behaviour remains as in master, and the experiment could be merged before MathML 4 becomes official. And when a11y becomes obviously the way to go, the guard can be dropped entirely.
• the showcase has the updated branch, and I added some minimal UI for removing unneeded examples (individually or in bulk)

Next up: covering more examples with simple dual-based macros.

[dginev]

I found a nasty limitation in my a11y extraction when an XMRef of a Dual points to a content branch of a sub-XMDual, left a comment here. Unsure if I should tackle this head on, or try to avoid it in the macros... since it is always avoidable, by just adding another layer of indirection in the TeX macros.

It's kind of a nasty precedent to be able to point in the content branch of a child dual, it almost feels like a bug, as it breaks the conceptual encapsulation (in my head) of XMDuals as black boxes, without cross-talk. Need to think about it some more, maybe more examples...

[dginev]

Rebased to latest master, though I haven't yet re-tested on the a11y demo, might be broken. So also not deployed anywhere post-rebase. Will comment with an update when I recheck.

[brucemiller]

[virtual jetlag.... lovely]

I think I may have caused you to break it; I'm not seeing any annotations

[dginev]

Sorry, for the jetlag! Could you elaborate which command you tried that had annotations disappear?

I am still seeing reasonable results with say:

latexmlc --preload=latexml.sty --preload=a11ymark.sty --whatsin=math --whatsout=math --format=html 'literal:\lxDeclare[role=FUNCTION]{$f$}\integral{f(x)}{x}'

producing with the latest state:

  <mrow data-semantic="integral(#1,#2)">
    <mo largeop="true" symmetric="true">∫</mo>
    <mrow>
      <mrow data-arg="1" data-semantic="#op(#1)">
        <mi data-arg="op">f</mi>
        <mo>⁡</mo>
        <mrow>
          <mo stretchy="false">(</mo>
          <mi data-arg="1">x</mi>
          <mo stretchy="false">)</mo>
        </mrow>
      </mrow>
      <mo>⁢</mo>
      <mrow>
        <mo rspace="0pt">𝑑</mo>
        <mi data-arg="2">x</mi>
      </mrow>
    </mrow>
  </mrow>

[dginev]

As I mentioned before, you need to do preload or usepackage a11ymark.sty to enable the annotations. I wonder if this was the source of confusion ? I made it enabled only when requested, so that tests pass and the PR is mergable without any changes from mainline latexml. I think I also mentioned that in one of our calls the week prior.

[brucemiller]

On 7/31/20 8:05 PM, Deyan Ginev wrote: you need to do preload or usepackage |a11ymark.sty| to enable the annotations.

Yup, I missed that.

[dginev]

Rebased to latest master, PR is starting to get smaller/cleaner. I think I should get my action enhancement to lxDeclare out next, as it is not essential to the annotation details. Maybe even I should do the inverted - make a fresh branch with only the annotations-adding code, and branch off of that with the "macro experimentation" work. Yeah, I'll do that.

[dginev]

I moved the core enhancement to #1329, and after I make a separate branch with the "macro experiments", I will close this PR entirely.

[dginev]

Since I wanted to keep the diff small, I pointed the macro experiment at the "a11y core" branch, and it remained entirely in my LaTeXML fork here.

With both of these pieces moved out, this branch is now completely deprecated, and I will close the PR. I have added backreferences, so that the discussion upto now can be tracked when needed.