Using custom highlight style (.theme file) not working

[pck1980]

pandoc exits with error when using a custom theme file, even when it's the unmodified output of --print-highlight

PS C:\test> pandoc --print-highlight kate > k.theme
PS C:\test> pandoc --highlight-style k.theme
Could not read highlighting theme k.theme

There's a different error message if the file does not exist, so it seems pandoc is unable to correctly parse it.

PS C:\test> pandoc --version
pandoc.exe 2.10.1
Compiled with pandoc-types 1.21, texmath 0.12.0.2, skylighting 0.8.5
[...]

Perhaps similar to issue #4133?

[jgm]

Hm, I cannot reproduce this. Maybe it works on non-windows platforms but not on windows?

[jgm]

One thing to check would be whether k.theme has CRLF line endings.
If so, does it help to change to LF line endings?
If not, does it help to change to CRLF line endings?

[jgm]

We just use aeson ToJSOn/FromJSON instances to parse and render the styles as themes, so it's hard to see how line endings could be mishandled. But I'm at a loss as to what else could be going wrong, without being able to reproduce. Does k.theme look okay? Can you upload it?

[pck1980]

I just tried to use LF endings instead of CRLF, but with no success. Attached are both versions of the exported theme.
k.zip

[tarleb]

The files seem to use the UTF-16 encoding, which is the default on Windows, while pandoc expects all input to use UTF-8. Can you try to convert your files with the methods described in this answer?

[pck1980]

Converting the theme file to UTF-8 does indeed solve the problem. Thank you for your help.

Perhaps pandoc could print out a more helpful error message when it is provided with UTF-16 encoded input.

[tarleb]

Maybe we could also support the -o/--output option to redirect the output of --print-highlighting-style. That should prevent this kind of problem for Windows users.

[jgm]

The problem is that we are passing a bytestring to aeson's eitherDecode function.

It does return an error message in this case, but it's so useless that we don't print it: "Error in $: Failed reading: not a valid json value." It would be nice if aeson reported that the input was not properly encoded, but it doesn't work that way.

@tarleb it does already work with --output; however, note the following from the manual:
"This option may be used with -o/--output to redirect output to a file, but -o/--output must come before --print-highlight-style on the command line." This is because pandoc immediately exits (and does no further option processing) after printing the theme.

[caressofsteel]

I ran into this issue today. I converted to UTF-8 with Powershell as per the docs but Pandoc stil reports "Could not read the highlighing theme X". Working with Markdown to PDF with Pandoc has been like hiking through 5ft of snow. It's no supposed to be this hard or this bad.

pandoc.exe 2.11.4
Compiled with pandoc-types 1.22, texmath 0.12.1, skylighting 0.10.2,
citeproc 0.3.0.5, ipynb 0.1.0.1

[jgm]

@caressofsteel - can you upload your custom theme so we can test with it?
With the OP's issue, the problem was UTF-8 encoding, but your custom theme may have a different problem.

[GiuseppeDiPalma]

I have the same problem too.
Reading here I converted my file from utf-16 to utf-8, but the problem still persists:
Could not read highlighting theme testMyStyleUTF8.theme

I run this command:
pandoc file.md --highlight-style testMyStyleUTF8.theme -o file.html

pandoc --version:(windows10)

pandoc.exe 2.13
Compiled with pandoc-types 1.22, texmath 0.12.2, skylighting 0.10.5,
citeproc 0.3.0.9, ipynb 0.1.0.1

[jgm]

If you upload the file we can take a look.

[GiuseppeDiPalma]

These are the two files. The first one created is in utf16, the second is converted to utf8:

files.zip

[jgm]

The problem is that your file has a BOM (byte order mark) at the beginning:

testMyStyleUTF8.theme: UTF-8 Unicode (with BOM) text, with CRLF line terminators

The BOM is illegal in JSON (see e.g. https://stackoverflow.com/questions/4990095/json-specification-and-usage-of-bom-charset-encoding). If you remove it, this theme will work.
I was able to remove it by copying the text and saving as a separate file in my editor (which does not use a BOM). If your editor is inserting a BOM, you'll have to find an option to disable that.

[GiuseppeDiPalma]

Thank you very much, I copied and pasted in notepad++ and then copied again in VSCode, and now it works 💯