Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable syntax highlighting for inline code #17585

Merged
merged 9 commits into from
Apr 2, 2021
Merged
3 changes: 2 additions & 1 deletion compiler/docgen.nim
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,8 @@ proc newDocumentor*(filename: AbsoluteFile; cache: IdentCache; conf: ConfigRef,
result.cache = cache
result.outDir = conf.outDir.string
initRstGenerator(result[], (if conf.cmd != cmdRst2tex: outHtml else: outLatex),
conf.configVars, filename.string, {roSupportRawDirective, roSupportMarkdown},
conf.configVars, filename.string,
{roSupportRawDirective, roSupportMarkdown, roNimFile},
docgenFindFile, compilerMsgHandler)

if conf.configVars.hasKey("doc.googleAnalytics"):
Expand Down
38 changes: 38 additions & 0 deletions doc/contributing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,44 @@ example below) from `Nim Index`_ can be used in doc comment this way:

.. _`Nim Index`: https://nim-lang.org/docs/theindex.html

Inline monospaced text can be input using \`single backticks\` or
\`\`double backticks\`\`. The former are syntactically highlighted,
the latter are not.
To avoid accidental highlighting follow this rule in `*.nim` files:

* use single backticks for fragments of code in Nim and other
programming languages, including identifiers, in `*.nim` files.

For languages other than Nim add a role after final backtick,
e.g. for C++ inline highlighting::

`#include <stdio.h>`:cpp:

For a currently unsupported language add the `:code:` role,
like for SQL in this example::

`SELECT * FROM <table_name>;`:code:

* prefer double backticks otherwise:

* for file names: \`\`os.nim\`\`
* for fragments of strings **not** enclosed by `"` and `"` and not
related to code, e.g. text of compiler messages
* for command line options: \`\`--docInternal\`\`
* also when code ends with a standalone ``\`` (otherwise a combination of
``\`` and a final \` would get escaped)

.. Note:: `*.rst` files have `:literal:` as their default role.
So for them the rule above is only applicable if the `:nim:` role
is set up manually as the default::

.. role:: nim(code)
:language: nim
.. default-role:: nim

The first 2 lines are for other RST implementations,
including Github one.

Best practices
==============

Expand Down
116 changes: 96 additions & 20 deletions lib/packages/docutils/rst.nim
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,13 @@
##
## * directives: ``code-block`` [cmp:Sphinx]_, ``title``,
## ``index`` [cmp:Sphinx]_
## * predefined roles ``:nim:`` (default), ``:c:`` (C programming language),
## ``:python:``, ``:yaml:``, ``:java:``, ``:cpp:`` (C++), ``:csharp`` (C#).
## That is every language that `highlite <highlite.html>`_ supports.
## They turn on appropriate syntax highlighting in inline code.
##
## .. Note:: default role for Nim files is ``:nim:``,
## for ``*.rst`` it's currently ``:literal:``.
##
## * ***triple emphasis*** (bold and italic) using \*\*\*
## * ``:idx:`` role for \`interpreted text\` to include the link to this
Expand Down Expand Up @@ -161,7 +168,9 @@ type
roSupportSmilies, ## make the RST parser support smilies like ``:)``
roSupportRawDirective, ## support the ``raw`` directive (don't support
## it for sandboxing)
roSupportMarkdown ## support additional features of Markdown
roSupportMarkdown, ## support additional features of Markdown
roNimFile ## set for Nim files where default interpreted
## text role should be :nim:

RstParseOptions* = set[RstParseOption]

Expand Down Expand Up @@ -454,6 +463,8 @@ type
hTitleCnt: int # =0 if no title, =1 if only main title,
# =2 if both title and subtitle are present
hCurLevel: int # current section level
currRole: string # current interpreted text role
currRoleKind: RstNodeKind # ... and its node kind
subs: seq[Substitution] # substitutions
refs: seq[Substitution] # references
anchors: seq[AnchorSubst] # internal target substitutions
Expand Down Expand Up @@ -514,10 +525,36 @@ proc defaultFindFile*(filename: string): string =
if fileExists(filename): result = filename
else: result = ""

proc defaultRole(options: RstParseOptions): string =
if roNimFile in options: "nim" else: "literal"
Copy link
Member

@timotheecour timotheecour Apr 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add

type BuiltinRoles = enum
  brUnknown = "unknown"
  brNim = "nim"
  brLiteral = "literal"
  brCode = "code"
  ... # doesn't need to be a complete list

and then use it everytime you're using one of those builtin roles, it's more typesafe eg:

if roNimFile in options: $brNim else: $brLiteral

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

too little profit for a new type. Most roles are met only once and immediately converted to RstNodeKind. The only exception is "nim" and "literal", which occur twice, but both occurrences are within a few lines.


# mirror highlite.nim sourceLanguageToStr with substitutions c++ cpp, c# csharp
const supportedLanguages = ["nim", "yaml", "python", "java", "c",
"cpp", "csharp"]

proc whichRoleAux(sym: string): RstNodeKind =
let r = sym.toLowerAscii
case r
of "idx": result = rnIdx
of "literal": result = rnInlineLiteral
of "strong": result = rnStrongEmphasis
of "emphasis": result = rnEmphasis
of "sub", "subscript": result = rnSub
of "sup", "superscript": result = rnSup
# literal and code are the same in our implementation
of "code": result = rnInlineLiteral
# c++ currently can be spelled only as cpp, c# only as csharp
elif r in supportedLanguages:
result = rnInlineCode
else: # unknown role
result = rnUnknownRole

proc newSharedState(options: RstParseOptions,
findFile: FindFileHandler,
msgHandler: MsgHandler): PSharedState =
new(result)
result.currRole = defaultRole(options)
result.currRoleKind = whichRoleAux(result.currRole)
result.subs = @[]
result.refs = @[]
result.options = options
Expand Down Expand Up @@ -1018,15 +1055,28 @@ proc fixupEmbeddedRef(n, a, b: PRstNode) =
for i in countup(0, sep - incr): a.add(n.sons[i])
for i in countup(sep + 1, n.len - 2): b.add(n.sons[i])

proc whichRole(sym: string): RstNodeKind =
case sym
of "idx": result = rnIdx
of "literal": result = rnInlineLiteral
of "strong": result = rnStrongEmphasis
of "emphasis": result = rnEmphasis
of "sub", "subscript": result = rnSub
of "sup", "superscript": result = rnSup
else: result = rnGeneralRole
proc whichRole(p: RstParser, sym: string): RstNodeKind =
result = whichRoleAux(sym)
if result == rnUnknownRole:
rstMessage(p, mwUnsupportedLanguage, p.s.currRole)

proc toInlineCode(n: PRstNode, language: string): PRstNode =
## Creates rnInlineCode and attaches `n` contents as code (in 3rd son).
result = newRstNode(rnInlineCode)
let args = newRstNode(rnDirArg)
var lang = language
if language == "cpp": lang = "c++"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

something is inconsistent:

.. code-block:: cpp

  #include <stdio.h>

.. code-block:: c++

  #include <stdio.h>

`#include <stdio.h>`:cpp:

`#include <stdio.h>`:c++:

gives
image

=> depending on block vs inline, the syntax that works is c++ vs cpp

we should have one that works in all contexts, and recommend it (including in example from eariler).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that :c++: is not parsed is another bug in our parser. Added to the list of minor bugs.

The proper fix (with code reusing) is not trivial, it's better to handle in another PR.

elif language == "csharp": lang = "c#"
args.add newLeaf(lang)
result.add args
result.add PRstNode(nil)
var lb = newRstNode(rnLiteralBlock)
var s: string
for i in n.sons:
assert i.kind == rnLeaf
s.add i.text
lb.add newLeaf(s)
result.add lb

proc parsePostfix(p: var RstParser, n: PRstNode): PRstNode =
var newKind = n.kind
Expand All @@ -1052,14 +1102,23 @@ proc parsePostfix(p: var RstParser, n: PRstNode): PRstNode =
result = newRstNode(newKind, newSons)
elif match(p, p.idx, ":w:"):
# a role:
newKind = whichRole(nextTok(p).symbol)
if newKind == rnGeneralRole:
let roleName = nextTok(p).symbol
newKind = whichRole(p, roleName)
if newKind == rnUnknownRole:
let newN = newRstNode(rnInner, n.sons)
newSons = @[newN, newLeaf(nextTok(p).symbol)]
newSons = @[newN, newLeaf(roleName)]
result = newRstNode(newKind, newSons)
elif newKind == rnInlineCode:
result = n.toInlineCode(language=roleName)
else:
result = newRstNode(newKind, newSons)
inc p.idx, 3
result = newRstNode(newKind, newSons)
else: # no change
result = n
else:
if p.s.currRoleKind == rnInlineCode:
result = n.toInlineCode(language=p.s.currRole)
else:
newKind = p.s.currRoleKind
result = newRstNode(newKind, newSons)

proc matchVerbatim(p: RstParser, start: int, expr: string): int =
result = start
Expand Down Expand Up @@ -1315,9 +1374,12 @@ proc parseInline(p: var RstParser, father: PRstNode) =
parseUntil(p, n, "``", false)
father.add(n)
elif match(p, p.idx, ":w:") and p.tok[p.idx+3].symbol == "`":
let k = whichRole(nextTok(p).symbol)
let n = newRstNode(k)
let roleName = nextTok(p).symbol
let k = whichRole(p, roleName)
var n = newRstNode(k)
inc p.idx, 3
if k == rnInlineCode:
n = n.toInlineCode(language=roleName)
parseUntil(p, n, "`", false) # bug #17260
father.add(n)
elif isInlineMarkupStart(p, "`"):
Expand Down Expand Up @@ -2421,6 +2483,18 @@ proc dirAdmonition(p: var RstParser, d: string): PRstNode =

proc dirDefaultRole(p: var RstParser): PRstNode =
result = parseDirective(p, rnDefaultRole, {hasArg}, nil)
if result.sons[0].len == 0: p.s.currRole = defaultRole(p.s.options)
else:
assert result.sons[0].sons[0].kind == rnLeaf
p.s.currRole = result.sons[0].sons[0].text
p.s.currRoleKind = whichRole(p, p.s.currRole)

proc dirRole(p: var RstParser): PRstNode =
result = parseDirective(p, rnDirective, {hasArg, hasOptions}, nil)
# just check that language is supported, TODO: real role association
let lang = getFieldValue(result, "language").strip
if lang != "" and lang notin supportedLanguages:
rstMessage(p, mwUnsupportedLanguage, lang)

proc dirRawAux(p: var RstParser, result: var PRstNode, kind: RstNodeKind,
contentParser: SectionParser) =
Expand Down Expand Up @@ -2465,7 +2539,9 @@ proc selectDir(p: var RstParser, d: string): PRstNode =
of "code-block": result = dirCodeBlock(p, nimExtension = true)
of "container": result = dirContainer(p)
of "contents": result = dirContents(p)
of "danger", "error": result = dirAdmonition(p, d)
of "danger": result = dirAdmonition(p, d)
of "default-role": result = dirDefaultRole(p)
of "error": result = dirAdmonition(p, d)
of "figure": result = dirFigure(p)
of "hint": result = dirAdmonition(p, d)
of "image": result = dirImage(p)
Expand All @@ -2478,10 +2554,10 @@ proc selectDir(p: var RstParser, d: string): PRstNode =
result = dirRaw(p)
else:
rstMessage(p, meInvalidDirective, d)
of "role": result = dirRole(p)
of "tip": result = dirAdmonition(p, d)
of "title": result = dirTitle(p)
of "warning": result = dirAdmonition(p, d)
of "default-role": result = dirDefaultRole(p)
else:
let tok = p.tok[p.idx-2] # report on directive in ".. directive::"
rstMessage(p, meInvalidDirective, d, tok.line, tok.col)
Expand Down
9 changes: 6 additions & 3 deletions lib/packages/docutils/rstast.nim
Original file line number Diff line number Diff line change
Expand Up @@ -55,12 +55,15 @@ type
# * `file#id <file#id>`_
# * `file#id <file#id>'_
rnSubstitutionDef, # a definition of a substitution
rnGeneralRole, # Inline markup:
# Inline markup:
rnInlineCode,
rnUnknownRole, # interpreted text with an unknown role
rnSub, rnSup, rnIdx,
rnEmphasis, # "*"
rnStrongEmphasis, # "**"
rnTripleEmphasis, # "***"
rnInterpretedText, # "`"
rnInterpretedText, # "`" an auxiliary role for parsing that will
# be converted into other kinds like rnInlineCode
rnInlineLiteral, # "``"
rnInlineTarget, # "_`target`"
rnSubstitutionReferences, # "|"
Expand Down Expand Up @@ -252,7 +255,7 @@ proc renderRstToRst(d: var RenderContext, n: PRstNode, result: var string) =
result.add(" <")
renderRstToRst(d, n.sons[1], result)
result.add(">`_")
of rnGeneralRole:
of rnUnknownRole:
result.add('`')
renderRstToRst(d, n.sons[0],result)
result.add("`:")
Expand Down
33 changes: 23 additions & 10 deletions lib/packages/docutils/rstgen.nim
Original file line number Diff line number Diff line change
Expand Up @@ -942,7 +942,7 @@ proc parseCodeBlockParams(d: PDoc, n: PRstNode): CodeBlockParams =
result.init
if n.isNil:
return
assert n.kind == rnCodeBlock
assert n.kind in {rnCodeBlock, rnInlineCode}
assert(not n.sons[2].isNil)

# Parse the field list for rendering parameters if there are any.
Expand Down Expand Up @@ -987,8 +987,8 @@ proc buildLinesHtmlTable(d: PDoc; params: CodeBlockParams, code: string,
"</td></tr></tbody></table>" & (
d.config.getOrDefault"doc.listing_button" % id)

proc renderCodeBlock(d: PDoc, n: PRstNode, result: var string) =
## Renders a code block, appending it to `result`.
proc renderCode(d: PDoc, n: PRstNode, result: var string) =
## Renders a code (code block or inline code), appending it to `result`.
##
## If the code block uses the ``number-lines`` option, a table will be
## generated with two columns, the first being a list of numbers and the
Expand All @@ -997,7 +997,7 @@ proc renderCodeBlock(d: PDoc, n: PRstNode, result: var string) =
## may also come from the parser through the internal ``default-language``
## option to differentiate between a plain code block and Nim's code block
## extension.
assert n.kind == rnCodeBlock
assert n.kind in {rnCodeBlock, rnInlineCode}
if n.sons[2] == nil: return
var params = d.parseCodeBlockParams(n)
var m = n.sons[2].sons[0]
Expand All @@ -1006,10 +1006,23 @@ proc renderCodeBlock(d: PDoc, n: PRstNode, result: var string) =
if params.testCmd.len > 0 and d.onTestSnippet != nil:
d.onTestSnippet(d, params.filename, params.testCmd, params.status, m.text)

let (blockStart, blockEnd) = buildLinesHtmlTable(d, params, m.text,
var blockStart, blockEnd: string
case d.target
of outHtml:
if n.kind == rnCodeBlock:
(blockStart, blockEnd) = buildLinesHtmlTable(d, params, m.text,
n.anchor.idS)
dispA(d.target, result, blockStart,
"\\begin{rstpre}\n" & n.anchor.idS & "\n", [])
else: # rnInlineCode
blockStart = "<tt class=\"docutils literal\"><span class=\"pre\">"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prefer """ ... """ to make it easier to copy paste

blockEnd = "</span></tt>"
of outLatex:
if n.kind == rnCodeBlock:
blockStart = "\n\n\\begin{rstpre}" & n.anchor.idS & "\n"
blockEnd = "\n\\end{rstpre}\n"
else: # rnInlineCode
blockStart = "\\texttt{"
blockEnd = "}"
dispA(d.target, result, blockStart, blockStart, [])
if params.lang == langNone:
if len(params.langStr) > 0:
d.msgHandler(d.filename, 1, 0, mwUnsupportedLanguage, params.langStr)
Expand All @@ -1028,7 +1041,7 @@ proc renderCodeBlock(d: PDoc, n: PRstNode, result: var string) =
esc(d.target, substr(m.text, g.start, g.length+g.start-1)),
tokenClassToStr[g.kind]])
deinitGeneralTokenizer(g)
dispA(d.target, result, blockEnd, "\n\\end{rstpre}\n")
dispA(d.target, result, blockEnd, blockEnd)

proc renderContainer(d: PDoc, n: PRstNode, result: var string) =
var tmp = ""
Expand Down Expand Up @@ -1294,13 +1307,13 @@ proc renderRstToOut(d: PDoc, n: PRstNode, result: var string) =
result.add addNodes(lastSon(n))

of rnImage, rnFigure: renderImage(d, n, result)
of rnCodeBlock: renderCodeBlock(d, n, result)
of rnCodeBlock, rnInlineCode: renderCode(d, n, result)
of rnContainer: renderContainer(d, n, result)
of rnSubstitutionReferences, rnSubstitutionDef:
renderAux(d, n, "|$1|", "|$1|", result)
of rnDirective:
renderAux(d, n, "", "", result)
of rnGeneralRole:
of rnUnknownRole:
var tmp0 = ""
var tmp1 = ""
renderRstToOut(d, n.sons[0], tmp0)
Expand Down
6 changes: 3 additions & 3 deletions nimdoc/rst2html/expected/rst_examples.html
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ <h1><a class="toc-backref" id="constants-and-constant-expressions" href="#consta
</dl>
<p>Let <tt class="docutils literal"><span class="pre">T</span></tt>'s be <tt class="docutils literal"><span class="pre">p</span></tt>'s return type. NRVO applies for <tt class="docutils literal"><span class="pre">T</span></tt> if <tt class="docutils literal"><span class="pre">sizeof(T) &gt;= N</span></tt> (where <tt class="docutils literal"><span class="pre">N</span></tt> is implementation dependent), in other words, it applies for &quot;big&quot; structures.</p>
<p>Apart from built-in operations like array indexing, memory allocation, etc. the <tt class="docutils literal"><span class="pre">raise</span></tt> statement is the only way to raise an exception.</p>
<p><tt class="docutils literal"><span class="pre">typedesc</span></tt> used as a parameter type also introduces an implicit generic. <tt class="docutils literal"><span class="pre">typedesc</span></tt> has its own set of rules:</p>
<p><tt class="docutils literal"><span class="pre"><span class="Identifier">typedesc</span></span></tt> used as a parameter type also introduces an implicit generic. <tt class="docutils literal"><span class="pre"><span class="Identifier">typedesc</span></span></tt> has its own set of rules:</p>
<p>The <tt class="docutils literal"><span class="pre">!=</span></tt>, <tt class="docutils literal"><span class="pre">&gt;</span></tt>, <tt class="docutils literal"><span class="pre">&gt;=</span></tt>, <tt class="docutils literal"><span class="pre">in</span></tt>, <tt class="docutils literal"><span class="pre">notin</span></tt>, <tt class="docutils literal"><span class="pre">isnot</span></tt> operators are in fact templates:</p>
<p><tt class="docutils literal"><span class="pre">a &gt; b</span></tt> is transformed into <tt class="docutils literal"><span class="pre">b &lt; a</span></tt>.<br/><tt class="docutils literal"><span class="pre">a in b</span></tt> is transformed into <tt class="docutils literal"><span class="pre">contains(b, a)</span></tt>.<br/><tt class="docutils literal"><span class="pre">notin</span></tt> and <tt class="docutils literal"><span class="pre">isnot</span></tt> have the obvious meanings.<br/></p><p>A template where every parameter is <tt class="docutils literal"><span class="pre">untyped</span></tt> is called an <span id="immediate_1">immediate</span> template. For historical reasons templates can be explicitly annotated with an <tt class="docutils literal"><span class="pre">immediate</span></tt> pragma and then these templates do not take part in overloading resolution and the parameters' types are <em>ignored</em> by the compiler. Explicit immediate templates are now deprecated.</p>

Expand Down Expand Up @@ -297,10 +297,10 @@ <h2><a class="toc-backref" id="code-reordering-time-measurement-with-garbage-col
<h1><a class="toc-backref" id="introduction" href="#introduction">Introduction</a></h1><blockquote><p>
"Der Mensch ist doch ein Augentier -- sch&ouml;ne Dinge w&uuml;nsch ich mir."
</p></blockquote><p>This document is a tutorial for the programming language <em>Nim</em>. This tutorial assumes that you are familiar with basic programming concepts like variables, types, or statements but is kept very basic. The <a class="reference external" href="manual.html">manual</a> contains many more examples of the advanced language features. All code examples in this tutorial, as well as the ones found in the rest of Nim's documentation, follow the <a class="reference external" href="nep1.html">Nim style guide</a>.</p>
<p>However, this does not work. The problem is that the procedure should not only <tt class="docutils literal"><span class="pre">return</span></tt>, but return and <strong>continue</strong> after an iteration has finished. This <em>return and continue</em> is called a <tt class="docutils literal"><span class="pre">yield</span></tt> statement. Now the only thing left to do is to replace the <tt class="docutils literal"><span class="pre">proc</span></tt> keyword by <tt class="docutils literal"><span class="pre">iterator</span></tt> and here it is - our first iterator:</p>
<p>However, this does not work. The problem is that the procedure should not only <tt class="docutils literal"><span class="pre">return</span></tt>, but return and <strong>continue</strong> after an iteration has finished. This <em>return and continue</em> is called a <tt class="docutils literal"><span class="pre"><span class="Keyword">yield</span></span></tt> statement. Now the only thing left to do is to replace the <tt class="docutils literal"><span class="pre">proc</span></tt> keyword by <tt class="docutils literal"><span class="pre">iterator</span></tt> and here it is - our first iterator:</p>
<table border="1" class="docutils"><tr><th>A1 header</th><th>A2 | not fooled</th></tr>
<tr><td>C1</td><td>C2 <strong>bold</strong></td></tr>
<tr><td>D1 <tt class="docutils literal"><span class="pre">code \|</span></tt></td><td>D2</td></tr>
<tr><td>D1 <tt class="docutils literal"><span class="pre"><span class="Identifier">code</span> <span class="Operator">\|</span></span></tt></td><td>D2</td></tr>
<tr><td>E1 | text</td><td></td></tr>
<tr><td></td><td>F2 without pipe</td></tr>
</table><p>not in table </p>
Expand Down
4 changes: 4 additions & 0 deletions nimdoc/rst2html/source/rst_examples.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ Not a Nim Manual
:Authors: Andreas Rumpf, Zahary Karadjov
:Version: |nimversion|

.. role:: nim(code)
:language: nim
.. default-role:: nim

.. contents::


Expand Down
Loading