Write inventory files (#2424)

* Write inventory files

* Add test of inventories

* Add documentation

* Write only `objects.inv`

Removes the code for writing `inventory.toml.gz`.

* Fix inventory for md filenames with spaces

* Set inventory version (#2449)

* Obtain inventory version from Project.toml

* Overwrite inventory version in `deploydocs`

This unconditionally overwrites an existing inventory `version` with the
version determined by `deploydocs`. This only happens when `deploydocs`
actually knows the version. For `dev`-deployments, an existing `version`
is left unchanged.

Under normal circumstances, the existing `version` will already be the
correct one, so this should do nothing. If the `deploydocs` actually has
to *change* the version, that would indicate that something is set up
incorrectly, and thus a warning will be printed.

* Restrict inventory version from `Project.toml` to standard setups

In order to prevent an auto-detected inventory version being incorrect
for unusual setups (e.g., monorepo), we only consider a
`docs/../Project.toml` file when looking up a version. Everything else
will leave the version blank in `makedocs`, respectively require
explicitly setting `inventory_version` manually

* Update the CHANGELOG

CHANGELOG.md

@@ -8,6 +8,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 
 * Added support for a `CollapsedDocStrings` key in every page's `@meta` block. Setting `CollapsedDocStrings = true` for a particular page essentially clicks the "Collapse all docstrings" in the navigation bar after the page loads, collapsing all docstrings on that page. This can make API documentation pages much more readable. ([#2282], [#2394], [#2459], [#2460])
+* Automatically write an `objects.inv` inventory file to be be included in every online deployment of the documentation. With the help of the [`DocumenterInterLinks` plugin](http://juliadocs.org/DocumenterInterLinks.jl/stable/), any other project using Documenter or [Sphinx](https://www.sphinx-doc.org/en/master/) can then externally link to any page, heading, or docstring in the documentation. ([#2366], [#2424])
+* Added a parameter `inventory_version` to the `HTML()` options that may be passed to `makedocs`. This option sets the `version` metadata field in the new `objects.inv` inventory file. In most cases, the project version for the inventory will be automatically detected from the main `Project.toml` file, respectively, during deployment in `deploydocs`, from the git tag of the release. Any project may still want to explicitly set `inventory_version` via, e.g., [`pkgversion(MyProject)`](https://docs.julialang.org/en/v1/base/base/#Base.pkgversion-Tuple{Module}) instead of relying on an auto-detected version. Projects with a non-standard setup (documentation-only-repos, monorepos) *should* modify their existing configuration to explicitly set `inventory_version`. ([#2449])
 
 ### Changed
 

Project.toml

@@ -1,11 +1,12 @@
 name = "Documenter"
 uuid = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
-version = "1.2.1"
+version = "1.3.0-dev"
 
 [deps]
 ANSIColoredPrinters = "a4c015fc-c6ff-483c-b24f-f7ea428134e9"
 AbstractTrees = "1520ce14-60c1-5f80-bbc7-55ef81b5835c"
 Base64 = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"
+CodecZlib = "944b1d66-785c-5afd-91f1-9de20f533193"
 Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
 DocStringExtensions = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
 Downloads = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
@@ -22,14 +23,17 @@ PrecompileTools = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
 REPL = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"
 RegistryInstances = "2792f1a3-b283-48e8-9a74-f99dce5104f3"
 SHA = "ea8e919c-243c-51af-8825-aaa63cd721ce"
+TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 Unicode = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"
 
 [compat]
 ANSIColoredPrinters = "0.0.1"
 AbstractTrees = "0.4"
 Base64 = "1.6"
+CodecZlib = "0.7"
 Dates = "1.6"
+DocInventories = "0.3.2"
 DocStringExtensions = "0.4, 0.5, 0.6, 0.7, 0.8, 0.9"
 Downloads = "1.4"
 Git = "1"
@@ -46,14 +50,16 @@ REPL = "1.6"
 Random = "1.6"
 RegistryInstances = "0.1"
 SHA = "0.7, 1.6"
+TOML = "1"
 Test = "1.6"
 UUIDs = "1.6"
 Unicode = "1.6"
 julia = "1.6"
 
 [extras]
+DocInventories = "43dc2714-ed3b-44b5-b226-857eda1aa7de"
 Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
 UUIDs = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"
 
 [targets]
-test = ["Random", "UUIDs"]
+test = ["Random", "UUIDs", "DocInventories"]

README.md

@@ -43,8 +43,9 @@ Contributions are very welcome, as are feature requests and suggestions. Please
 There are several packages that extend Documenter in different ways. The JuliaDocs organization maintains:
 
 * [DocumenterCitations.jl](https://github.com/JuliaDocs/DocumenterCitations.jl)
-* [DocumenterTools.jl](https://github.com/JuliaDocs/DocumenterTools.jl)
+* [DocumenterInterLinks.jl](https://github.com/JuliaDocs/DocumenterInterLinks.jl)
 * [DocumenterMarkdown.jl](https://github.com/JuliaDocs/DocumenterMarkdown.jl)
+* [DocumenterTools.jl](https://github.com/JuliaDocs/DocumenterTools.jl)
 
 Other third-party packages that can be combined with Documenter include:
 

docs/make.jl

@@ -32,6 +32,7 @@ makedocs(
             highlights = ["yaml"],
             ansicolor = true,
             size_threshold_ignore = ["release-notes.md"],
+            inventory_version = Documenter.DOCUMENTER_VERSION,
         )
     end,
     build = ("pdf" in ARGS) ? "build-pdf" : "build",

docs/src/lib/internals/writers.md

@@ -12,7 +12,7 @@ Modules = [
     Documenter.LaTeXWriter,
 ]
 Filter = t -> t !== asset
-Pages = ["writers.jl", "html/HTMLWriter.jl", "html/RD.jl", "latex/LaTeXWriter.jl"]
+Pages = ["writers.jl", "html/HTMLWriter.jl", "html/RD.jl", "html/write_inventory.jl", "latex/LaTeXWriter.jl"]
 ```
 ```@docs
 Documenter.Plugin

docs/src/man/guide.md

@@ -321,6 +321,36 @@ This also works across different pages in the same way. Note that these sections
 docstrings must be unique within a document.
 
 
+## External Cross-References
+
+Any project building its documentation with the most recent release of Documenter will
+generate an [`objects.inv` inventory](https://juliadocs.org/DocInventories.jl/stable/formats/#Sphinx-Inventory-Format)
+that can be found in the root of the [deployed documentation](@ref Hosting-Documentation).
+The [`DocumenterInterLinks` plugin](https://github.com/JuliaDocs/DocumenterInterLinks.jl#readme)
+allows to define a mapping in your `make.jl` file between an external project name
+and its inventory file, e.g.,
+
+```julia
+using DocumenterInterLinks
+
+links = InterLinks(
+    "Documenter" => "https://documenter.juliadocs.org/stable/objects.inv"
+)
+```
+
+That `InterLinks` object should then be passed to [`makedocs`](@ref) as an element of
+`plugins`. This enables the ability to cross-reference into the external documentation,
+e.g.,  of the `Documenter` package, using an [`@extref` link](@ref) with a syntax similar
+to the above [`@ref`](@ref Cross-Referencing), e.g.,
+
+```markdown
+See the [`Documenter.makedocs`](@extref) function.
+```
+
+See the [documentation of the `DocumenterInterLinks` package](http://juliadocs.org/DocumenterInterLinks.jl/stable/)
+for more details.
+
+
 ## Navigation
 
 Documenter can auto-generate tables of contents and docstring indexes for your document with

docs/src/man/syntax.md

@@ -268,6 +268,18 @@ document more than once is disallowed.
     Both user-defined and internally generated header reference labels take precedence over
     docstring references, in case there is a conflict.
 
+### `@extref` link
+
+Using the [`DocumenterInterLinks` plugin](https://github.com/JuliaDocs/DocumenterInterLinks.jl#readme),
+it is possible to cross-reference the documentation of other projects whose documentation
+is generated by Documenter or [Sphinx](https://www.sphinx-doc.org/en/master/), using
+`@extref` links similar to the built-in `@ref` links for internal references.
+
+See [External Cross-References](@ref) and the
+[documentation of the `DocumenterInterLinks` package](http://juliadocs.org/DocumenterInterLinks.jl/stable/)
+for more details.
+
+
 ## `@meta` block
 
 This block type is used to define metadata key/value pairs that can be used elsewhere in the

src/deploydocs.jl

@@ -268,6 +268,17 @@ function deploydocs(;
                 @debug "running extra build steps."
                 make()
             end
+            objects_inv = joinpath(realpath(target), "objects.inv")
+            if isfile(objects_inv)
+                inventory_version = _get_inventory_version(objects_inv)
+                deploy_version = _get_deploy_version(deploy_subfolder)
+                if !isempty(deploy_version) && (inventory_version != deploy_version)
+                    if !isempty(inventory_version)
+                        @warn "Inventory declares version `$inventory_version`, but `deploydocs` is for version `$deploy_version`. Overwriting inventory version."
+                    end
+                    _patch_inventory_version(objects_inv, deploy_version)
+                end
+            end
             @debug "pushing new documentation to remote: '$deploy_repo:$deploy_branch'."
             mktempdir() do temp
                 git_push(
@@ -282,6 +293,45 @@ function deploydocs(;
     end
 end
 
+
+function _get_inventory_version(objects_inv)
+    open(objects_inv) do input
+        for line in eachline(input)
+            if startswith(line, "# Version:")
+                return strip(line[11:end])
+            end
+        end
+        error("Invalid $objects_inv: missing or invalid version line")
+    end
+end
+
+
+function _patch_inventory_version(objects_inv, version)
+    objects_inv_patched = tempname()
+    open(objects_inv) do input
+        open(objects_inv_patched, "w") do output
+            for line in eachline(input; keep=true)
+                if startswith(line, "# Version:")
+                    @debug "Patched $objects_inv with version=$version"
+                    line = "# Version: $version\n"
+                end
+                write(output, line)
+            end
+        end
+    end
+    mv(objects_inv_patched, objects_inv; force=true)
+end
+
+
+function _get_deploy_version(foldername)
+    try
+        return string(VersionNumber(foldername))  # strips the leading "v" from foldername
+    catch
+        return ""
+    end
+end
+
+
 """
     git_push(
         root, tmp, repo;

src/html/HTMLWriter.jl

@@ -11,7 +11,8 @@ keyword arguments: `analytics`, `assets`, `canonical`, `disable_git`, `edit_link
 `prettyurls`, `collapselevel`, `sidebar_sitename`, `highlights`, `mathengine` and `footer`.
 
 **`sitename`** is the site's title displayed in the title bar and at the top of the
-*navigation menu. This argument is mandatory for [`HTMLWriter`](@ref).
+*navigation menu. It is also written into the inventory (see below).
+This argument is mandatory for [`HTMLWriter`](@ref).
 
 **`pages`** defines the hierarchy of the navigation menu.
 
@@ -24,7 +25,7 @@ selector will be hidden. The special value `git-commit` sets the value in the ou
 
 # `HTML` `Plugin` options
 
-The [`HTML`](@ref) [`Documenter.Plugin`](@ref) provides additional customization options
+The [`HTML`](@ref) object provides additional customization options
 for the [`HTMLWriter`](@ref). For more information, see the [`HTML`](@ref) documentation.
 
 # Page outline
@@ -37,15 +38,39 @@ then it is intended as the page title. This has two consequences:
    and in the `<title>` tag, unless specified in the `.pages` option.
 2. If the first heading is interpreted as being the page title, it is not displayed
    in the navigation sidebar.
+
+# Inventory
+
+The `HTMLWriter` automatically generates an `objects.inv` "inventory" file in
+the output `build` folder. This file contains a list of all pages, headers and
+docstrings in the documentation, and a relative URL that can be used to link to
+those items from an external source.
+
+Other projects that build their documentation with Documenter can use the
+[`DocumenterInterLinks` plugin](https://github.com/JuliaDocs/DocumenterInterLinks.jl#readme)
+to link to any other project with an inventory file, see
+[External Cross-References](@ref).
+
+The [format of the `objects.inv` file](https://juliadocs.org/DocInventories.jl/stable/formats/#Sphinx-Inventory-Format)
+is borrowed from the [Sphinx project](https://www.sphinx-doc.org/en/master/). It consists
+of a plain text header that includes the project name, taken from the `sitename` argument
+to [`Documenter.makedocs`](@ref), and a project `version` taken from the
+`inventory_version` argument of the [`HTML`](@ref) options, or automatically
+determined by [`deploydocs`](@ref Documenter.deploydocs) for tagged releases.
+The bulk of the file is a list of plain text records, compressed with gzip. See
+[Inventory Generation](http://juliadocs.org/DocumenterInterLinks.jl/stable/write_inventory/)
+for details on these records.
 """
 module HTMLWriter
 
 using Dates: Dates, @dateformat_str, now
 import Markdown
 using MarkdownAST: MarkdownAST, Node
+using TOML
 import JSON
 import Base64
 import SHA
+using CodecZlib
 
 import ..Documenter
 using Documenter: NavNode
@@ -393,6 +418,13 @@ executable to be available in `PATH` or to be passed as the `node` keyword.
 
 **`highlightjs`** file path to custom highglight.js library to be used with prerendering.
 
+**`inventory_version`** a version string to write to the header of the
+`objects.inv` inventory file. This should be a valid version number without a `v` prefix.
+Defaults to the `version` defined in the `Project.toml` file in the parent folder of the
+documentation root. Setting this to an empty string leaves the `version` in the inventory
+unspecified until [`deploydocs`](@ref Documenter.deploydocs) runs and automatically sets the
+`version` for any tagged release.
+
 # Default and custom assets
 
 Documenter copies all files under the source directory (e.g. `/docs/src/`) over
@@ -454,6 +486,7 @@ struct HTML <: Documenter.Writer
     size_threshold_warn :: Int
     size_threshold_ignore :: Vector{String}
     example_size_threshold :: Int
+    inventory_version ::  Union{String,Nothing}
 
     function HTML(;
             prettyurls    :: Bool = true,
@@ -482,6 +515,7 @@ struct HTML <: Documenter.Writer
             # seems reasonable, and that would lead to ~80 KiB, which is still fine
             # and leaves a buffer before hitting `size_threshold_warn`.
             example_size_threshold :: Union{Integer, Nothing} = 8 * 2^10, # 8 KiB
+            inventory_version = nothing,
 
             # deprecated keywords
             edit_branch   :: Union{String, Nothing, Default} = Default(nothing),
@@ -537,6 +571,7 @@ struct HTML <: Documenter.Writer
             collapselevel, sidebar_sitename, highlights, mathengine, description, footer,
             ansicolor, lang, warn_outdated, prerender, node, highlightjs,
             size_threshold, size_threshold_warn, size_threshold_ignore, example_size_threshold,
+            (isnothing(inventory_version) ? nothing : string(inventory_version))
         )
     end
 end
@@ -582,6 +617,7 @@ function prepare_prerendering(prerender, node, highlightjs, highlights)
 end
 
 include("RD.jl")
+include("write_inventory.jl")
 
 struct SearchRecord
     src :: String
@@ -782,6 +818,8 @@ function render(doc::Documenter.Document, settings::HTML=HTML())
         println(io, json_jsescape(ctx.search_index), "\n}")
     end
 
+    write_inventory(doc, ctx)
+
     generate_siteinfo_json(doc.user.build)
 end