Improve documentation generation #1892

bkamins · 2019-07-19T11:13:07Z

The original reason for this PR is #1891. The problem I have is that locally the documentation of the functions mentioned there is generated correctly, but it is broken on GitHub for some strange reason.

So I have started to look at documentation generation mechanics and fixed some deprecations. Currently when I build documentation locally I get the following output (sorry that it is long, but maybe something important is lurking there). I will ask the questions I have below the output.

julia> include("make.jl")
[ Info: SetupBuildDirectory: setting up build directory.
[ Info: Doctest: skipped.
[ Info: ExpandTemplates: expanding markdown templates.
┌ Warning: duplicate docs found for 'DataFrame' in `@docs` block in src\lib\types.md:112-122
│ ```@docs
│ AbstractDataFrame
│ DataFrame
│ DataFrameRow
│ GroupedDataFrame
│ SubDataFrame
│ DataFrameRows
│ DataFrameColumns
│ RepeatedVector
│ StackedVector
│ ```
└ @ Documenter.Expanders C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\Expanders.jl:313
[ Info: CrossReferences: building cross-references.
┌ Warning: no doc found for reference '[`sum`](@ref)' in src\lib\functions.md.
└ @ Documenter.CrossReferences C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\CrossReferences.jl:160
┌ Warning: no doc found for reference '[`isequal`](@ref)' in src\lib\functions.md.
└ @ Documenter.CrossReferences C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\CrossReferences.jl:160
┌ Warning: no doc found for reference '[`sum`](@ref)' in src\lib\functions.md.
└ @ Documenter.CrossReferences C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\CrossReferences.jl:160
┌ Warning: no doc found for reference '[`skipmissing`](@ref)' in src\lib\functions.md.
└ @ Documenter.CrossReferences C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\CrossReferences.jl:160
┌ Warning: no doc found for reference '[`unique`](@ref)' in src\lib\functions.md.
└ @ Documenter.CrossReferences C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\CrossReferences.jl:160
┌ Warning: no doc found for reference '[`size`](@ref)' in src\lib\functions.md.
└ @ Documenter.CrossReferences C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\CrossReferences.jl:160
┌ Warning: no doc found for reference '[`size`](@ref)' in src\lib\functions.md.
└ @ Documenter.CrossReferences C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\CrossReferences.jl:160
┌ Warning: no doc found for reference '[`summary`](@ref)' in src\lib\types.md.
└ @ Documenter.CrossReferences C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\CrossReferences.jl:160
┌ Warning: no doc found for reference '[`first`](@ref)' in src\lib\types.md.
└ @ Documenter.CrossReferences C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\CrossReferences.jl:160
┌ Warning: no doc found for reference '[`last`](@ref)' in src\lib\types.md.
└ @ Documenter.CrossReferences C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\CrossReferences.jl:160
[ Info: CheckDocument: running document checks.
┌ Warning: 24 docstrings potentially missing:
│
│     DataFrames.showrowindices :: Tuple{IO,AbstractDataFrame,AbstractArray{Int64,1},Array{Int64,1},Int64,Int64,Any}
│     DataFrames.@csv2_str :: Tuple{Any,Vararg{Any,N} where N}
│     DataFrames.@csv_str :: Tuple{Any,Vararg{Any,N} where N}
│     Base.parent :: Tuple{GroupedDataFrame}
│     Base.first :: Tuple{AbstractDataFrame}
│     Base.first :: Tuple{AbstractDataFrame,Integer}
│     DataFrames.getprintedwidth :: Tuple{Array{Int64,1}}
│     Base.last :: Tuple{AbstractDataFrame}
│     Base.last :: Tuple{AbstractDataFrame,Integer}
│     DataFrames.writetable :: Tuple{AbstractString,AbstractDataFrame}
│     DataFrames.@wsv_str :: Tuple{Any,Vararg{Any,N} where N}
│     Base.issorted :: Union{Tuple{AbstractDataFrame}, Tuple{AbstractDataFrame,Any}}
│     Base.unique
│     DataFrames.ourshow :: Tuple{IO,Any}
│     DataFrames.@tsv_str :: Tuple{Any,Vararg{Any,N} where N}
│     DataFrames.compacttype :: Union{Tuple{Type}, Tuple{Type,Int64}}
│     DataFrames.showrows :: Union{Tuple{IO,AbstractDataFrame,AbstractArray{Int64,1},AbstractArray{Int64,1},Array{Int64,1}}, Tuple{IO,AbstractDataFrame,AbstractArray{Int64,1},AbstractArray{Int64,1},Array{Int64,1},Bool}, Tuple{IO,AbstractDataFrame,AbstractArray{Int64,1},AbstractArray{Int64,1},Array{Int64,1},Bool,Bool}, Tuple{IO,AbstractDataFrame,AbstractArray{Int64,1},AbstractArray{Int64,1},Array{Int64,1},Bool,Bool,Symbol}, Tuple{IO,AbstractDataFrame,AbstractArray{Int64,1},AbstractArray{Int64,1},Array{Int64,1},Bool,Bool,Symbol,Bool}, Tuple{IO,AbstractDataFrame,AbstractArray{Int64,1},AbstractArray{Int64,1},Array{Int64,1},Bool,Bool,Symbol,Bool,Any}}
│     DataFrames.inlinetable :: Tuple{AbstractString}
│     DataFrames.gennames :: Tuple{Integer}
│     Base.similar :: Union{Tuple{AbstractDataFrame}, Tuple{AbstractDataFrame,Integer}}
│     DataFrames.readtable :: Tuple{AbstractString}
│     Base.sortperm :: Tuple{AbstractDataFrame,Any}
│     DataFrames.getmaxwidths :: Union{Tuple{AbstractDataFrame,IO,AbstractArray{Int64,1},AbstractArray{Int64,1},Symbol}, Tuple{AbstractDataFrame,IO,AbstractArray{Int64,1},AbstractArray{Int64,1},Symbol,Any}}
│     DataFrames.getchunkbounds :: Tuple{Array{Int64,1},Bool,Int64}
└ @ Documenter.DocChecks C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\DocChecks.jl:63
[ Info: Populate: populating indices.
[ Info: RenderDocument: rendering document.
[ Info: HTMLWriter: rendering HTML pages.
┌ Warning: invalid local link: unresolved path in lib\functions.md
│   link = "@ref"
└ @ Documenter.Writers.HTMLWriter C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\Writers\HTMLWriter.jl:1173
┌ Warning: invalid local link: unresolved path in lib\functions.md
│   link = "@ref"
└ @ Documenter.Writers.HTMLWriter C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\Writers\HTMLWriter.jl:1173
┌ Warning: invalid local link: unresolved path in lib\functions.md
│   link = "@ref"
└ @ Documenter.Writers.HTMLWriter C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\Writers\HTMLWriter.jl:1173
┌ Warning: invalid local link: unresolved path in lib\functions.md
│   link = "@ref"
└ @ Documenter.Writers.HTMLWriter C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\Writers\HTMLWriter.jl:1173
┌ Warning: invalid local link: unresolved path in lib\functions.md
│   link = "@ref"
└ @ Documenter.Writers.HTMLWriter C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\Writers\HTMLWriter.jl:1173
┌ Warning: invalid local link: unresolved path in lib\functions.md
│   link = "@ref"
└ @ Documenter.Writers.HTMLWriter C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\Writers\HTMLWriter.jl:1173
┌ Warning: invalid local link: unresolved path in lib\functions.md
│   link = "lib/@ref"
└ @ Documenter.Writers.HTMLWriter C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\Writers\HTMLWriter.jl:1173
┌ Warning: invalid local link: unresolved path in lib\types.md
│   link = "@ref"
└ @ Documenter.Writers.HTMLWriter C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\Writers\HTMLWriter.jl:1173
┌ Warning: invalid local link: unresolved path in lib\types.md
│   link = "@ref"
└ @ Documenter.Writers.HTMLWriter C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\Writers\HTMLWriter.jl:1173
┌ Warning: invalid local link: unresolved path in lib\types.md
│   link = "@ref"
└ @ Documenter.Writers.HTMLWriter C:\Users\bogum\.julia\packages\Documenter\oEuOX\src\Writers\HTMLWriter.jl:1173
┌ Info: Deployment criteria:
│ - ✔ ENV["TRAVIS_REPO_SLUG"]="" occurs in repo="github.com/JuliaData/DataFrames.jl.git"
│ - ✘ ENV["TRAVIS_PULL_REQUEST"]="" is "false"
│ - ✔ ENV["TRAVIS_TAG"]="" is (i) empty or (ii) a valid VersionNumber
│ - ✘ ENV["TRAVIS_BRANCH"]="" matches devbranch="master" (if tag is empty)
│ - ✘ ENV["DOCUMENTER_KEY"] exists
│ - ✔ ENV["TRAVIS_EVENT_TYPE"]="" is not "cron"
└ Deploying: ✘

Questions (maybe @mortenpi or @fredrikekre could give advice how to fix it - thank you for your support in advance 😄):

Question 1: why do I get different results locally and on remote when building the documentation (as I have said - the original problem locally is not present, but some other problems pop up that are not present on remote). What I should do to ensure that both are identical (as this is essential for testing for me)

Question 2: how can I check what documentation is generated remotely on GitHub during CI (but before merging the PR)

Question 3: I have a warning Warning: duplicate docs found for 'DataFrame' in @docs block in src\lib\types.md:112-122 but I cannot see these duplicate docs - what is the reason of this warning?

Question 4: I get warnings like (this is one example, there are many) Warning: no doc found for reference '[isequal](@ref)' in src\lib\functions.md. - this is a reference to Base - how to make it work in a package? (so that I get a cross-reference from a package to Base)

Question 5: I get warnings Warning: 24 docstrings potentially missing: - how can I indicate to Documenter.jl that I do not intend to write a documentation of some function (this is mostly needed for unexported or deprecated functionality or the functionality that extends Base; I know we have still some docstrings missing where they are needed 😢 - these will get added)

Question 6: I get warnings like Warning: invalid local link: unresolved path in lib\functions.md │ link = "@ref" - how can I check which link is actually invalid (there are hundreds of them in the generated file)

(sorry for the laundry list - if the answer is "read the Documenter.jl documentation" then please say so I will go through it again 😄 - I did it once, and did not notice the answers for these question)

mortenpi · 2019-07-19T23:55:34Z

Question 1: why do I get different results locally and on remote when building the documentation (as I have said - the original problem locally is not present, but some other problems pop up that are not present on remote). What I should do to ensure that both are identical (as this is essential for testing for me)

What differences do you see?

Edit: regarding the "Basics" section being different -- when I build master locally, I also see that the docstrings are missing. It seems to have fixed itself on this branch though with a newer Documenter. Not sure if it's something here that fixed it, or perhaps some bugfix in Documenter.

Question 2: how can I check what documentation is generated remotely on GitHub during CI (but before merging the PR)

No good method for this, unfortunately. You'd have to set up something that uploads the generated docs to somewhere yourself.

Question 3: I have a warning Warning: duplicate docs found for 'DataFrame' in `@docs` block in src\lib\types.md:112-122 but I cannot see these duplicate docs - what is the reason of this warning?

DataFrame is included on types.md and on functions.md.

Question 4: I get warnings like (this is one example, there are many) Warning: no doc found for reference '[isequal](@ref)' in src\lib\functions.md. - this is a reference to Base - how to make it work in a package? (so that I get a cross-reference from a package to Base)

You can't at the moment. One issue is that trying to figure out the correct URLs is non-trivial (JuliaDocs/Documenter.jl#688).

Question 5: I get warnings Warning: 24 docstrings potentially missing: - how can I indicate to Documenter.jl that I do not intend to write a documentation of some function (this is mostly needed for unexported or deprecated functionality or the functionality that extends Base; I know we have still some docstrings missing where they are needed - these will get added)

Docstring missing warnings are for docstrings that you have already written, but are not being included on any manual page. So you should just add them into the manual to get rid of the warnings.

E.g. the first docstring missing warning in the OP refers to this I believe:

DataFrames.jl/src/abstractdataframe/show.jl

Lines 239 to 275 in b0d8a87

    
           """ 
        
               showrowindices(io::IO, 
        
                              df::AbstractDataFrame, 
        
                              rowindices::AbstractVector{Int}, 
        
                              maxwidths::Vector{Int}, 
        
                              leftcol::Int, 
        
                              rightcol::Int, 
        
                              rowid::Union{Int,Nothing}) 
        
           Render a subset of rows and columns of an `AbstractDataFrame` to an 
        
           I/O stream. For chunked printing, this function is used to print a 
        
           single chunk, starting from the first indicated column and ending with 
        
           the last indicated column. Assumes that the maximum string widths 
        
           required for printing have been precomputed. 
        
           # Arguments 
        
           - `io::IO`: The I/O stream to which `df` will be printed. 
        
           - `df::AbstractDataFrame`: An AbstractDataFrame. 
        
           - `rowindices::AbstractVector{Int}`: The indices of the subset of rows 
        
             that will be rendered to `io`. 
        
           - `maxwidths::Vector{Int}`: The pre-computed maximum string width 
        
             required to render each column. 
        
           - `leftcol::Int`: The index of the first column in a chunk to be rendered. 
        
           - `rightcol::Int`: The index of the last column in a chunk to be rendered. 
        
           - `rowid`: Used to handle showing `DataFrameRow`. 
        
           # Examples 
        
           ```jldoctest 
        
           julia> using DataFrames 
        
           julia> df = DataFrame(A = 1:3, B = ["x", "y", "z"]); 
        
           julia> DataFrames.showrowindices(stdout, df, 1:2, [1, 1, 5], 1, 2) 
        
           │ 1     │ 1 │ x │ 
        
           │ 2     │ 2 │ y │ 
        
           ``` 
        
           """

Question 6: I get warnings like Warning: invalid local link: unresolved path in lib\functions.md │ link = "@ref" - how can I check which link is actually invalid (there are hundreds of them in the generated file)

These errors are somewhat unhelpful indeed. I think they all come from the "no doc found for X" errors. In that situation, Documenter just keeps the @refs as the link URLs and then complains later that it's not a sane link.

bkamins · 2019-07-20T06:02:19Z

@mortenpi - a fantastic answer. Thank you!

bkamins · 2019-07-20T07:58:53Z

This should be good to merge (and test if all works OK again during deployment)

matbesancon · 2019-07-20T08:46:53Z

Looks good

bkamins · 2019-07-20T09:06:55Z

Thank you. I need an approval before I can merge this.

matbesancon · 2019-07-20T10:35:32Z

Done

bkamins · 2019-07-20T14:08:07Z

Thank you

small fixes in documentation

05faedf

small fixes

30d69aa

matbesancon approved these changes Jul 20, 2019

View reviewed changes

ararslan approved these changes Jul 20, 2019

View reviewed changes

ararslan merged commit f24c331 into JuliaData:master Jul 20, 2019

bkamins deleted the update_documenter branch July 20, 2019 14:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve documentation generation #1892

Improve documentation generation #1892

bkamins commented Jul 19, 2019

mortenpi commented Jul 19, 2019 •

edited

Loading

bkamins commented Jul 20, 2019

bkamins commented Jul 20, 2019

matbesancon commented Jul 20, 2019

bkamins commented Jul 20, 2019

matbesancon commented Jul 20, 2019

bkamins commented Jul 20, 2019

Improve documentation generation #1892

Improve documentation generation #1892

Conversation

bkamins commented Jul 19, 2019

mortenpi commented Jul 19, 2019 • edited Loading

bkamins commented Jul 20, 2019

bkamins commented Jul 20, 2019

matbesancon commented Jul 20, 2019

bkamins commented Jul 20, 2019

matbesancon commented Jul 20, 2019

bkamins commented Jul 20, 2019

mortenpi commented Jul 19, 2019 •

edited

Loading