Rewrite documentation

.gitignore

@@ -1,15 +1,12 @@
 # Generated manuscript output files
-output/
+output/*
+!output/README.md
 
 webpage/*.ots
 
 # Manubot cache directory
 ci/cache
 
-# Generated reference files
-references/generated/*
-!references/generated/README.md
-
 # Python
 __pycache__/
 *.pyc

CONTRIBUTING.md

@@ -1,30 +1,120 @@
-# Manuscript contribution guidelines
+# Manubot usage & contribution guidelines
 
-## Markdown
+This repository uses the [Manubot](https://github.com/greenelab/manubot) to automatically produce a manuscript from its source.
 
-The paper will be written using markdown. Markdown files use the `.md` extension.
-Check out the [CommonMark Help](http://commonmark.org/help/) page for an introduction to formatting options provided by markdown.
-In addition, to standard markdown features, we support [markdown tables](https://help.github.com/articles/organizing-information-with-tables/ "GitHub Help: Organizing information with tables") and a custom citation syntax.
-Check out [Tables Generator](http://www.tablesgenerator.com/markdown_tables) for creating markdown tables.
+## Manubot Markdown
 
-The custom citation guidelines are as follows:
+Manuscript text should be written in markdown files, which should be located in [`content`](content) with a digit prefix for ordering (e.g. `01.`, `02.`, etc.) and a `.md` extension.
 
-1. Always use a DOI for the version of record if available.
-  Cite DOIs like `[@doi:10.15363/thinklab.4]`.
-2. If the version of record doesn't have a DOI but does have a PubMed ID, cite like `[@pmid:26158728]`.
-3. If the article is an _arXiv_ preprint, cite like `[@arxiv:1508.06576]`.
-4. If and only if the article has none of the above, cite with the URL like `[@url:http://openreview.net/pdf?id=Sk-oDY9ge]`.
+For basic formatting, check out the [CommonMark Help](http://commonmark.org/help/) page for an introduction to the formatting options provided by standard markdown.
+In addition, manubot supports an extended version of markdown, tailored for scholarly writing.
 
-You cite multiple items at once like `[@doi:10.15363/thinklab.4 @pmid:26158728 @arxiv:1508.06576]`.
+### Tables
 
-The system also supports tags, which may be helpful when a single reference is cited many times.
-For example, you can add a reference to the tag `study_x` using the following syntax: `[@tag:study_x]`.
-If you add references that use tags, make sure to add those tags and their corresponding citations to [`references/tags.tsv`](references/tags.tsv).
+Manubot supports [markdown tables](https://help.github.com/articles/organizing-information-with-tables/ "GitHub Help: Organizing information with tables").
 
-If the automatically extracted reference information contains errors, it can be [manually overridden](references/README.md#reference-overrides).
+```md
+| Column 1 | Column 2 | Column 3 |
+|----------|----------|----------|
+| value_a | 1 | 47 |
+| value_b | 2 | 56 |
 
-## Authorship information
+Table: Caption for this example table. {#tbl:example-id}
+```
 
-Authorship information and order is extracted from [`authors.tsv`](../content/authors.tsv).
-To add an author, insert a row into this table.
-We recommend authors add themselves to `authors.tsv` via pull request (when requested by a maintainer), thereby signaling that they've read and approved the manuscript.
+Support for table numbering and citation is provided by [`pandoc-tablenos`](https://github.com/tomduck/pandoc-tablenos).
+Above, `{#tbl:example-id}` sets the table ID, which creates an HTML anchor and allows citing the table like `@tbl:example-id`.
+For easy creation of markdown tables, check out the [Tables Generator](http://www.tablesgenerator.com/markdown_tables) webapp.
+
+### Figures
+
+Figures can be included with the following markdown:
+
+```md
+![Caption for the example figure.](url_or_path_to figure){#fig:example-id}
+```
+
+Support for figure numbering and citation is provided by [`pandoc-fignos`](https://github.com/tomduck/pandoc-fignos).
+This figure can be cited in the text using `@fig:example-id`.
+In context, a figure citation may look like: `Figure {@fig:example-id}B shows…`.
+
+For images created by the manuscript authors that are hosted elsewhere on GitHub, we recommend using a versioned GitHub URL to embed figures, thereby preserving exact image provenance.
+When embedding SVG images hosted on GitHub, passing the URL through [RawGit](https://rawgit.com/) is necessary.
+An example of a URL that has been passed through RawGit is:
+
+```
+https://cdn.rawgit.com/greenelab/scihub/572d6947cb958e797d1a07fdb273157ad9154273/figure/citescore.svg
+```
+
+Figures placed in the [`content/images`](content/images) directory can be embedded using their relative path.
+For example, we embed an [ORCID](https://orcid.org/) icon inline using:
+
+```md
+![ORCID icon](images/orcid.svg){height="13px"}
+```
+
+The bracketed text following the image declaration is interpreted by Pandoc's [`link_attributes`](http://pandoc.org/MANUAL.html#extension-link_attributes) extension.
+For example, the following will override the figure number to be "S1" and set the image width to 5 inches:
+
+```md
+{#fig:supplement tag="S1" width="5in"}
+```
+
+### Citations
+
+Manubot supports Pandoc [citations](http://pandoc.org/MANUAL.html#citations) via `pandoc-citeproc`.
+However, Manubot performs an automated citation processing and metadata retrieval layer on top of citations.
+Therefore, citations must by of the following form: `@source:identifier`, where `source` is one of the options described below.
+When choosing which source to use for a citation, we recommend the following order:
+
+1. DOI (Digital Object Identifier), cite like `@doi:10.15363/thinklab.4`.
+2. PubMed ID, cite like `@pmid:26158728`.
+3. _arXiv_ ID, cite like `@arxiv:1508.06576v2`.
+4. URL / webpage, cite like `@url:http://openreview.net/pdf?id=Sk-oDY9ge`.
+
+You cite multiple items at once like:
+
+```md
+Here is a sentence with several citations [@doi:10.15363/thinklab.4; @pmid:26158728; @arxiv:1508.06576].
+```
+
+Note that multiple citations must be `; `-separated.
+
+#### Citation tags
+
+The system also supports citation tags, which are recommended in the following applications:
+
+1. A citation's identifier contains forbidden characters, such as `;` or ending in a non-alphanumeric character other than `/`. In these instances, you must use a tag.
+2. A single reference is cited many times.
+  Therefore, it may be easier to define a tag, so if the citation updates (e.g. a newer version available), the update must only be applied in one place.
+
+Tags should be defined in [`content/citation-tags.tsv`](content/citation-tags.tsv).
+If `citation-tags.tsv` defines the tag `study-x`, then this study can be cited like `@tag:study-x`.
+
+## Reference metadata
+
+The Manubot workflow requires the bibliographic details for references (the set of all cited works) as CSL (Citation Style Language) Items / [citeproc JSON](http://citeproc-js.readthedocs.io/en/latest/csl-json/markup.html#csl-json-items).
+The Manubot attempts to automatically retrieve metadata and generate valid citeproc JSON for references, which is exported to `output/references.json`.
+However, in some cases the Manubot fails to retrieve metadata or generates incorrect or incomplete citeproc metadata.
+Errors are most common for `url` references.
+For these references, you can manually specify the citeproc in [`content/manual-references.json`](content/manual-references.json), which will override the automatically generated reference data.
+To do so, create a new citeproc record that contains the field `"standard_citation"` with the appropriate reference identifier as its value.
+The identifier can be obtained from the `standard_citation` column of `citations.tsv`, which is located in the base directory of the `output` branch or in the `output` subdirectory if you build the manuscript locally.
+As an example, `manual-references.json` contains the field:
+
+```json
+"standard_citation": "url:https://github.com/greenelab/manubot-rootstock"
+```
+
+For guidance on how to cite different document types, please refer to [these CSL examples](https://github.com/aurimasv/zotero-import-export-formats/blob/a51c342e66bebd97b73a7230047b801c8f7bb690/CSL%20JSON.json).
+
+## Manuscript metadata
+
+[`content/metadata.yaml`](content/metadata.yaml) contains manuscript metadata that gets passed through to Pandoc, via a [`yaml_metadata_block`](http://pandoc.org/MANUAL.html#extension-yaml_metadata_block).
+`metadata.yaml` should contain the manuscript `title`, `authors` list, and `keywords`.
+Additional metadata, such as `date`, will automatically be created by the Manubot.
+We recommend authors add themselves to `metadata.yaml` via pull request (when requested by a maintainer), thereby signaling that they've read and approved the manuscript.
+
+## Manubot Feedback
+
+If you experience any issues with the Manubot or would like to contribute to its source code, please visit [`greenelab/manubot`](https://github.com/greenelab/manubot) or [`greenelab/manubot-rootstock`](https://github.com/greenelab/manubot-rootstock).

README.md

@@ -15,10 +15,35 @@ To see what's incoming, check the [open pull requests](https://github.com/greene
 Instructions for using Manubot Rootstock for your own manuscript are still evolving.
 The recommended approach is to clone this repository, as detailed [here](https://github.com/greenelab/manubot-rootstock/issues/6#issuecomment-314541837).
 
-## Source
+## Repository directories & files
 
-The manuscript source is located in [`content`](content).
-Text should be written in markdown files, with a digit prefix for ordering (e.g. `01.`, `02.`, etc.) and a `.md` suffix.
+See [`CONTRIBUTING.md`](CONTRIBUTING.md) for documentation on how to create or contribute to a manuscript.
+
+The directories are as follows:
+
++ [`content`](content) contains the manuscript source, which includes markdown files as well as inputs for citations and references.
++ [`output`](output) contains the outputs (generated files) from the manubot including the resulting manuscripts.
+  You should not edit these files manually, because they will get overwritten.
++ [`webapp`](webapp) is a directory meant to be rendered a webpage for viewing the HTML manuscript.
++ [`build`](build) contains commands and tools for building the manuscript.
++ [`ci`](ci) contains files necessary for deployment via continuous integration.
+  For the CI configuration, see [`.travis.yml`](.travis.yml).
+
+## Local execution
+
+You can build the manuscript locally on POSIX systems by running the following commands.
+
+```sh
+# Activate the manubot conda environment
+source activate manubot
+
+# Build the manuscript
+sh build/build.sh
+
+# View the manuscript locally at http://localhost:8000/
+cd webpage
+python -m http.server
+```
 
 ## Continuous Integration
 
@@ -28,7 +53,7 @@ When you make a pull request, Travis CI will test whether your changes break the
 The build process aims to detect common errors, such as invalid references.
 If your build fails, see the Travis CI logs for the cause of failure and revise your pull request accordingly.
 
-When a pull request is merged, Travis CI performs the build and writes the results to the [`gh-pages`](https://github.com/greenelab/manubot-rootstock/tree/gh-pages) and [`references`](https://github.com/greenelab/manubot-rootstock/tree/references) branches.
+When a pull request is merged, Travis CI performs the build and writes the results to the [`gh-pages`](https://github.com/greenelab/manubot-rootstock/tree/gh-pages) and [`output`](https://github.com/greenelab/manubot-rootstock/tree/output) branches.
 The `gh-pages` branch hosts the following URLs:
 
 + **HTML manuscript** at https://greenelab.github.io/manubot-rootstock/
@@ -49,7 +74,7 @@ All files matched by the following glob patterns are dual licensed under CC BY 4
 
 + `*.sh`
 + `*.py`
-+ `*.yml`
++ `*.yml` / `*.yaml`
 + `*.json`
 + `*.bib`
 + `*.tsv`
@@ -60,5 +85,6 @@ All other files are only available under CC BY 4.0, including:
 + `*.md`
 + `*.html`
 + `*.pdf`
++ `*.docx`
 
 Please open [an issue](https://github.com/greenelab/manubot-rootstock/issues) for any question related to licensing.

build/environment.yml

@@ -13,7 +13,7 @@ dependencies:
   - pip:
     - errorhandler==2.0.1
     - ghp-import==0.5.5
-    - git+https://github.com/dhimmel/manubot@21bf789294c0e9c9231aef5de73fc0111853b6bf
+    - git+https://github.com/dhimmel/manubot@bac27de72e3c424da151a789e1f034163c45730e
     - pandoc-eqnos==0.16
     - pandoc-fignos==0.20
     - pandoc-tablenos==0.16

output/README.md

@@ -0,0 +1,16 @@
+# Generated citation / reference files
+
+The `output` branch contains files automatically generated by the manuscript build process.
+It consists of the contents of the `output` directory of the `master` branch.
+These files are not tracked in `master`, but instead written to the `output` branch by continuous Travis CI builds.
+
+## Files
+
+This directory contains the following files:
+
++ [`citations.tsv`](citations.tsv) is a table of citations extracted from the manuscript and the corresponding standard citations and citation IDs.
++ [`manuscript.md`](manuscript.md) is a markdown document of all manuscript sections, with citation strings replaced by citation IDs.
++ [`references.json`](references.json) is CSL-JSON file of bibliographic item metadata ([see specification](https://github.com/citation-style-language/schema/blob/master/csl-data.json)) for all references.
++ [`variables.json`](variables.json) contains variables that were passed to the jinja2 templater. These variables contain those automatically generated by the manubot as well as those provided by the user via the `--template-variables-path` option.
+
+Pandoc consumes `manuscript.md` and `references.json` to create the formatted manuscript, which is exported to `manuscript.html`, `manuscript.pdf`, and optionally `manuscript.docx`.