Merge pull request #199 from ConorMacBride/migrate-docs

Migrate docs from `README.rst` to website

docs/conf.py

@@ -38,6 +38,7 @@
 # ones.
 extensions = [
     'sample_summaries',
+    'sphinx.ext.intersphinx',
     'sphinx_design',
 ]
 
@@ -49,6 +50,9 @@
 # This pattern also affects html_static_path and html_extra_path.
 exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
 
+intersphinx_mapping = {
+    "matplotlib": ("https://matplotlib.org/stable", None),
+}
 
 # -- Options for HTML output -------------------------------------------------
 

docs/hash_mode.rst

@@ -0,0 +1,83 @@
+.. title:: Hash comparison mode
+
+#####################
+Hash Comparison Mode
+#####################
+
+This how-to guide will show you how to use the hash comparison mode of ``pytest-mpl``.
+
+In this mode, the hash of the image is compared to the hash of the baseline image.
+Only the hash value of the baseline image, rather than the full image, needs to be stored in the repository.
+This means that the repository size is reduced, and the images can be regenerated if necessary.
+This approach does however make it more difficult to visually inspect any changes to the images.
+
+If your goal is to not commit any images to the code repository, then you should consider using :doc:`hybrid mode <hybrid_mode>` instead.
+In this mode, the hashes can be stored in the code repository, while the baseline images are stored in a separate repository and accessed through a URL when testing.
+
+Generating baseline hashes
+==========================
+
+Once a suite of image comparison tests have been written, baseline hashes should be generated by setting ``--mpl-generate-hash-library``:
+
+.. code-block:: bash
+
+    pytest --mpl-generate-hash-library=your_project/tests/hashes.json
+
+It is important to visually inspect the figures before generating baseline hashes.
+So, as well as generating baseline hashes, this command runs baseline image comparison tests.
+If no baseline images exist in the default directory, this command will fail.
+
+A better option is to generate baseline images along with the baseline hashes to ensure that the images are as expected, even if you do not wish to use them for comparison:
+
+.. code-block:: bash
+
+    pytest \
+      --mpl-generate-hash-library=your_project/tests/mpl35_ft261.json \
+      --mpl-generate-path=baseline
+
+To assist with inspecting the generated images (and hashes), a HTML summary report can be generated by setting ``--mpl-generate-summary``:
+
+.. code-block:: bash
+
+    pytest \
+      --mpl-generate-hash-library=test_hashes.json \
+      --mpl-generate-path=baseline \
+      --mpl-results-path=results \
+      --mpl-generate-summary=html,json
+
+:summary:`test_html_generate`
+
+You should choose a directory within you repository to store the baseline hashes.
+It's usually a good idea to encode the Matplotlib version and the FreeType version in the filename, e.g. ``mpl35_ft261.json``.
+The hash library file should then be committed to the repository.
+
+Running hash comparison tests
+=============================
+
+When running the tests, the ``--mpl`` flag should be used along with a :ref:`configured hash library path <hash-library>` to enable baseline hash comparison testing:
+
+.. code-block:: bash
+
+    pytest --mpl \
+      --mpl-hash-library=your_project/tests/mpl35_ft261.json
+
+Optionally, a HTML summary report can be generated by setting ``--mpl-generate-summary``:
+
+.. code-block:: bash
+
+    pytest --mpl \
+      --mpl-hash-library=your_project/tests/mpl35_ft261.json \
+      --mpl-results-path=results \
+      --mpl-generate-summary=html,json
+
+:summary:`test_html_hashes_only`
+
+The ``--mpl-results-path`` flag can be used to set the directory where the generated HTML summary will be stored.
+If this is not set, the images will be stored in a temporary directory.
+
+Continue reading
+================
+
+``pytest-mpl`` has many configuration options that can be used to customize the behavior of the hash comparison mode.
+Only a few of the most commonly used options are covered in this guide.
+See the :doc:`configuration options documentation <configuration>` for full details.

docs/hybrid_mode.rst

@@ -0,0 +1,90 @@
+.. title:: Hybrid mode
+
+##############################
+Hybrid Mode: Hashes and Images
+##############################
+
+This how-to guide will show you how to use the hybrid mode of ``pytest-mpl``.
+
+For a full description of the hybrid mode, see the :ref:`hybrid mode section of the get started guide <hybrid-usage>`.
+In summary, hybrid mode uses both baseline images and hashes.
+First, the hash of the image is compared to the hash of the baseline image.
+If the hashes match, the test passes.
+If the hashes do not match, the test fails.
+
+The difference with hybrid mode is that a baseline image comparison will also be carried out if the hashes do not match, or always :ref:`if this has been configured <results-always>`.
+The purpose of the additional image comparison (which does not affect the test result) is to allow the user to visually inspect the difference between the baseline image and the image generated by the test.
+
+In order to keep the code repository size small, it is recommended to store the baseline hashes in the code repository, and the baseline images in a separate repository.
+The baseline hashes should be updated where appropriate in PRs to the code repository.
+However, the baseline images are not updated in these PRs.
+Instead, they should be updated once the PR has been merged, preferably by a CI job.
+
+Another benefit of only updating the baseline images once the PR has been merged is that the PR tests will show the difference between the remote baseline images and the images generated by the PR.
+Even though the tests will pass when the baseline hash matches, the images will still be compared and the difference will be shown in the HTML test summary report, which is useful when reviewing the PR.
+
+Generating baseline hashes and images
+=====================================
+
+Once a suite of image comparison tests have been written, baseline hashes and images should be generated by setting ``--mpl-generate-path`` and ``--mpl-generate-hash-library``:
+
+.. code-block:: bash
+
+    pytest \
+      --mpl-generate-hash-library=your_project/tests/test_hashes.json \
+      --mpl-generate-path=baseline \
+      --mpl-results-path=results \
+      --mpl-generate-summary=html,json
+
+:summary:`test_html_generate`
+
+Open the HTML summary file and inspect the figures to ensure that the baseline images are correct.
+If they are, the baseline hashes can be committed to the code repository.
+It's usually a good idea to encode the Matplotlib version and the FreeType version in the filename, e.g. ``mpl35_ft261.json``.
+The baseline images should be copied to a separate repository; preferably within a version specific directory, e.g. ``mpl35_ft261/``.
+
+Running hash comparison tests
+=============================
+
+When running the tests, the ``--mpl`` flag should be used along with a configured :ref:`hash library path <hash-library>` and :ref:`baseline image path <baseline-dir>` to enable hybrid mode testing:
+
+.. code-block:: bash
+
+    pytest --mpl \
+      --mpl-hash-library=your_project/tests/mpl35_ft261.json \
+      --mpl-baseline-path=https://raw.githubusercontent.com/your-org/your-project-figure-tests/mpl35_ft261/ \
+      --mpl-results-path=results \
+      --mpl-generate-summary=html,json
+
+:summary:`test_html`
+
+The ``--mpl-results-path`` flag can be used to set the directory where the generated HTML summary will be stored.
+If this is not set, the images will be stored in a temporary directory.
+
+Notice that the baseline image path is set to a URL, which is the location of the baseline images in the separate repository.
+When the baseline image comparison is carried out, the baseline images will be downloaded from this URL.
+
+It is recommended to create a CI job that updates the baseline images in the separate repository once the PR has been merged.
+The CI job should run when new commits are pushed to the default branch.
+The baseline images should only be regenerated and updated if the tests pass in the hash comparison mode.
+
+.. rubric:: Aside: basic HTML summary
+
+This is what the basic HTML summary looks like for the same test as above:
+
+.. code-block:: bash
+
+    pytest --mpl \
+      --mpl-hash-library=your_project/tests/mpl35_ft261.json \
+      --mpl-baseline-path=https://raw.githubusercontent.com/your-org/your-project-figure-tests/mpl35_ft261/ \
+      --mpl-results-path=results \
+      --mpl-generate-summary=basic-html,json
+
+:summary:`test_basic_html`
+
+Continue reading
+================
+
+``pytest-mpl`` has many configuration options that can be used to customize the behavior of the hybrid mode.
+Only a few of the most commonly used options are covered in this guide.
+See the :doc:`configuration options documentation <configuration>` for full details.

docs/image_mode.rst

@@ -0,0 +1,70 @@
+.. title:: Image comparison mode
+
+#####################
+Image Comparison Mode
+#####################
+
+This how-to guide will show you how to use the image comparison mode of ``pytest-mpl``.
+This is the default mode when ``pytest-mpl`` is invoked with ``--mpl``.
+
+In this mode, ``pytest-mpl`` will compare the images generated by the test with a baseline image.
+If the images are different, :ref:`to a specified RMS tolerance <tolerance>`, the test will fail.
+The test will also fail if the baseline image is not found or if the test does not generate a figure.
+If the figure has a different aspect ratio to the baseline image, the test will also fail.
+
+Generating baseline images
+==========================
+
+Once a suite of image comparison tests have been written, baseline images should be generated by setting ``--mpl-generate-path``:
+
+.. code-block:: bash
+
+    pytest --mpl-generate-path=your_project/tests/baseline
+
+You should choose a directory within you repository to store the baseline images.
+The generated images should be checked to ensure they are as expected.
+The images should then be committed to the repository.
+
+To assist with inspecting the generated images, a HTML summary report can be generated by setting ``--mpl-generate-summary``:
+
+.. code-block:: bash
+
+    pytest \
+      --mpl-generate-path=your_project/tests/baseline \
+      --mpl-generate-summary=html,json
+
+:summary:`test_html_generate_images_only`
+
+Running image comparison tests
+==============================
+
+When running the tests, the ``--mpl`` flag should be used to enable baseline image comparison testing:
+
+.. code-block:: bash
+
+    pytest --mpl \
+      --mpl-baseline-path=your_project/tests/baseline
+
+The :ref:`baseline path <baseline-dir>` should be set to the directory containing the baseline images.
+If the baseline images are not found, the test will fail.
+
+Optionally, a HTML summary report can be generated by setting ``--mpl-generate-summary``:
+
+.. code-block:: bash
+
+    pytest --mpl \
+      --mpl-baseline-path=your_project/tests/baseline \
+      --mpl-results-path=results \
+      --mpl-generate-summary=html,json
+
+:summary:`test_html_images_only`
+
+The ``--mpl-results-path`` flag can be used to set the directory where the generated HTML summary will be stored.
+If this is not set, the images will be stored in a temporary directory.
+
+Continue reading
+================
+
+``pytest-mpl`` has many configuration options that can be used to customize the behavior of the image comparison mode.
+Only a few of the most commonly used options are covered in this guide.
+See the :doc:`configuration options documentation <configuration>` for full details.

docs/index.rst

@@ -7,14 +7,20 @@
 
     installing
     usage
+    image_mode
+    hash_mode
+    hybrid_mode
     configuration
-    summaries
 
 ##################################
 pytest-mpl |release| documentation
 ##################################
 
-This is a plugin to facilitate image comparison for Matplotlib figures in pytest.
+``pytest-mpl`` is a `pytest <https://docs.pytest.org>`__ plugin to facilitate image comparison for `Matplotlib <http://www.matplotlib.org>`__ figures.
+
+For each figure to test, an image is generated and then subtracted from an existing reference image.
+If the RMS of the residual is larger than a user-specified tolerance, the test will fail.
+Alternatively, the generated image can be hashed and compared to an expected value.
 
 ************
 Installation
@@ -53,21 +59,25 @@ Learning resources
         Tutorials
         ^^^
 
-        - :doc:`Basic usage <usage>`
+        - :doc:`Get started <usage>`
 
     .. grid-item-card::
         :padding: 2
 
         How-tos
         ^^^
 
+        - :doc:`Image comparison mode <image_mode>`
+        - :doc:`Hash comparison mode <hash_mode>`
+        - :doc:`Hybrid mode <hybrid_mode>`
 
     .. grid-item-card::
         :padding: 2
 
         Understand how pytest-mpl works
         ^^^
 
+        Explanatory information is included where relevant throughout the documentation.
 
     .. grid-item-card::
         :padding: 2
@@ -76,14 +86,12 @@ Learning resources
         ^^^
 
         - :doc:`Configuration <configuration>`
-        - :doc:`Summary Reports <summaries>`
-
 
 ************
 Contributing
 ************
 
-pytest-mpl is a community project maintained for and by its users.
+``pytest-mpl`` is a community project maintained for and by its users.
 There are many ways you can help!
 
 - Report a bug or request a feature `on GitHub <https://github.com/matplotlib/pytest-mpl/issues>`__