Refactor the spec to better explain the semantic model

The semantic model is a much more important concept now that we have two
representations for models (the AST and IDL), and these representations
are meant to build up the same graph of shapes regardless of their
different syntactic features. The spec is also now laid out in an easier
to understand way since it explains the high-level concepts, shapes, and
traits before diving into the details of the IDL ABNF.

Various aspects of the model have been clarified:

* Applying traits externally is the same as applying them inline in the
  semantic model.
* Simplifications to the documentation_comment ABNF so that it is now
  just one of the two kinds of `comment` productions.
* A broader overview of the model, including transformations.
* More information on namespaces, including that there can be multiple
  namespaces loaded into the semantic model.
* More information on how and why models would be merged together (it's
  left up to tooling to combine and merge models, but the semantics of
  merging models are described in the spec).
* Lots of minor error corrections in the spec.
* The special behavior around resolving relative shape IDs in the
  "conflicts" property of the `trait` trait have been removed since that
  is at odds with the rest of the model.

This commit moves various headings around and removes some pages, but
redirects are issued when a removed page is visited.

This commit also ensures that requirements.txt is used for docs.
Since we're using a github repo for the redirects package, we need to
install it using a special syntax. There some docs around how to pull
this off with setup.py, but I couldn't get it to work:
https://python-packaging.readthedocs.io/en/latest/dependencies.html#packages-not-on-pypi

docs/Makefile

@@ -10,6 +10,7 @@ help:
 	@echo "  openhtml    to preview the built index.html"
 
 install:
+	pip install -r requirements.txt .
 	pip install -e .
 
 clean:

docs/requirements.txt

@@ -1,3 +1,6 @@
 sphinx>=1.7.0,<1.8.0
 pygments==2.4.2
 sphinx-tabs==1.1.7
+
+# This package has not been published to pypi
+-e git+https://github.com/munnerz/redirects.git#egg=sphinxcontrib-redirects

docs/smithy/__init__.py

@@ -1,13 +1,17 @@
 import re
-from sphinx.writers.html import HTMLTranslator as SphinxHTMLTranslator
-from docutils.writers.html4css1 import HTMLTranslator as BaseTranslator
-from docutils import nodes
 
 from sphinx.locale import _
+from sphinx.writers.html import HTMLTranslator as SphinxHTMLTranslator
+
+from docutils import nodes
+from docutils.parsers.rst import Directive, directives
+from docutils.statemachine import StringList
+from docutils.writers.html4css1 import HTMLTranslator as BaseTranslator
 
 
 def setup(app):
     app.set_translator('html', HTMLTranslator)
+    app.add_directive("text-figure", TextFigure)
 
 # Finds the href part of a header.
 HREF = re.compile('href="([^"]+)"')
@@ -51,3 +55,47 @@ def visit_productionlist(self, node):
             self.body.append('\n')
         self.body.append('</pre>\n')
         raise nodes.SkipNode
+
+
+# Use .. text-figure to create text diagrams that look like figures.
+class TextFigure(Directive):
+
+    required_arguments = 0
+    optional_arguments = 2
+    option_spec = {
+        'caption': directives.unchanged_required,
+        'name': directives.unchanged,
+    }
+    has_content = True
+    final_argument_whitespace = False
+
+    def run(self):
+        self.assert_has_content()
+        text = '\n'.join(self.content)
+        literal = nodes.literal_block(text, text)
+        literal = self.__container_wrapper(literal, self.options.get('caption'))
+        self.add_name(literal)
+        literal['classes'].append("text-figure")
+        return [literal]
+
+    # This is a modified version of https://github.com/sphinx-doc/sphinx/blob/3.x/sphinx/directives/code.py
+    # that places the caption after the text rather than before.
+    def __container_wrapper(self, literal_node, caption):
+        container_node = nodes.container('', literal_block=True,
+                                         classes=['literal-block-wrapper'])
+        parsed = nodes.Element()
+        self.state.nested_parse(StringList([caption], source=''),
+                                self.content_offset, parsed)
+        if isinstance(parsed[0], nodes.system_message):
+            msg = __('Invalid caption: %s' % parsed[0].astext())
+            raise ValueError(msg)
+        elif isinstance(parsed[0], nodes.Element):
+            caption_node = nodes.caption(parsed[0].rawsource, '',
+                                         *parsed[0].children)
+            caption_node.source = literal_node.source
+            caption_node.line = literal_node.line
+            container_node += literal_node
+            container_node += caption_node
+            return container_node
+        else:
+            raise RuntimeError  # never reached

docs/source/1.0/guides/building-models/build-config.rst

@@ -746,8 +746,8 @@ orphaned shapes.
 excludeMetadata
 ---------------
 
-Removes :ref:`metadata` key-value pairs from a model if the key is in the
-provided ``keys`` list.
+Removes model :ref:`metadata <metadata>` key-value pairs from a model if the
+key is in the provided ``keys`` list.
 
 .. list-table::
     :header-rows: 1
@@ -786,8 +786,8 @@ provided ``keys`` list.
 includeMetadata
 ---------------
 
-Removes :ref:`metadata` key-value pairs from a model if the key is not in
-the provided ``keys`` list.
+Removes model :ref:`metadata <metadata>` key-value pairs from a model if the
+key is not in the provided ``keys`` list.
 
 .. list-table::
     :header-rows: 1

docs/source/1.0/guides/style-guide.rst

@@ -14,7 +14,7 @@ style guide makes models easier to read.
 Model files
 ===========
 
-Smithy models SHOULD be authored using the :ref:`Smithy IDL <lexical-structure>`.
+Smithy models SHOULD be authored using the :ref:`Smithy IDL <idl>`.
 Smithy models SHOULD resemble the following example:
 
 .. code-block:: smithy

docs/source/1.0/spec/aws/aws-ec2-query-protocol.rst

@@ -29,7 +29,7 @@ Value type
 
 .. important::
 
-    This protocol does not support :ref:`inline document types <document-type>`.
+    This protocol does not support document types.
 
 .. tabs::
 

docs/source/1.0/spec/aws/aws-query-protocol.rst

@@ -58,6 +58,6 @@ See
 
 .. important::
 
-    This protocol does not support :ref:`inline document types <document-type>`.
+    This protocol does not support document types.
 
 *TODO: Add specifications, protocol examples, etc.*

docs/source/1.0/spec/aws/aws-restxml-protocol.rst

@@ -194,7 +194,7 @@ that affect serialization:
 
 .. important::
 
-    This protocol does not support :ref:`inline document types <document-type>`.
+    This protocol does not support document types.
 
 
 ------------

docs/source/1.0/spec/core/http-traits.rst

@@ -670,7 +670,7 @@ Serialization rules:
 #. When a string or blob member is referenced, the raw value is serialized
    as the body of the message.
 #. When a :ref:`structure <structure>`, :ref:`union <union>`, or
-   :ref:`document <document-type>` is targeted, the shape value is serialized
+   document type is targeted, the shape value is serialized
    as a :ref:`protocol-specific <protocolDefinition-trait>` document that is
    sent as the body of the message.