Rework documentation

[FFY00]

The documentation is not super accessible to new users. I would like to rewrite it so that it is more accessible to new users, while still presenting the required information for advanced users.

[FFY00]

@CAM-Gerlach, if you have time, would you like to have a look at our documentation and provide feedback?

[CAM-Gerlach]

Sure, thanks for asking. I'll keep my initial feedback and suggestions mostly fairly high level, focusing on your goals above (making the docs more accessible to beginners, while enhancing the key information for more advanced users) and the docs you have so far (maybe there are more than this, somewhere else besides the example?). For some context, this comes from the perspective of both a fairly experienced documentarian familiar with the Diataxis principles (that the NumPy, SciPy etc. docs also follow), but also a new user having only minimal familiarity with Meson and MesonPy.

Initial impressions and feedback

Below, I present what I noticed and the questions I had as a first time user of the site. I've embedded it in a details block in case you want to skip right to the suggestions/recommendations.

Initial impressions and feedback

• Overall most of the basic pieces of information are there, once I dig around for them, but the structure and organization doesn't feel that intuitive as someone not already familiar with it, which both marks it harder to find information I'm looking for, and makes it feel like I don't have a complete, coherent picture of what Meson is and how to use it. Re-organizing it to better serve readers' needs, using the Diataxis framework as a guid, should help here.
• The main page contains a brief "How to use section" (good), which served as a jumping off point to the Getting Started, which I figured would be a good beginner entry point. However, it then dives right into details of how Mesonpy works with sdists and wheels, which seem like it would be much better suited for a separate page. The (fairly significant) existing limitations are then only listed at the bottom (which is good mainpage content, but obscured by the bulk of the technical details in the "How does it work" section.
• The "Getting Started" page is missing some of the content users might expect, as whether/how to install Meson/Mesonpy (of course, users generally don't need to due to PEP 517, but you should tell them this as many are unlikely to be familiar with how that works), and how to build their project (build). There's a detailed example demonstrating PEP 621, but nothing that actually tells, or at least links users to what a pyproject.toml file is or where to put it (which likewise a lot of your target audience isn't going to be automatically familiar with). Finally, even though the example includes one, I don't see anything about whether I need a Meson.build file or whether/how to set up Meson itself (which again, much of your audience isn't going to know coming from Setuptools and numpy.distutils).
• The content on the Build Options page was good, if brief reference documentation—focused, clear and to the point. I was initially a little confused by how the options get from build to Mesonpy, despite being familiar with the tools and the relevant standards, until I realized that you were passing a key-value pair through the build -C option rather than passing the build (confusingly-named) value to a Cbuilddir build option. Spacing and/or using the long --config-option option name would really help here.
• "Specific behaviors" was, at least to me, a confusing and unclear title that actually was something between a list of environment variables and a how-to guide—which should really be one or the other.
• The dates in the changelog were a little confusing, since they look like standard ISO 8601 dates but are actually "backwards"

Recommended Structure/Content

These are my high-level suggestions, taking into account both the above and your overall goals.Aside from adding a step by step tutorial, the primary focus is on structuring and plugging the key gaps in what you already have and laying the groundwork for future content you may want to write, as opposed to adding a bunch of new material right away.

Consider organizing the content with the basic Diataxis content types that serve different reader needs in mind: Tutorial, How-to, Reference and Explanation. This helps the reader jump to the type of content they are looking for (a beginner quickstart? solutions to specific problems? the command/API reference? Or in-depth behind the scenes details?), and you see what needs you serve well and what you're missing.

• Keep the main page as a high-level overview, with a few brief sections directing readers to where they can go next. In particular, I suggest the following brief sections, focused giving readers a brief taste and directing them to where they can learn more:
  • A short intro describing what MesonPy is (more or less what you have, but made much more accessible to non-packaging experts without jargon and unfamiliar terminology)
  • A brief "Why MesonPy" section) describing a little more completely what MesonPy's purpose and role is as a build backend (and what that is), how it fits into the ecosystem, and who might want to use it and why. Right now, its not clear who its for (those using C-extensions, scientific projects, etc), why they should use it over other options, or even what it replaces. If you want to go into more detail, linking an Explanation on the topic might be a great idea.
  • A "Getting Started with MesonPy" section based on your existing "How to use it" giving a very minimal overview of how to get started (linking rather than describing any details), leading into a prominent link to take readers to the tutorial to dive in.
  • A "Using MesonPy" section, where you could very briefly introduce and link the How-Tos (or at least the most common)
  • A "Configuring MesonPy" section, introducing and linking to the environment variables, CLI options and configuration pages
  • The limitations, as you have now, but with very brief summaries linking to the relevant How-tos to work around them, as available
• Under the Tutorial category, start with a beginner tutorial for now; you can add more advanced tutorials later as your time and needs allow:
  • Step by step, walk users through exactly what they need to do to go from a basic Python project to a working build with MesonPy. Assume readers have at least a basic understanding of Python and C, but may not be familiar with the modern Python packaging workflow (or even much Python packaging knowledge at all), and it should make sure to clarify what users don't need to do as well (e.g. manually install mesonpy/meson, run it directly, or tell users to do so, etc).
  • You should link to, and .. include as desired, relevant parts of the example project (that is very helpful, but currently aren't visible at all unless users browse the repo source code)
• The How-To category is where users will go to find solutions to common situations, or "how to" do different things with Mesonpy. You address several of these through the docs, but they are buried under other sections with names that do not at all make it clear what problem they are solving. Good existing candidates include:
  • How to set the target macOS version (from Specific Behaviors -> MACOSX_DEPLOYMENT_TARGET)
  • How to control what source files get included (basically the entire content of the sdist section)
  • How to include external shared libraries in binary distributions (wheels) (second half of the Binary distribution section)
  • How to include data files in the wheel (Limitations -> No data)
  • This would also be a great place to have dedicated porting guides, e.g. from distutils, numpy.distutils and/.or setuptools, describing the Meson equivalents
  • Describing how users can do editable installs (if they can) would be good here too
  • This can be expanded with more how-tos, either as you think of them or as users ask them), and split into separate pages later
• The Reference category is where you concisely but completely describe MesonPy's CLI, API, configuration and similar. Right now, it looks like you don't have much to put there, but it will grow with time. You could, for now, simply put all of this in top-level sections of one Configuration document. Specifically:
  • CLI Build Options page: As you have it now, just with a clarification that these are options passed from build to Mesonpy, and using the clearer --config-setting builddir=custom_build_directory rather than -Cbuilddir=build
  • Environment Variables: State their purpose, syntax and semantics, linking elsewhere for how-to or other ancillary information (just MACOS_DEPLOYMENT_TARGET for now, which can link to the relevant how-to for details like other ways to specify this)
  • Configuration page: Here is where you'd describe/link to the details on PEP 621, specifying version as dynamic, automatic metadata and any [tool.mesonpy] config settings.
• The Explanation category is where you peer behind to curtain to explain how and why it does what it does, and give readers background on the choices they might have to make using it
  • There would be a good place to describe the more general/internal, non-task-oriented workings of how Meson builds a sdist and a wheel, as you describe on the front page
  • Now or later, you might want to talk in more detail how Meson compares to other build backends, e.g. distutils, numpy.distutils, Setuptools and Scikit-Build
  • Likewise, you could describe how Meson fits in with the overall ecosystem of frontends, backends, indicies and integration tools, etc.
  • You could also talk about things like your development philosophy here

Other suggestions:

• You might want to add a short FAQ with brief answers to common questions that aren't substantial or task-oriented enough for How-Tos ("Do I need to install Meson?" "Do I need a setup.py?" "How will other users install my package", "Does Mesonpy replace distutils/numpy.distutils/setuptools/pip etc)
• Add packaging.python.org as an intersphinx target, so you can link terms with :term:, specs and guides with :ref:, etc.
• Use standard ISO 8601 dates in the changelog, that match reader expectations and avoid ambiguity

I'm happy to give further more focused feedback and/or suggestions if there's something more specific you'd like my input on.

[rgommers]

This is great, thanks so much for the extensive feedback @CAM-Gerlach!

I agree with the big picture of this - organizing everything according to the Diataxis framework. A couple of other scattered thoughts:

• We should state what we assume the reader already knows about Meson and pyproject.toml hooks, maybe give a one paragraph intro, and make some recommendations about what in the Meson docs to read (they're excellent).
• There's a set of docs somewhere (in a place I can't find right now; not the Python Packaging Guide I think) about all the various PEP 517 backends, side by side in a tabbed interface. We should add meson-python to that and link to it.
• We should also add an entry to https://packaging.python.org/en/latest/key_projects
• We should list some adopters. The list is growing quickly; SciPy, scikit-image merged support recently, Pandas PR is ready, NumPy implementation is moving finally, quite a few smalller domain-specific projects.

[CAM-Gerlach]

This is great, thanks so much for the extensive feedback @CAM-Gerlach!

I'm so glad it is helpful to you! I was worried it was way too much and would be overwhelming.

• We should state what we assume the reader already knows about Meson and pyproject.toml hooks, maybe give a one paragraph intro, and make some recommendations about what in the Meson docs to read (they're excellent).

+1

• There's a set of docs somewhere (in a place I can't find right now; not the Python Packaging Guide I think) about all the various PEP 517 backends, side by side in a tabbed interface. We should add meson-python to that and link to it.

I'm guessing you might be referring to the Python Packaging User Guide tutorial (despite being the de-facto flagship tutorial of the PyPUG, its not nearly as visible/prominent as it could be on the homepage/ToC).

I was going to recommend linking there, though I'm not totally sure about how it might be best to integrate backends like MesonPy (and Scikit-Build, once Henry stabilizes the PEP 517 support—funny enough, it was he that recently revised it to tabbed-by-backend organization you mention), since the main focus of the page is on beginners with pure-Python code, whereas introducing C extensions and separate build systems adds a fair amount of additional considerations and complexity.

That would be something the various folks involved (you, Henry, the PyPUG maintainers, and the other packaging folks) would want to discuss and think about, once MesonPy and its docs are ready to be included—a Discourse thread on the packaging category might be best.

• We should also add an entry to https://packaging.python.org/en/latest/key_projects

+1

We should list some adopters. The list is growing quickly; SciPy, scikit-image merged support recently, Pandas PR is ready, NumPy implementation is moving finally, quite a few smalller domain-specific projects.

That would be really helpful and I neglected to mention it above; I went looking for some early adopters to get a sense of an example usage, but didn't see a list in the docs.

[eli-schwartz]

If you have any comments on https://mesonbuild.com/ as well, I'd be happy to help incorporate feedback. I'm not a documentation expert by any means, although I do believe Meson's API reference docs are excellent at a minimum.

[CAM-Gerlach]

@eli-schwartz My first high-level impression, as both a first-time visitor to the site with minimal Meson experience and as a documentarian, is that the docs look quite complete and comprehensive, with all the major pieces there (tutorials, how-tos, user reference, API reference, some amount of explanations, and a healthy helping of ancillary, meta- and developer-focused docs. However, at least to me, the current organization and structure of the docs feels a little overwhelming, not so well-organized, hard to navigate easily and logically and potentially confusing.

Specific first impressions

For example, as a new user starting out, there's a "Getting Meson" guide, an "Absolute Beginners Guide to Installing and Using", a "Quickstart Guide", a Tutorial, what appears to be another tutorial with a very specific name (building SDL2 app), all at the top level—and under Additional Documentation, there's another In-Depth Tutorial. Where should I start, and where should I go from there? Most of these give some kind of introduction, but I don't see a lot connections between these documents (now that you've done XXX, you can move on to YYY and ZZZ in order to...). It could use a coherent organizational structure, and a high-level page/section listing these learning resources in one place, and at least briefly introducing each and providing a suggested sequence.

Likewise, I went looking for the "API reference docs" you referred to above, and was a bit confounded by the fact that there's a Manual, a Reference Manual, Reference Tables, and a Quick Reference menu, all at the top level. Peering inside the "Manual", I was hit with a massive flat list of topics, from which I quickly retreated. After a bit of digging and looking at things, I figured out you must be referring to the sections in the Reference Manual. Indeed, seems clear enough—within that section, it's well organized, and suitably formatted and structured for API reference docs. I was a bit confused at first when the topics I was seeing at the top of the top-level Reference Manual page didn't match those in the sidebar, though I did figure out it was just in a different order, for some reason. Searching for methods, even by fully qualified name, was also not so friendly or easy to navigate, and didn't always work reliably, though I assume that has more to do with your docs system (I haven't heard of hotdoc before) than anything.

Aside from that, if I were looking for how to do something specific with Meson (i..e. a "How to" guide), there's the How do I do X in Meson with a long flat list of tasks, but there's also a number of misclanious how-to-type documents floating around in the root, others under "Additional documentation" as well as some "How to" type material in the FAQ. Likewise, I'd expect to find most of the developer documentation and meta-documents under "Contributing to Meson", but some of that is under Additional Documentation too. And overall, the ordering did not seem the most logical.

I could provide more detailed recommendations for a more helpful, logical and organized structure here or in a more appropriate venue, but overall, I'd suggest:

• Greatly reduce the number of top-level items to just those that serve common and distinct user needs, and in a logical order, You don't have to strictly follow Diataxis, but it provides a good starting point—Tutorials [for learning], How-to guides [for tasks], Reference [for information], and Explanation [for background material], plus developer/contributing/meta docs, a FAQ and maybe a one installing/quickstart guide.
• Likewise, avoid having top-level sections that have a very lengthy flat list of documents directly underneath them, especially without a clear and unambiguous order to them. Instead, move these into logical subsections as much as practical.
• For each top-level section, provide a landing page that gives a brief overview of the major pages/subsections within, in a manner appropriate to that user need—for example, for the tutorials, you might want to clarify which audience each tutorial/etc. is aimed at, and a suggested progression between each.
• Within prominient first and second-level docs, especially things like tutorials, link related resources at the top, and next steps at the bottom, to help the reader navigate the docs and feel a sense of coherence and direction.
• Avoid standalone or overly specific docs floating around in the root; they just clutter things up and could typically go well under a more appropriate subsection.

Again, all the moving pieces look to more or less be there—compared to all the work that's clearly gone into them so far, a very modest amount of thoughtful organization should help go a long way toward making all the detailed content you have friendlier to new users, quicker to navigate for experienced users, and more accessible, discoverable and useful to everyone in between.

[rgommers]

Thanks again for the detailed analysis @CAM-Gerlach!

However, at least to me, the current organization and structure of the docs feels a little overwhelming, not so well-organized, hard to navigate easily and logically and potentially confusing.

I kinda agree with that. I really like the Meson docs, pages are well-written and the content is quite complete. However, I've spent a lot of time in those docs, and navigating them is still confusing on occasion. Multiple sidebar and foldouts, content not always logically organized, and the way the search bar works is odd. For the last point, maybe migrating to a new doc generator would help - also makes it easier to build the docs, I had trouble with hotdoc - but not sure how feasible/welcome that is.

[CAM-Gerlach]

For the last point, maybe migrating to a new doc generator would help - also makes it easier to build the docs, I had trouble with hotdoc - but not sure how feasible/welcome that is.

FWIW, hotdoc seems to have relatively little use beyond Meson and doesn't look all that actively developed. And not that it means that much, but mainstream Python doc generators like Sphinx and mkdocs have 100-300x the stars (even MyST has 10x). If you want to stick to the same Markdown syntax and keep things pretty dead simple, with a similar UX, themes and overall "style", mkdocs is a very popular option these days. MyST will give you all the power of Sphinx while still remaining fully compatible with your existing Markdown syntax, but that might be overkill for your specific needs and not be as close a fit for the overall workflow you have now.