RFC 100: Enhancing headless support in Wagtail core

[thibaudcolas]

View as an HTML document. This RFC attempts to set a direction for Wagtail’s future headless support improvement. It covers:

1. A definition of Wagtail’s support goals when it comes to headless support. Long-term, that’ll help us decide what headless-related capabilities should be in Wagtail itself, vs. in packages, vs. for site implementers to build. This is the most crucial thing to validate as part of the RFC process.
2. An overview of the current areas of interest and known opportunities for improvement. This is meant more as a recap of current points of friction, gaps, opportunities, than a mandate that we must address them all.

I’ve also published a 2024 Wagtail headless survey so we get input from a bigger group of developers. Some of you might recall the 2022 survey, which helped us tremendously back then in documenting the current state. Those survey results will help us understand where it’d be most helpful to direct headless support contributions.

[ahosgood]

I had to add extra code to get redirects to work in the API - would that be helpful to have included?

[thibaudcolas]

@ahosgood I would assume so? Redirects is a contrib module so not as "core" as some of the more fundamental aspects of the CMS, but certainly something that many sites would consider core CMS functionality.

[lb-]

I think it would be good to somehow include the pattern for JSON rendering using normal Page routing. Essentially each Page path can have a different request header and then return JSON instead of HTML.

It's quite an intuitive approach and aligns with how other CMSs (e.g. Adobe Experience Manager) can provide their APIs. It's not going to make sense for every installation but worth reviewing as part of this RFC.

In the abstract, this also could align with applications that may want to build out HTMX style applications, still returning HTML but providing a partial render of 'inner' HTML based on request headers.

See wagtail/wagtail#11752 & wagtail/wagtail#8374

[saevarom]

Headless was on the agenda for Wagtail Space NL but this only involved incorporating the areweheadlessyet.org website into the Wagtail documentation. The resulting PR is here: wagtail/wagtail#12039
The sprint topic discussion is here: https://paper.dropbox.com/doc/Sprint-topics-Wagtail-Space-NL-2024--CUVkcdc_H1K~k2BWz8SS9UUbAg-3rkJ5imATtyajKY2Ey1Cr#:uid=172696553243645599787735&h2=Headless-Wagtail

[lb-]

Another long-standing issue with the current API is that we cannot easily generate an OpenAPI specification

See wagtail/wagtail#6209

I would say that we must have a documented or even out of the box way to generate specifications for our APIs if we want to truly say we have headless support.

[thibaudcolas]

Thank you everyone for the feedback! I’ve heavily updated the RFC. Now’s the time for further reviews and feedback (approval?) The RFC now has two well-separated sections:

1. A definition of Wagtail’s support goals when it comes to headless support. Long-term, that’ll help us decide what headless-related capabilities should be in Wagtail itself, vs. in packages, vs. for site implementers to build. This is the most crucial thing to validate as part of the RFC process.
2. An overview of the current areas of interest and known opportunities for improvement. This is meant more as a recap of current points of friction, gaps, opportunities, than a mandate that we must address them all.

I’ve also published a 2024 Wagtail headless survey so we get input from a bigger group of developers. Some of you might recall the 2022 survey, which helped us tremendously back then in documenting the current state. Those survey results will help us understand where it’d be most helpful to direct headless support contributions.

If you want to help

1. Review the RFC. You’re here already so you’re aware of how this all works and how important input is.
2. Take the survey and share it with colleagues. Should be a quick one.
3. Upvote headless issues?

We’re not big on issue emoji reactions / votes as a way to make decision, but there’s a big corpus of existing tickets here so could be interesting. Here are all open issues sorted by theme:

Headless-related existing issues

• REST API
  • Allow unlimited API results with limit=-1 #12604
  • Wagtail API: Greater than/less than filters #7497
  • Wagtail API: OR filtering related models #5755
  • Add ability to include and filter via detail fields on pages' list endpoints #11404
  • Random order in the API leading to an AttributeError #3512
  • Wagtail API should have a browsable root URL #1921
  • Duplicate db query on api detail page #6133
  • Adding per-subclass filter backend/class to the v2 API Pages Endpoint #3403
  • Future release enhancement: Implement APIField for listing_default_fields, too #3562
  • Document nested_default_fields in APIv2 #11587
  • Add a UUIDField to the Page model #6162
  • Support customizing/encoding IDs in the API #6917
• API schemas
  • Support OpenAPI Schema generation for Wagtail API #6209.
• Images
  • ImagesAPIViewSet query params for rendition? #7973
  • No Image URL in the API for ImageChooserBlock #2087
  • Feature request: Include image's title in ImageRenditionField #5435
  • Documentation for using the "Dynamic image serve" view for generating image renditions on a headless site #6113
• Page URL Routing
  • Keep queries other than html_path in /api/v2/pages/find/ #6577
  • Finding pages by HTML Path - redirect missing port #7595
  • Improvements to API router reverse lookups. #11310
  • Add ability to support custom renderers for Page views (e.g. JSON API response at page URLs) #11752
  • Accept page change in before_serve_page handling #5753
• Redirects
  • Refactor redirect middleware to use its lookup logic for redirects API #11524.
• Rich Text
  • API should provide access to the HTML-converted versions of rich text fields #2695
• Multi-site support
• Form submissions
  • Ability to accept Form Builder Page as a POST request to the API (headless) #6950
• Password-protected Pages
• Internationalisation
  • Support internationalized routes in API pages/find/?html_path view #6679.
• StreamField
  • Override serializing StreamBlocks to JSON api #3454
  • Apply custom-per-block-type StreamField serializers #5174
  • parent_context or just "parent Page" for Block.get_api_representation() #7976

---

Last thing, re long-standing issues and schema specifications: At this stage I think we need to move past the No True Scotsman chain of thought, asking ourselves whether Wagtail is truly headless or not. Wagtail is a hybrid system, it’s pretty clear. Some features aren’t headless-compatible, but clearly there’s hundreds if not thousands of headless sites out there built with Wagtail, some pretty high-profile. So yes it’s truly headless. We don’t want to mislead people, so there are gaps to fill (docs in particular), but it’s already a thing and has been for years.

[itzomen]

A frontend client like wagtail-js will make integrating wagtail for many frontend developers easier.

[thibaudcolas]

@itzomen I’m not sure I follow your point, as you shared it, that front-end client already exists? Is there a need for more than one JS client?

[itzomen]

@itzomen I’m not sure I follow your point, as you shared it, that front-end client already exists? Is there a need for more than one JS client?

Oh no, I meant the continuous development of a JS client (could be that one or something else) will be a great addition to the efforts to promote the usage of Wagtail as a headless cms.

And this is definitely something I will like to provide support with

[thibaudcolas]

Ah yes! that makes a lot of sense :) I just looked at the analytics on our now-retired "Are we headless yet?" website, the most-viewed page was REST API support, so a JS client working with the REST API is a big win.

[thibaudcolas]

I have updated the RFC based on the Results of the 2024 Wagtail headless survey. Thank you to everyone who took part in that – I think it was exactly the type of input I was hoping we would get as far as setting priorities.

@laymonage @ahosgood @allcaps @lb- @dopry @zerolab would you be ok to re-review the RFC and approve or otherwise provide further feedback? It’s a pretty high-level RFC as it is, more of a statement of intentions than an implementation plan. So to be a bit more specific, in #106 we have three follow-up items shaping up that will depend on feedback here:

• A plan to bring headless preview and userbar support in core
• "Quick win" API and documentation improvements based on priority order expressed here
• An official headless demo site – primarily so our contributors and maintainers can more easily work on headless improvements

[Scotchester]

• A plan to bring headless preview and userbar support in core
• "Quick win" API and documentation improvements based on priority order expressed here
• An official headless demo site – primarily so our contributors and maintainers can more easily work on headless improvements

These sounds like three really great next steps to me!

[dopry]

Looks good. Personally I'd prefer to see GraphQL move towards first class treatment. I don't use REST for any headless sites anymore. The network overhead of multiple requests to assemble complex views adds a lot of latency that negatively impacts the UX. With GraphQL I get excellent tooling for code generation, specs are built in, and there is GraphIQL for exploring the model. Even if it remains an External Package it would be nice to see that it is something that is getting stronger consideration from the core team. @zerolab does a great job of supporting it. It could use quite a bit of documentation love and it could be a lot easier to work with when it comes to custom blocks, search, multisite handling. In my own project I've had to modify a lot to get it to be fully functional in my use case.. While I contribute a lot of that back, there are still pain points. It would be nice to annotate the ideal state as Improved External Package.

[auvipy]

Looks good. Personally I'd prefer to see GraphQL move towards first class treatment. I don't use REST for any headless sites anymore. The network overhead of multiple requests to assemble complex views adds a lot of latency that negatively impacts the UX. With GraphQL I get excellent tooling for code generation, specs are built in, and there is GraphIQL for exploring the model. Even if it remains an External Package it would be nice to see that it is something that is getting stronger consideration from the core team. @zerolab does a great job of supporting it. It could use quite a bit of documentation love and it could be a lot easier to work with when it comes to custom blocks, search, multisite handling. In my own project I've had to modify a lot to get it to be fully functional in my use case.. While I contribute a lot of that back, there are still pain points. It would be nice to annotate the ideal state as Improved External Package.

I don't think graphQL would be the best API for headless wagtail, but that could be an option. I would suggest following JSON:API 1.0/1.1 standard for headless REST API. Here is a old related issues #22

[dopry]

@auvipy

I don't think graphQL would be the best API for headless wagtail, but that could be an option. I would suggest following JSON:API 1.0/1.1 standard for headless REST API. Here is a old related issues #22

I disagree with your assertion regarding what would be the best API, but I also wasn't suggesting abandoning REST. It still has a userbase whose needs should be met, and it is low overhead for the core team to maintain. I just prefer GraphQL, as do a lot of other frontend developers, and would prefer to see it get more consideration than it currently does.

[auvipy]

You can do graph like query with json:api

[laymonage]

The reasoning, compromises, and the plan seems solid to me. I think having this RFC in is a good first step before we start any work on the actual improvements in Wagtail. Thanks for the great write-up!

[laymonage]

If we're considering moving away from DRF for the next version of the API, I'd suggest making whatever package we use be optional. Right now DRF is a mandatory requirement for Wagtail installation, even if you don't use it at all. Granted, there are internal admin views that are built with DRF, but I'm sure it can be refactored to use plain Django just fine.

And then if people want to use the API, they can install Wagtail with an optional dependency group, e.g. wagtail[api].

[thibaudcolas]

Sounds great to me!

[thibaudcolas]

@dopry @auvipy thanks both! I didn’t think GraphQL would be a good candidate for core simply because it doesn’t seem that would have big advantages over the status quo. I’ve also heard good things about other implementations than Graphene. But certainly we need to invest more into it, package or no.

For the first time ever, I have an open source implementation of GraphQL I can point people to: bakerydemo-nextjs. I hope this will make it easier for me to make progress on docs at least.

[dopry]

@dopry @auvipy thanks both! I didn’t think GraphQL would be a good candidate for core simply because it doesn’t seem that would have big advantages over the status quo. I’ve also heard good things about other implementations than Graphene. But certainly we need to invest more into it, package or no.

For the first time ever, I have an open source implementation of GraphQL I can point people to: bakerydemo-nextjs. I hope this will make it easier for me to make progress on docs at least.

I've never really had a problem with Graphene, I've had problems with graphene helpers, but ultimately it hasn't been a problem working with it. You need to do some optimization with your resolvers and prefetch/select_related when it's called for.

[lb-]

@thibaudcolas Thank you for the incredible work here, taking on feedback and cross-linking various issues.

I am more than happy to approve in it's current state, I have added a few small comments though if they can get looked at.

[lb-]

Would we consider providing a mechanism for sending the password via a HTTP header instead? Especially as this 'password' is actively moving towards more of a shared secret. I know this is not the same as an API key but a namespaced header could be a suitable approach.

[dopry]

That wouldn't work too well for a Static Site Generation scenario where the frontend may render an .htaccess or implement it's own authentication methodology. It would probably be better to transmit the hash and hashing details to the headless front end.

[lb-]

I would like to recommend that as part of our documentation tasks, we also aim to expose the full API for our Wagtail Guide https://guide.wagtail.org/en-latest/

This is a great way to show the features of the API (as is, and evolving) and may also allow developers to create new content/remixes of that Guide site.