Maintenance Mode & Maintenance Windows

[ErisDS]

The latest migration (Forcing UTC) can take a while. This along with some other issues we've seen whilst migration 0.8 suggests we need to think about our tooling around upgrades & migrations. The first tool we've started adding already is Version Mismatch handling.

Another improvement we can add is support for putting Ghost into maintenance mode. There are two parts to this:

Maintenance window messaging support

Providing a way to tell Ghost that there will be a maintenance window at some time in the future. Maintenance windows take the form of a from and to datetime and an optional URL for more information.

When there is a maintenance window set that is in the future, Ghost should show a dismissible alert which displays the dates and a link to the URL for more info. I think we have an orange alert for warnings?

When the maintenance window is active (i.e. we are currently between the two dates) Ghost should show a dismissible alert with a slightly different message, saying that maintenance is in progress.

The idea of the maintenance windows is to provide fair warning that maintenance is going to happen, it does not do anything beyond provide some messaging.

Maintenance mode

Where the maintenance window messaging feature is only intended to provide warning to users, maintenance mode is intended to make changes to the behaviour of Ghost. The idea being, you may schedule a maintenance window for an hour, but you may only need to put a blog into maintenance mode for 5 minutes.

The first change that maintenance mode should make, is to disable all write requests to the API. Read (GET) requests should succeed, but all requests which modify data should result in a 503 Maintenance Error.

This means that the blog itself will be unaffected, but anyone trying to modify content will get an error for a period of time. This prevents data being written whilst the blog is in an unknown state. This should be enough to protect Ghost blogs from harm during maintenance.

We could also add support for other forms of maintenance, for example serving an entirely different page for Ghost admin to disable the admin completely. Additionally we could add support for serving a maintenance.hbs from themes to also show a maintenance message to blog readers.

I think each type of maintenance mode would have different use cases. I think the API mode is most important, but I think it should be possible to set each mode individually.

Implementation

We have a identified that there are a collection of values we'd like to set.

E.g maintenance.window = false

Or maintenance.window = {start: '2016-08-08 02:00', end: '2016-08-08 04:00', url: 'status.mysite.com'}

And also something like maintenance.api = true

The first idea that comes to mind is to support these as configuration values in config.js.
The downside with using config is that Ghost would need a restart to pick up the details.

The alternative is some record in the database - perhaps the settings table. However, I don't really think a UI makes sense for this as it needs to be set by the server administrator, not the blog administrator and the settings table has an in-memory cache so manually editing the table wouldn't work.

Another option is environment variables. This seems like a lot of data for environment variables though.

Something else might be a watch on a file called maintenance.json which contains the info. This would load the extra config if the file became available. This seems quite resource intensive for a rarely-to-be used feature.

TLDR; It's easy (ish) to implement the actual maintenance mode, and to have alerts in the admin UI, but I am not 100% on what the best way to set these values would be.

I'm leaning towards having values in config.js as a first very quick version?

[kirrg001]

We had a discussion about maintenance mode/migrations (#6574) in slack.

We have agreed on:
If a rollback happens (in case we run 0.9, but the database version remains in 004 or 005), the blog will shutdown. (this is similar to the current behaviour). The reason is that we cannot run a ghost version code base with a different database version.

We will always serve 503/maintenance page (also for GET requests), because it can cause trouble when the code base already changed, but the migrations didn't finish.

We will run the migrations async within the node process. (instead of having a runner in a separate process). The reason for this is that it brings the same result then having a migration runner.
When we run the migrations async, we can easy enable/disable the maintenance mode, because we are inside of the blog process.

[kevinansfield]

Regarding the maintenance window alerts in the admin client, should we show these on every login or once dismissed should we remember that and not display it again (quick solution might be to remember only locally so it won't be displayed again on the same machine)?

[kirrg001]

The maintenance window is not implemented yet, because i wanted a merge of the maintenance mode and the priority was the lowest.

I think we can easy realise it with statuspage.io (https://doers.statuspage.io/api/basics/). So admin calls an endpoint to the server (like GET /status) and server sends an object back containing the status information. Which is in this case something like:

{ 
  maintenance: {
    message: "....",
    url: "...."
  }
}

@kevinansfield
quick solution might be to remember only locally so it won't be displayed again on the same machine - agree. i would show it all the time until he clicks the message as read.

[ErisDS]

Gonna scrap the maintenance window feature from this for now, in lieu of coming up with a better solution (status page integration?!) later.