Introduce publish blocks and data sinks

[traut]

Background

The content generation pipeline needs to deliver the produced content somewhere where it can be stored, edited, and collaborated on. Currently, Fabric outputs the content into a stdout or a file, delegating the further delivery to the user. This is simplistic, and while often it is enough, in the production deployments, it is cumbersome and requires additional scripting.

Since Fabric plugins include data sources (input data integrations), the output integrations also fit into the plugins

Design

We are introducing new block type -- publish blocks -- that define outgoing integrations. These block do not produce any content and are executed after the document is fully rendered.

Another name option is "deliver", but "publish" seems more appropriate.

publish blocks

The blocks behave very similar to data blocks:

• publish blocks can be defined in and outside the document and be named or anonymous.
• they can be referenced with attribute override
• they must be defined on the root level if defined inside the document

document "greeting" {

  publish google_docs {
    title = "greeting-doc"
  }

  publish local_file {
    path = "/tmp/greeting.md"
  }

  title = "The Greeting"

  content text {
    text = "Hello, Fabric!"
  }
}

Data sinks

publish blocks are executed by data sinks. Like data sources and content providers, plugins implement data sinks.

• the data sinks support config blocks and might require configuration attributes (for example, a data sink for Google Doc requires auth details)
• the data sinks might require execution attributes. For example, the stdout data sink does not take any attributes, but the google_docs sink might require a filename template.

The context access

The data sinks must receive the context during the execution, similar to the content providers.

The sink might access the data defined in data blocks (.data....), the template metadata (.document.meta...), or the rendered content, to prepare the metadata that the data storage needs. By allowing the data sinks to accept the execution arguments and access the context, we enable the plugins to keep all the complexity of the delivery away from the templates / Fabric source code.

The context that the sink receives has additional format-specific fields:

• .format -- format set in the doc_format attribute
• .format_ext -- an extension for a selected format

Generic attributes

• doc_format -- an enum that can contain markdown (default), html (as implemented by Add support for html output format #117), pdf (tbd), etc
  • the data sinks provide the list of supported doc formats. If the format specified in the argument is not supported by the data sink, the error is raised.

config publish dropbox {
  username = "john"
  password = "spring2024!"
}

document "greeting" {

  publish dropbox {
    filename = "greeting-doc.{{ .format_ext }}"
    doc_format = "pdf"
  }

  title = "The Greeting"

  content text {
    text = "Hello, Fabric!"
  }
}

Built-in data sinks

• stdout -- the rendered document is printed to stdout. This data sink is used by default if no publish blocks are defined in the document.
  • the sink supports only markdown and html formats
  • no configuration attributes
  • no sink-specific execution attributes
• local_file -- the rendered document is saved into a local file
  • the sink supports markdown and html formats (not a catch-all but an ever-expanding list)
  • no configuration attributes
  • execution attributes:
    • path -- (required) string attribute. It contains a path to a local file. The sink treats the value as a Go template string, similar to the text value in the content.text blocks.
    • permissions -- (optional) permissions to be set on the file

CLI overrides

It should be possible to disable or override publish blocks during execution. We can achieve this with CLI arguments:

• --no-publish -- (optional) bool flag. When set, the document is rendered but not printed out anywhere. It can be useful for testing the rendering without producing content. // (?) I'm not 100% sure this is needed
• --publish -- (optional) string argument, accepts stdout or local_file values.
  • if local_file value is set, --local-file-path is required (instead of --out-file from Streamline Fabric CLI interface  #70)
• --publish-format -- (optional) string argument, accepts markdown and html

Conditions:

• when --publish is set, it overrides all publish blocks in the target document.
• --publish-format can be provided only when --publish is set
• --publish-format defines the format for the publishing method set in --publish

Examples

For example,

config publish dropbox {
  username = "john"
  password = "spring2024!"
}

publish local_file "local_version" {
    path = "/tmp/{{ .document.meta.title }}-{{ now.Date() }}.{{.format_ext}}"
}

document "greeting" {

  publish dropbox {
    filename = "greeting-doc.{{ .format_ext }}"
    doc_format = "pdf"
  }

  publish ref {
    base = publish.local_file.local_version
  }

  title = "The Greeting"

  content text {
    text = "Hello, Fabric!"
  }
}