feat(developer): manage shared fonts

[mcdurdin]

Fonts are currently difficult to manage in Keyman Developer projects, particularly because we want to share fonts between keyboards. The developer must either clone the entire keyboards repository, or else jump through convoluted hoops in order to properly use shared fonts.

Proposal, in short: allow packages to reference files from GitHub repos, not just local files. Then the compiler will download (and maybe cache) files. The keyboard author could reference, e.g. https://github.com/keymanapp/keyboards/blob/master/release/shared/fonts/aegyptus/Aegyptus.otf. (Query, should we use sha instead of branch names?)

Ideally, we can leverage an existing WSTech repository for this. Perhaps LFF. In theory, we could allow any URL in the .kps file. But we would probably want to have a whitelist for perf, stability, and security reasons for the keyboard repository.

This also resolves the single-keyboard-repo conundrum of how to share fonts.

It would be important to have a UI that helps keyboard devs to find the fonts as well.

Things to do

• feat(developer): kmc-package support remote fonts and files 🐻 #12667
• Update package schema min version for Source field support
• Caching of resources retrieved online (offline support?)
• A lot of work will need to be done with the Package Editor in TIKE to support this feature.
• Author documentation (once Package Editor has been updated)
• Add a keyboards repo action which does a verify against all names and reports on changes
• After reporting on changes, build a script to automatically update all keyboards that refer to a given font and open a PR

[mcdurdin]

@DavidLRowe @LornaSIL @srl295 your thoughts appreciated. I think that this would go a long way towards addressing remaining font sharing issues and allow us to eventually deprecate having fonts in the Keyman keyboards repository itself.

Obviously this could be applied to more than just fonts but would like to limit the conversation here to the font puzzle.

[LornaSIL]

Proposal, in short: allow packages to reference files from GitHub repos, not just local files.

Yes, if that is possible it would be fine.

• What about referencing the Noto fonts online?
  • Use the embed code:

• Or referencing LFF (that would mean we would need to include all the desired fonts on LFF).

• What would we do about online help files needing fonts? Would we still need to add them to s.keyman.com? Or could we again use the github repo or LFF for display of the help files? Or use the embed code from Noto.

• There are fonts we don't really want in LFF (PUA encoded, hacked fonts, etc.) so the referencing the Keyman Github repo might be best.

[mcdurdin]

What about referencing the Noto fonts online?

• Use the embed code:

We couldn't use the embed code as that is a HTML/CSS construct, as we need to be able to download a .ttf directly for embedding into the package. That may be possible also, but it needs to be a stable URL, so I would lean towards requiring a GitHub repo that we control, or that has good stability guarantees.

• What would we do about online help files needing fonts? Would we still need to add them to s.keyman.com? Or could we again use the github repo or LFF for display of the help files? Or use the embed code from Noto.

My thinking is we would automatically deploy all fonts found during the build to s.keyman.com. The primary problem would be if more than one package used the same font in different versions.

• Or referencing LFF (that would mean we would need to include all the desired fonts on LFF).
• There are fonts we don't really want in LFF (PUA encoded, hacked fonts, etc.) so the referencing the Keyman Github repo might be best.

We can certainly support more than one source, and it makes sense to make the mechanism generic for future extension, even if we limit the sources we will support in the keyboards repository. We can have LFF and other fonts in the keymanapp/keyboards repo or even in a separate keymanapp/keyboard-fonts repo to keep it clean.

Technical note

This would slow down the build, as it would now need to source online content. It also introduces some fragility relating to network availability. Caching of fonts could mitigate both of those concerns, but caching also adds complexity.

[mcdurdin]

The Name element may be either a local filename, or a GitHub raw permanent URL, conforming to the pattern:

<Name>https://github.com/{owner}/{repo}/raw/{commit}/[path/]{filename}</Name>

We will add a Source element to .kps Package.Files.File. We would support several sources:

• local - local file system
• flo - fonts.languagetechnology.org
• github
• noto - in the future

The <Source> element may always be omitted -- in which case the compiler does not check for updates.

local: <Source> element is omitted

This is the same as previous versions. <Name> references a file on the local filesystem.

<File>
  <Name>[relative-or-absolute-path/]{filename}</Name>
</File>

Example

<File>
  <Name>Alkalami-Regular.ttf</Name>
</File>

<File>
  <Name>../../shared/fonts/Alkalami-Regular.ttf</Name>
</File>

fonts.languagetechnology.org flo:<id>

References a font by ID from fonts.languagetechnology.org. The resource must be resolved to a permanent URL. If the <Name> field is omitted, the compiler will return an error, reporting the URL it discovers, so the author should be able to paste that URL in.

If the URL in the <Name> field differs from the resolved URL given by FLO, then a hint will be issued by the compiler, allowing the author to easily update to a new version of the resource at their convenience. It is anticipated that this may be scripted on the keyboards repository in the future.

<File>
  <Name>https://github.com/{owner}/{repo}/raw/{commit}/[path/]{filename}</Name>
  <Source>flo:{id}</Source>
</File>

Example

<File>
  <Name>https://github.com/silnrsi/fonts/raw/b88c7af5d16681bd137156929ff8baec82526560/fonts/sil/alkalami/Alkalami-Regular.ttf</Name>
  <Source>flo:alkalami</Source>
</File>

GitHub

A reference to a resource located on GitHub. The resource must be resolved to a permanent URL. If the <Name> field is omitted, the compiler will return an error, reporting the URL it discovers, so the author should be able to paste that URL in.

If the URL in the <Name> field differs from the resolved URL from the <Source> element, then a hint will be issued by the compiler, allowing the author to easily update to a new version of the resource at their convenience. It is anticipated that this may be scripted on the keyboards repository in the future.

<File>
  <Name>https://github.com/{owner}/{repo}/raw/{commit}/[path/]{filename}</Name>
  <Source>https://github.com/{owner}/{repo}/(raw|blob)/([refs/heads/]{branch}|[refs/tags/]{tag}|{commit})/[path/]{filename}</Source>
</File>

Examples

A GitHub raw reference, from a commit, resolves identically.

<File>
  <Name>https://github.com/silnrsi/fonts/raw/b88c7af5d16681bd137156929ff8baec82526560/fonts/sil/alkalami/Alkalami-Regular.ttf</Name>
  <Source>https://github.com/silnrsi/fonts/raw/b88c7af5d16681bd137156929ff8baec82526560/fonts/sil/alkalami/Alkalami-Regular.ttf</Source>
</File>

A GitHub raw reference, from a branch, with refs/heads/ excluded:

<File>
  <Name>https://github.com/silnrsi/fonts/raw/b88c7af5d16681bd137156929ff8baec82526560/fonts/sil/alkalami/Alkalami-Regular.ttf</Name>
  <Source>https://github.com/silnrsi/fonts/raw/main/fonts/sil/alkalami/Alkalami-Regular.ttf</Source>
</File>

A GitHub raw reference, from a branch, with refs/heads/ included:

<File>
  <Name>https://github.com/silnrsi/fonts/raw/b88c7af5d16681bd137156929ff8baec82526560/fonts/sil/alkalami/Alkalami-Regular.ttf</Name>
  <Source>https://github.com/silnrsi/fonts/raw/refs/heads/main/fonts/sil/alkalami/Alkalami-Regular.ttf</Source>
</File>

A GitHub blob reference, as copied from a URL in GitHub:

<File>
  <Name>https://github.com/silnrsi/fonts/raw/b88c7af5d16681bd137156929ff8baec82526560/fonts/sil/alkalami/Alkalami-Regular.ttf</Name>
  <Source>https://github.com/silnrsi/fonts/blob/main/fonts/sil/alkalami/Alkalami-Regular.ttf</Source>
</File>

If the Source element is omitted, then the compiler will not check for updates:

<File>
  <Name>https://github.com/silnrsi/fonts/raw/b88c7af5d16681bd137156929ff8baec82526560/fonts/sil/alkalami/Alkalami-Regular.ttf</Name>
</File>

noto

In a future version.

Additional notes

At the same time CopyLocation, Description, and FileType elements should be deprecated and totally ignored by the compiler. They are busydata.

This would bump Package.System.FileVersion to e.g. 18.0.

There would be no change to the corresponding .kmp file format.