RFC for additional metadata for app

[iainsproat]

This is a proposal to allow additional metadata related to the app to be provided and exported to the image manifest.

[jkutner]

I want think about the finer points some more, it in general I’m a fan of this idea. Thanks for writing it up!

[iainsproat]

@sclevine - we've revised the RFC following your comments in our discussion on Monday. It now proposes a new io.buildpacks.app.metadata property, instead of co-opting the existing io.buildpacks.lifecycle.metadata. The sha property is renamed to ref to make it more generic and less tied to git.

[cmccarthy101]

@sclevine We have made changes to the RfC to reflect our chat about updating the How It Works section. This RfC should be good to vote on now.

[nebhale]

Currently this proposal doesn't describe optionality or extensibility of this and probably needs to. Consider the following cases:

1. Many projects build and test their application artifact using CI tools and only create the deployment artifact (sluglet before, image for CNB) towards the end of their CD pipeline. This results in the input to the buildpack lifecycle not being obviously connected to a repo and commit. Is this metadata required to be populated?
2. Many enterprise applications use VCSs that not only not Git, but in many cases don't use hashes for ids (e.g. Subversion, CVS). Is this metadata extensible to allow identification schemes that don't match current DVCS conventions?

[sclevine]

How about something like this:

Label: io.buildpacks.source.metadata (optional)
Example values:

Git:

{
   "id": "60d5fb7a7ad7c3b357a9d783b740f765d2a0d4d5", 
   "type": "git",
   "metadata": {
      "commit": "60d5fb7a7ad7c3b357a9d783b740f765d2a0d4d5",
      "refs": ["master", "v3.0"],
      "url": "https://github.com/example/source"
   }
}

Source upload:

{
   "id": "146c4bce42545e6a4575283b32a7f01924ef86ce848273079693a42b52b27321", 
   "type": "image",
   "metadata": {
      "digest": "146c4bce42545e6a4575283b32a7f01924ef86ce848273079693a42b52b27321",
      "path": "/source",
      "repository": "hub.docker.io/example/image",
      "refs": ["example/image:mytag"]
   }
}

Where id is a unique identifier for an exact, immutable revision of the source code.

[jchesterpivotal]

In Concourse the comparable concept to id is version, which can be an arbitrary JSON object. Almost always it's a single-key object (eg, the git resource uses {"ref":"...."}.

But occasionally it's necessary to use multiple fields to properly specify the version. The PR resource does this by including the PR number, repository and git commit as keys on each version.

[iainsproat]

I like @sclevine's idea in principle, it feels nicely extendable.

But it feels as if this "wheel" should have been invented somewhere else before (wishful thinking maybe?!).

If we're to develop this metadata for source code identification for other (non-git) systems (present and/or future), then I think we should try to explore whether there's an existing standard ontology that we can adopt.

A quick search surfaced the following existing work or bodies working on software identification ontologies:

• https://github.com/codemeta/codemeta
• OntoSoft
• Research Data Alliance, Software Source Code Identification Working Group
• Software Citation Principles

However, at first glance I don't think they even particularly suit referencing commits within branches for git. One of the recommendations in the Force11 principle is to use DOI's, which feel like a layer of indirection.

I'm not sure if there's something else out there; perhaps copying other software (as in @jchesterpivotal suggestion to follow Concourse's standards) to converge towards a de facto standard?

[jchesterpivotal]

SPDX and CycloneDX also have relevant standards (and rely on other standards like Package URL, which I believe can refer to git commits).

In general, I agree that if there's a good standard we can adopt, we ought to.

[sclevine]

Not sure if I like this better (as I think there are advantages to a single unique string for the version), but here's a concourse-like version:

Git:

{
   "type": "git",
   "version": { "commit": "60d5fb7a7ad7c3b357a9d783b740f765d2a0d4d5" }, 
   "metadata": {
      "refs": ["master", "v3.0"],
      "url": "https://github.com/example/source"
   }
}

Source:

{
   "type": "image",
   "version": { "digest": "146c4bce42545e6a4575283b32a7f01924ef86ce848273079693a42b52b27321" }, 
   "metadata": {
      "path": "/source",
      "repository": "hub.docker.io/example/image",
      "refs": ["example/image:mytag"]
   }
}

[sclevine]

@iainsproat I think the above format addresses the concerns raised, as long as the metadata is clearly optional. Are you willing to update this RFC accordingly?

[iainsproat]

Thanks Stephen. It's in our backlog to update this RFC, we'll make the changes soon (hopefully within the next couple of days).

…

On Fri, Jul 12, 2019 at 3:49 AM Stephen Levine ***@***.***> wrote: @iainsproat <https://github.com/iainsproat> I think the above format addresses the concerns raised, as long as the metadata is clearly optional. Are you willing to update this RFC accordingly? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAQYMMUUBAISZXDHZ6CREDP67WLZANCNFSM4HWT7URQ> .

[cmccarthy101]

@sclevine
We have updated the format of the source information in the RFC given the comments above.

We have a question about the source information. Currently, buildpacks build images according to a single source repo. Is there any necessity to extend this to multiple sources? Eg having the label named sources and having an array of sources.

[hone]

Overall, I like where this RFC is going.

@cmccarthy101 For multiple sources are these sources mirrors or multiple sources needed to construct the image?

[iainsproat]

@hone - these would be multiple sources needed to construct the image.