Proposed HCS standard to normalize voting data in a topic ID

[HGraphPunks]

UPDATE (1/16) :

Since the initial submission of this HIP, we've discussed a lot in the Hedera DAO working group about metadata standards for HCS and have come to the conclusion that i's much more suitable for this HIP to address metadata standards for DAO / Voting use cases only.

Here is the updated proposal metadata object schemas:

Ballot Schema:

{
  "type": "string",
  "id": "string",
  "title": "string ",
  "tokenIds": "Array<string>"
  "description": "string",
  "discussion": "string",
  "scheme": "string",
  "choices":  "Array<string>",
  "startTimestamp": "1672696800000000000",
  "endTimestamp": "1673042400000000000",
}

Vote Schema:

{
  "type": "string",
  "choice": "string",
  "tokenId": "string",
  "serialNumber": "integer",
  "ballotId": "string",
  "accountId" (optional): "string",
}

Using these two formats, here's some examples filling in data for some of the information.

Ballot Example:

{
  "type": "ballot",
  "id": "mnshls_ballot_004",
  "title": "Which of these are the best dessert? ",
  "tokenIds": [
    "0.0.787797",
    "0.0.1062200"
  ],
  "description": "We've taken 6 desserts from around the globe, which is the Moon Shells' favorite one?",
  "discussion": "https://discord.gg/cfwuuFVDXg",
  "scheme": "single-choice",
  "choices": [
    "Cream Brulee (FRANCE) ",
    "Mochi (JAPAN)",
    "Apple Pie (USA)",
    "Gulab Jamun (INDIA)",
    "Kardinalschnitten (AUSTRIA)",
    "Nanaimo Bar (CANADA) "
  ],
  "startTimestamp": "1672696800000000000",
  "endTimestamp": "1673042400000000000"
}

This is what a vote on the ballot above could look like.

Vote Example:

{
  "type": "vote",
  "choice": "Apple Pie (USA)",
  "tokenId": "0.0.1062200",
  "serialNumber": 1,
  "ballotId": "mnshls_ballot_004",
  "accountId": "0.0.475028"
}

Running an initial test ballot in production, these base metadata attributes I believe will help external voting audit sites to easily understand and display the outcome of the votes.

I do want to stress, that adding data to this object for certain use cases shouldn't be restricted by this HIP. The intention of this HIP is only to solidify the minimum required fields to help normalize voting and DAO voting on Hedera using HCS

---

Initial HIP Message is below.

Hello!

I've been working on a standardization of topic id data for a little and think it's at a place where I'd like to start officially discussing with the community about what the HIP standard should include.

Goal:
Normalize all metadata submitted to topics in a flexible way to provide easy reading ability to third parties while not constraining developers with data structures that don't concern their use case. This will allow external applications to easily read and access any topic ID information without needing to rewrite or re architect software to map to new attributes for every new topic id.

This structure will also help new developers organize the data they need submitted to the topic id without having to guess or make new JSON objects that no other application uses, fragmenting the data sets across topic ids.

Initial Proposed Idea:
First part of the proposal is a header object. This would likely be the first post on a topic id. The header object would give context to the data being logged on this topic id, along with the format of the data. The description, creator, id, and name would all be optional but can help others reading the data understand its purpose and point of origin.

The most important attributes in the header are type and format

type defines this is the header for easy reading of any application
format defines the expected JSON structure of the records contained in messages on this topic id

 {
        "type": "header",
        "id": "header_moonshells_voting",
        "name": "Moon Shells Voting",
        "creator": "Turtle Moon Command Center (TMCC)",
        "description": "TMCC DAO votes for the NFT project Moon Shells",
        "format": "dao-voting" 
}

There will be several default formats proposed in this HIP for dao voting, farming, staking, p2p transactions, auctions, and custom. Each format will use the base attributes of the JSON object below. The root attributes that should be present on every submitted message are type, id, properties.

{
        "type": "sting",
        "id": "string",
        "properties": "object"
}

The configurable portions of this object will reside in the properties object. format attribute will define both the types one should expect in the topic id messages, and the properties object that will exist in each type.

Example:
format: "dao-voting" types:

{
        "type": "create-ballot",
        "id": "string",
        "properties": {
	    "tokenId": "string",
	    "title": "string",
	    "description": "string",
	    "discussion": "string",
	    "scheme": "string",
	    "choices": "array",
	    "startTimestamp": "integer",
	    "endTimestamp": "integer",
       }
 }
{
        "type": "cast-vote",
        "id": "string",
        "properties": {
             "vote": "integer",
	     "ballotId”: ”string”
        }
}

The examples below use the dao-voting format in practice. This is an adaptation of the Calaxy voting format placed into the structure of the boilerplate object structure being proposed above.

Examples of voting use cases.
First a create ballot call that defines an id and contains properties set out for “create-ballot” type.

{
        "type": "create-ballot",
        "id": "ballot_moonshells_cgip-4",
        "properties": {
	    "tokenId": "0.0.859814",
	    "title": "CGIP-4 Community Governance & Voting Model",
	    "description": "https://github.com/.../cgip-4.md",
	    "discussion": "https://github.com/.../discussions",
	    "scheme": "single-choice",
	    "choices": ["Yes", "No"],
	    "startTimestamp": "1655323200.236702000",
	    "endTimestamp": "1655928000.236702000",
       }
 }

Second a cast-vote call that uses the properties set out for “cast-vote” type.

{
        "type": "cast-vote",
        "id": "vote_moonshells_0.0.123456",
        "properties": {
             "vote": 0,
	     "ballotId”: ”ballot_moonshells_cgip-4”
        }
}

Custom format will allow the most flexibility for future use cases before a HIP defines the format formally.
If format:custom is in the header, the creator of the topic id should then submit a new message with a glossary for the properties and types that will be present in the topic id.

 {
        "type": "glossary",
        "id": "glossary_moonshells_game",
        “setTypes”:[
	    {
		    “type”: "highscore”,
		    "playerId": "string",
		    "playerName": "string",
	            "accountId": "string",
                    "score": "integer",
	    },
	    {
		    "type": "achievement",
		    "playerId": "string",
		    "playerName": "string",
	            "accountId": "string",
                    "achievementId": "integer",
	    },
        ]
}

I'm very open to changing and adapting this format and believe it's a great time to start discussing as a community on wants / desires for object definitions inside of HCS messages.

[Cooper-Kunz]

Hey @HGraphPunks - thanks for taking the initiative here.

I understand that this is of interest to you, and this seems like a small lightweight change required to make our hcs-governance software accomodate (messages/usage of which is outlined here in CGIP-4) so I'm happy to support. But could I ask for a bit of an elaboration on expected use case? If we we update our repository to accomodate this schema/structure, what do you expect to gain? Do you see a L2 network of "HCS validators" that are picking up on all topics/messages using this schema, or something similar?

cc @bugbytesinc for his thoughts as the lead dev on the voting system.

[mgarbs]

I like this idea. Having a type and id standard does make sense to me as it would give the ability to applications wanting to aggregate topics for big data queries/processing.

[mgarbs]

Thinking about this and conferring with the team. It would be good to add version numbers in this standard.

Also, rather than create a new JSON format, there are standard mechanisms that describe a schema like Avro or Parquet and are more efficient for big data analytics, so it may be valuable to consider these. They've done a lot of work ensuring schemas have fewer mistakes, help in migrations etc.

edit:
We could use one of the json schema drafts to validate these schemas. @HGraphPunks, for example the first schema you wrote

 {
        "type": "header",
        "id": "header_moonshells_voting",
        "name": "Moon Shells Voting",
        "creator": "Turtle Moon Command Center (TMCC)",
        "description": "TMCC DAO votes for the NFT project Moon Shells",
        "format": "dao-voting" 
}

using a draft07 json schema converter tool with versioning added, I got this

{
	"definitions": {},
	"$schema": "http://json-schema.org/draft-07/schema#", 
	"$id": "https://example.com/object1670955561.json", 
	"title": "Root", 
	"type": "object",
	"required": [
		"version",
		"type",
		"id",
	],
	"properties": {
		"version": {
			"$id": "#root/version", 
			"title": "Version", 
			"type": "integer",
			"examples": [
				1
			],
			"default": 0
		},
		"type": {
			"$id": "#root/type", 
			"title": "Type", 
			"type": "string",
			"default": "",
			"examples": [
				"header"
			],
			"pattern": "^.*$"
		},
		"id": {
			"$id": "#root/id", 
			"title": "Id", 
			"type": "string",
			"default": "",
			"examples": [
				"header_moonshells_voting"
			],
			"pattern": "^.*$"
		}
}

[nickpoorman]

It's not clear to me if we are talking only about the HCS message data or strictly the metadata for HCS messages here. Either way, I have some thoughts since they go hand-in-hand.

I'm wondering if it wouldn't be better to have a separate schema registry. I can imagine how mixing the data plane and control plane could lead to edge cases that developers have to deal with.

Schema Registry

If there was a schema registry, the process could look like this:

1. Create a topic.
2. Create a schema and register it to the schema registry for a given topic id. Each schema is immutable and has version numbers, so the first one starts at version 0.
3. A message comes in on a topic, the client can make a request to look up the schema from the schema registry and cache it in memory (these could be surfaced in mirror node).
4. The returned schema could be used to validate and unmarshal data; ideally, data could be in any format... more on this below.
5. The owner of a topic could update the schema in the registry with an update schema transaction, thus incrementing the version number of the schema.
6. When a user sees a message with a different schema version number, the user could make a request to get the new schema, cache it in memory, and unmarshal the data.

Data Formats

Maybe we should avoid locking anyone into a given data format. HTTP has ways of doing this with its content-type. The schema itself could be registered as a given content-type, i.e. JSON, Flatbuffers, Protobuf, Avro, Parquet, Arrow-Flight, some other text or binary format, etc...

If we don't want to open it up to all types, we could start by supporting a small list of common formats. I don't think all the formats make sense as some formats such as Parquet contain all the schema metadata necessary for each file to be parsed (that's a lot of duplicate data to include in each message), but there are some formats such as JSON that can be validated with side channel mechanisms like JSON Schema or Protobuf, etc. Generally, formats such Avro and Parquet are used to aggregate messages that were transmitted via something like JSON or Protobuf in order to compress them into a chunked file.

My suggestion would be not to bin anything into a specific data format as some users may wish to use a more efficient binary format (of which JSON is not an efficient binary format) and instead provide native schema registry functionality.

Schema Evolution

Left for future discussion, it seems reasonable to have safeguards in place that disallow the evolution of schemas that would break backward compatibility for downstream consumers. For example, if users were pumping their topic messages into a data lake or data warehouse, data engineers downstream would be in disfavor of schema evolution that drops a field / column that was previously defined in the schema as a non-null or required field.

[HGraphPunks]

I really love the idea of a schema registry and appreciate the time you put into this post.

I think there's a great discussion to continue here about HCS standards or at least the initial architecture on where schemas could live to help client applications easily read topics externally when that is a want from the creator of that topic. I thought a possible solution could be adding the sequence numbers (in a JSON format) to the memo of the topic on which messages contain that schema?

I have updated this discussion to lessen the scope to only voting to push this HIP forward for the Hedera DAO working group and look forward to working on solutions for HCS and standardization to increase it's usage and usability.

[web3-nomad]

I'd like to be able see more schema choices being available,
and potentially the 'choices' field be optional, as some of these schemas may not need them.
for example:

schema - quroum (where it is just about people being present)
schema - Yes/No

now these could be done via choices, but they don't really need to be.

also, for certain proposals, a simple 'text' field might not be sufficient for description. (imagine a 10-page budget proposal). we might need some kind of reference link (like IPFS) that could be referenced.
This wouldn't be the same as 'discussion' (which would be mutable)

[web3-nomad]

I'm going to suggest another field called 'schema-extension' where, based on the schema, other fields may occur.

here are 3-5 other schema options, as discussed in today's DAO call. (in less deranged, to more deranged order).

ranked-choice

similar to multiple choice, but where ranking is important.. and you could limit the choices to rank your top-3.

the write-in

In this option, this would allow people to be able add an option manually, similar to how a Write-in candidate works.

dynamic choices / the hottest-100 scenario. from Triple J

In this voting scheme, people would first propose choices to be voted on. once the nomination period is complete, people can vote on which of the choices they would like. (in this case a ranked-10 choice, which is slightly different than multiple-choice)

choices by lookup

In this voting scheme the list of choices are drawn from a external source. (for example NFTs from a PFP collection, or players from a sports team).

(ok..these are now admittedly whacky)

non-discrete choices

predict the weather, choose a rational number between 1 & 10.. where the ballot provides a range instead of a set of discrete choices.
another use case could be selecting a fee percentage, where people can input a number between 0 & 100%, and the proposal would perform some median/weighting to them to determine the final outcome.