From 2c56e0d992ca828f539e72f45fd63c3a47093d15 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Tue, 12 Oct 2021 05:48:48 -0700 Subject: [PATCH 1/6] README for the cmd package --- README.md | 2 +- cmd/gossamer/README.md | 127 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 128 insertions(+), 1 deletion(-) create mode 100644 cmd/gossamer/README.md diff --git a/README.md b/README.md index 656a36e7de..fb93006df6 100644 --- a/README.md +++ b/README.md @@ -154,6 +154,6 @@ _GNU Lesser General Public License v3.0_

- +

diff --git a/cmd/gossamer/README.md b/cmd/gossamer/README.md new file mode 100644 index 0000000000..78024ecce5 --- /dev/null +++ b/cmd/gossamer/README.md @@ -0,0 +1,127 @@ +# Gossamer `cmd` Package + +This package encapsulates the entry point to Gossamer - it uses the popular +[`cli` package from `urfave`](https://github.com/urfave/cli/blob/master/docs/v1/manual.md) to expose a command-line +interface (CLI). The Gossamer CLI accepts several subcommands, each of which is associated with an "action"; these +subcommands and their corresponding actions are defined in [`main.go`](main.go). When the Gossamer CLI is executed +without a subcommand, the `gossamerAction` is invoked. What follows is a list of the Gossamer subcommands, as well as an +overview of some of the flags/parameters they accept. The flags/parameters that the Gossamer CLI supports are defined in +[`flags.go`](flags.go). For an exhaustive reference of the Gossamer CLI capabilities, follow +[the installation instructions](../../README.md#installation) and execute `./bin/gossamer --help`. + +- `gossamerAction` - This the default Gossamer execution action - it will launch a Gossamer blockchain client. The + details of how Gossamer orchestrates a blockchain client are [described below](#client-components). + - `chain` - specifies the [chain configuration](../../chain) that the + [Gossamer host node](https://chainsafe.github.io/gossamer/getting-started/overview/host-architecture/) should load + - `log` - supports levels `crit` (silent) to `trce` (detailed), default is `info` + - `name` - node name, as it will appear in, e.g., [telemetry](https://telemetry.polkadot.io/) +- `account` - The `account` subcommand provides the user with capabilities related to generating and using `ed25519`, + `secp256k1`, and `sr25519` [account keys](https://wiki.polkadot.network/docs/learn-keys), and managing the keys + present in the [Gossamer keystore](#keystore). The action for this subcommand is defined in [account.go](account.go); + it is an interface to the capabilities defined in the [`crypto`](../../lib/crypto) and + [`keystore`](../../lib/keystore) packages. This subcommand provides capabilities that are similar to + [Parity's Subkey utility](https://docs.substrate.io/v3/tools/subkey). + - `generate` - creates a new key pair; specify `--ed25519`, `--secp256k1`, or `--sr25519` (default) + - `list` - lists the keys in the Gossamer keystore + - `password` - allows the user to provide a password for encrypting a generated key, or unlocking the Gossamer + keystore +- `import-runtime` - This subcommand takes a [Wasm runtime binary](https://wiki.polkadot.network/docs/learn-wasm) and + uses it to generate a [genesis](https://wiki.polkadot.network/docs/glossary#genesis) configuration file; it does not + require any flags, but expects the path to a Wasm file to be provided as a command-line parameter (example: + `./bin/gossamer import-runtime runtime.wasm > genesis.json`). +- `build-spec` - This subcommand allows the user to "compile" a human-readable Gossamer genesis configuration file into + a format that the Gossamer node can consume. If the `--genesis` parameter is not provided, the generated genesis + configuration will represent the Gossamer default configuration. + - `genesis` - path to the human-readable configuration file that should be compiled into a format that Gossamer can + consume +- `import-state` - The `import-state` subcommand allows a user to seed [Gossamer storage](../../dot/state) with + key-value pairs in the form of a JSON file. The input for this subcommand can be retrieved from + [the state_getPairs RPC endpoint](https://github.com/w3f/PSPs/blob/master/PSPs/drafts/psp-6.md#1114-state_getpairs). + - `first-slot` - the first [BABE](https://wiki.polkadot.network/docs/learn-consensus#block-production-babe) slot, + which is the timestamp of the first block divided by the slot duration + - `header` - path to a JSON file that describes the block header corresponding to the given state + - `state` - path to a JSON file that contains the key-value pairs with which to seed Gossamer storage +- `init` - This subcommand accepts a genesis configuration file and uses it to initialise the Gossamer node and its + state. + - `genesis` - path to the "compiled" genesis configuration file that should be used to initialise the Gossamer node + and its state +- `export` - The `export` subcommand transforms a genesis configuration and Gossamer state into a TOML configuration + file. + - `config` - path to a TOML configuration file (e.g. [Polkadot](../../chain/polkadot/config.toml)) + - `basepath` - path to the Gossamer data directory that defines the state to export + +## Client Components + +In its default method of execution, Gossamer orchestrates a number of modular services that run +[concurrently as goroutines](https://www.golang-book.com/books/intro/10) and work together to implement the protocols of +a blockchain network. Alongside these services, Gossamer manages [a keystore](#keystore), [a runtime](#runtime), and +[monitoring utilities](#monitoring), all of which are described in greater detail below. The entry point to the Gossamer +blockchain client capabilities is the `gossamerAction` function that is defined in [main.go](main.go), which in turn +invokes the `NewNode` function in [dot/node.go](../../dot/node.go). `NewNode` calls into functions that are defined in +[dot/services.go](../../dot/services.go) and starts the services that power a Gossamer node. What follows is a list that +describes the services and capabilities that inform a Gossamer blockchain client: + +- State - This service is a wrapper around an instance of [`chaindb`](https://github.com/ChainSafe/chaindb), a key-value + database that is built on top of [BadgerDB](https://github.com/dgraph-io/badger) from [Dgraph](https://dgraph.io/). + The state service provides storage capabilities for the other Gossamer services - each service is assigned a prefix + that is added to its storage keys. The state service is defined in [dot/state/service.go](../../dot/state/service.go). +- Network - The network service, which is defined in [dot/network/service.go](../../dot/network/service.go), is built on + top of [the Go implementation](https://github.com/libp2p/go-libp2p) of [the `libp2p` protocol](https://libp2p.io/). + This service manages a `libp2p` "host", a peer-to-peer networking term for a network participant that is providing + both client _and_ server capabilities to a peer-to-peer network. Gossamer's network service manages the discovery of + other hosts as well as the connections with these hosts that allow Gossamer to communicate with its network peers. +- Digest Handler - The digest handler ([dot/digest/digest.go](../../dot/digest/digest.go)) manages the verification of + the [cryptographic digests](https://docs.substrate.io/v3/getting-started/glossary/#digest) that are present in block + headers. +- Consensus - The BABE and GRANDPA services work together to provide Gossamer with its + [hybrid consensus](https://wiki.polkadot.network/docs/learn-consensus#hybrid-consensus) capabilities. The term "hybrid + consensus" refers to the fact that block _production_ is decoupled from block _finalisation_. Block production is + handled by the BABE service, which is defined in [lib/babe/babe.go](../../lib/babe/babe.go); block finalisation is + handled by the GRANDPA service, which is defined in [lib/grandpa/grandpa.go](../../lib/grandpa/grandpa.go). +- Sync - This service is concerned with keeping Gossamer in sync with a blockchain - it implements a "bootstrap" mode, + to download and verify blocks that are part of an existing chain's history, and a "tip-syncing" mode that manages the + multiple candidate forks that may exist at the head of a live chain. The sync service makes use of + [a block verification utility](../../lib/babe/verify.go) that implements BABE logic and is used by Gossamer to verify + blocks that were produced by other other nodes in the network. The sync service is defined + [dot/sync/syncer.go](../../dot/sync/syncer.go). +- RPC - This service, which is defined in [dot/rpc/service.go](../../dot/rpc/service.go), exposes a JSON-RPC interface + that is used by client applications like [Polkadot JS Apps UI](https://polkadot.js.org/apps/). The RPC interface is + used to interact with Gossamer to perform tasks such as key management, as well as for interacting with the "inner" + runtime by querying storage and submitting transactions. +- System - The system service is defined in [dot/system/service.go](../../dot/system/service.go) and exposes metadata + about the Gossamer system, such as the names and versions of the protocols that it implements. +- Core - As its name implies, the core service ([dot/core/service.go](../../dot/core/service.go)) encapsulates a range + of capabilities that are central to the functioning of a Gossamer node. In general, the core service is a type of + dispatcher that coordinates interactions between services, e.g. writing blocks to the database, reloading + [the runtime](#runtime) when its definition is updated, etc. + +### Keystore + +The Gossamer keystore ([lib/keystore](../../lib/keystore)) is used for managing the public/private cryptographic key +pairs that are used for participating in a blockchain network. Public keys are used to identify network participants; +network participants use their private keys to sign messages in order to authorise privileged actions. In addition to +informing the Gossamer blockchain client capabilities, the Gossamer keystore is accessible by way of the `account` +subcommand. The Gossamer keystore manages a number of key types, some of which are listed below: + +- `babe` - This key is used for signing messages related to the BABE block production algorithm. +- `gran` - The GRANDPA key is used for participating in GRANDPA block finalisation. +- `imon` - The name of this key is a reference to "ImOnline", which is an + [online message](https://wiki.polkadot.network/docs/glossary#online-message) that Gossamer nodes use to report + liveliness. + +### Runtime + +In addition to the above-described services, Gossamer hosts a Wasm execution environment that is used to manage an +upgradeable blockchain runtime. The runtime must be implemented in Wasm, and must expose an interface that is specified +in [lib/runtime/interface.go](../../lib/runtime/interface.go). The runtime defines the blockchain's state transition +function, and the various Gossamer services consume this capability in order to author blocks, as well as to verify +blocks that were authored by network peers. + +### Monitoring + +Gossamer publishes telemetry data and also includes an embedded Prometheus server that reports metrics. The metrics +capabilities are defined in the [dot/metrics](../../dot/metrics) package and build on +[the metrics library that is included with Go Ethereum](https://github.com/ethereum/go-ethereum/blob/master/metrics/README.md). +The Gossamer telemetry server publishes telemetry data that is compatible with +[Polkadot Telemetry](https://github.com/paritytech/substrate-telemetry) and +[its helpful UI](https://telemetry.polkadot.io/). From 1598ebade188975343a8fd7e95f6111e5a8d5465 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Tue, 26 Oct 2021 10:48:34 -0700 Subject: [PATCH 2/6] Apply @qdm12 suggestions from code review Co-authored-by: Quentin McGaw --- cmd/gossamer/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/cmd/gossamer/README.md b/cmd/gossamer/README.md index 78024ecce5..0c1865a68b 100644 --- a/cmd/gossamer/README.md +++ b/cmd/gossamer/README.md @@ -10,20 +10,20 @@ overview of some of the flags/parameters they accept. The flags/parameters that [the installation instructions](../../README.md#installation) and execute `./bin/gossamer --help`. - `gossamerAction` - This the default Gossamer execution action - it will launch a Gossamer blockchain client. The - details of how Gossamer orchestrates a blockchain client are [described below](#client-components). + details of how Gossamer orchestrates a blockchain client are [described below in the Client Components section](#client-components). - `chain` - specifies the [chain configuration](../../chain) that the [Gossamer host node](https://chainsafe.github.io/gossamer/getting-started/overview/host-architecture/) should load - - `log` - supports levels `crit` (silent) to `trce` (detailed), default is `info` + - `log` - supports levels `crit` (silent), `error`, `warn`, `info` and `trce` (detailed), default is `info` - `name` - node name, as it will appear in, e.g., [telemetry](https://telemetry.polkadot.io/) - `account` - The `account` subcommand provides the user with capabilities related to generating and using `ed25519`, `secp256k1`, and `sr25519` [account keys](https://wiki.polkadot.network/docs/learn-keys), and managing the keys present in the [Gossamer keystore](#keystore). The action for this subcommand is defined in [account.go](account.go); - it is an interface to the capabilities defined in the [`crypto`](../../lib/crypto) and - [`keystore`](../../lib/keystore) packages. This subcommand provides capabilities that are similar to + it is an interface to the capabilities defined in the [`lib/crypto`](../../lib/crypto) and + [`lib/keystore`](../../lib/keystore) packages. This subcommand provides capabilities that are similar to [Parity's Subkey utility](https://docs.substrate.io/v3/tools/subkey). - `generate` - creates a new key pair; specify `--ed25519`, `--secp256k1`, or `--sr25519` (default) - `list` - lists the keys in the Gossamer keystore - - `password` - allows the user to provide a password for encrypting a generated key, or unlocking the Gossamer + - `password` - allows the user to provide a password to either encrypt a generated key or unlock the Gossamer keystore - `import-runtime` - This subcommand takes a [Wasm runtime binary](https://wiki.polkadot.network/docs/learn-wasm) and uses it to generate a [genesis](https://wiki.polkadot.network/docs/glossary#genesis) configuration file; it does not @@ -36,7 +36,7 @@ overview of some of the flags/parameters they accept. The flags/parameters that consume - `import-state` - The `import-state` subcommand allows a user to seed [Gossamer storage](../../dot/state) with key-value pairs in the form of a JSON file. The input for this subcommand can be retrieved from - [the state_getPairs RPC endpoint](https://github.com/w3f/PSPs/blob/master/PSPs/drafts/psp-6.md#1114-state_getpairs). + [the `state_getPairs` RPC endpoint](https://github.com/w3f/PSPs/blob/master/PSPs/drafts/psp-6.md#1114-state_getpairs). - `first-slot` - the first [BABE](https://wiki.polkadot.network/docs/learn-consensus#block-production-babe) slot, which is the timestamp of the first block divided by the slot duration - `header` - path to a JSON file that describes the block header corresponding to the given state From 97ccceeed9c2676eb19730ee0d863fa1a37001b1 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Tue, 2 Nov 2021 11:33:49 -0700 Subject: [PATCH 3/6] Review changes --- cmd/gossamer/README.md | 227 ++++++++++++++++++++++++++--------------- 1 file changed, 145 insertions(+), 82 deletions(-) diff --git a/cmd/gossamer/README.md b/cmd/gossamer/README.md index 0c1865a68b..193b4f5834 100644 --- a/cmd/gossamer/README.md +++ b/cmd/gossamer/README.md @@ -4,51 +4,87 @@ This package encapsulates the entry point to Gossamer - it uses the popular [`cli` package from `urfave`](https://github.com/urfave/cli/blob/master/docs/v1/manual.md) to expose a command-line interface (CLI). The Gossamer CLI accepts several subcommands, each of which is associated with an "action"; these subcommands and their corresponding actions are defined in [`main.go`](main.go). When the Gossamer CLI is executed -without a subcommand, the `gossamerAction` is invoked. What follows is a list of the Gossamer subcommands, as well as an -overview of some of the flags/parameters they accept. The flags/parameters that the Gossamer CLI supports are defined in -[`flags.go`](flags.go). For an exhaustive reference of the Gossamer CLI capabilities, follow -[the installation instructions](../../README.md#installation) and execute `./bin/gossamer --help`. - -- `gossamerAction` - This the default Gossamer execution action - it will launch a Gossamer blockchain client. The - details of how Gossamer orchestrates a blockchain client are [described below in the Client Components section](#client-components). - - `chain` - specifies the [chain configuration](../../chain) that the - [Gossamer host node](https://chainsafe.github.io/gossamer/getting-started/overview/host-architecture/) should load - - `log` - supports levels `crit` (silent), `error`, `warn`, `info` and `trce` (detailed), default is `info` - - `name` - node name, as it will appear in, e.g., [telemetry](https://telemetry.polkadot.io/) -- `account` - The `account` subcommand provides the user with capabilities related to generating and using `ed25519`, - `secp256k1`, and `sr25519` [account keys](https://wiki.polkadot.network/docs/learn-keys), and managing the keys - present in the [Gossamer keystore](#keystore). The action for this subcommand is defined in [account.go](account.go); - it is an interface to the capabilities defined in the [`lib/crypto`](../../lib/crypto) and - [`lib/keystore`](../../lib/keystore) packages. This subcommand provides capabilities that are similar to - [Parity's Subkey utility](https://docs.substrate.io/v3/tools/subkey). - - `generate` - creates a new key pair; specify `--ed25519`, `--secp256k1`, or `--sr25519` (default) - - `list` - lists the keys in the Gossamer keystore - - `password` - allows the user to provide a password to either encrypt a generated key or unlock the Gossamer - keystore -- `import-runtime` - This subcommand takes a [Wasm runtime binary](https://wiki.polkadot.network/docs/learn-wasm) and - uses it to generate a [genesis](https://wiki.polkadot.network/docs/glossary#genesis) configuration file; it does not - require any flags, but expects the path to a Wasm file to be provided as a command-line parameter (example: - `./bin/gossamer import-runtime runtime.wasm > genesis.json`). -- `build-spec` - This subcommand allows the user to "compile" a human-readable Gossamer genesis configuration file into - a format that the Gossamer node can consume. If the `--genesis` parameter is not provided, the generated genesis - configuration will represent the Gossamer default configuration. - - `genesis` - path to the human-readable configuration file that should be compiled into a format that Gossamer can - consume -- `import-state` - The `import-state` subcommand allows a user to seed [Gossamer storage](../../dot/state) with - key-value pairs in the form of a JSON file. The input for this subcommand can be retrieved from - [the `state_getPairs` RPC endpoint](https://github.com/w3f/PSPs/blob/master/PSPs/drafts/psp-6.md#1114-state_getpairs). - - `first-slot` - the first [BABE](https://wiki.polkadot.network/docs/learn-consensus#block-production-babe) slot, - which is the timestamp of the first block divided by the slot duration - - `header` - path to a JSON file that describes the block header corresponding to the given state - - `state` - path to a JSON file that contains the key-value pairs with which to seed Gossamer storage -- `init` - This subcommand accepts a genesis configuration file and uses it to initialise the Gossamer node and its - state. - - `genesis` - path to the "compiled" genesis configuration file that should be used to initialise the Gossamer node - and its state -- `export` - The `export` subcommand transforms a genesis configuration and Gossamer state into a TOML configuration - file. - - `config` - path to a TOML configuration file (e.g. [Polkadot](../../chain/polkadot/config.toml)) - - `basepath` - path to the Gossamer data directory that defines the state to export +without a subcommand, the `gossamerAction` is invoked. + +## Actions & Subcommands + +What follows is a list of the Gossamer subcommands, as well as an overview of some of the flags/parameters they accept. +The flags/parameters that the Gossamer CLI supports are defined in [`flags.go`](flags.go). For an exhaustive reference +of the Gossamer CLI capabilities, follow [the installation instructions](../../README.md#installation) and execute +`./bin/gossamer --help`. + +### Default Command + +This the default Gossamer execution method, which invokes the `gossamerAction` function defined in +[`main.go`](main.go) - it will launch a Gossamer blockchain client. The details of how Gossamer orchestrates a +blockchain client are [described below in the Client Components section](#client-components). + +- `basepath` - the path to the directory where Gossamer will store its data +- `chain` - specifies the [chain configuration](../../chain) that the + [Gossamer host node](https://chainsafe.github.io/gossamer/getting-started/overview/host-architecture/) should load +- `key` - specifies a test keyring account to use (e.g. `--key=alice`) +- `log` - supports levels `crit` (silent), `error`, `warn`, `info`, `debug`, and `trce` (detailed), default is `info` +- `name` - node name, as it will appear in, e.g., [telemetry](https://telemetry.polkadot.io/) + +### Init Subcommand + +This subcommand accepts a genesis configuration file and uses it to initialise the Gossamer node and its state. The +`init` subcommand invokes the `initAction` function defined in [`main.go`](main.go). + +- `genesis` - path to the "compiled" genesis configuration file that should be used to initialise the Gossamer node and + its state + +### Account Subcommand + +The `account` subcommand provides the user with capabilities related to generating and using `ed25519`, `secp256k1`, and +`sr25519` [account keys](https://wiki.polkadot.network/docs/learn-keys), and managing the keys present in the +[Gossamer keystore](#keystore). The `actionAction` function is defined in [account.go](account.go); it is an interface +to the capabilities defined in the [`lib/crypto`](../../lib/crypto) and [`lib/keystore`](../../lib/keystore) packages. +This subcommand provides capabilities that are similar to +[Parity's Subkey utility](https://docs.substrate.io/v3/tools/subkey). + +- `generate` - creates a new key pair; specify `--ed25519`, `--secp256k1`, or `--sr25519` (default) +- `list` - lists the keys in the Gossamer keystore +- `password` - allows the user to provide a password to either encrypt a generated key or unlock the Gossamer keystore + +### Import Runtime Subcommand + +This subcommand takes a [Wasm runtime binary](https://wiki.polkadot.network/docs/learn-wasm) and uses it to generate a +[genesis](https://wiki.polkadot.network/docs/glossary#genesis) configuration file; it does not require any flags, but +expects the path to a Wasm file to be provided as a command-line parameter (example: +`./bin/gossamer import-runtime runtime.wasm > genesis.json`). The `import-runtime` subcommand invokes the +`importRuntimeAction` function defined in [`main.go`](main.go). + +### Build Spec Subcommand + +This subcommand allows the user to "compile" a human-readable Gossamer genesis configuration file into a format that the +Gossamer node can consume. If the `--genesis` parameter is not provided, the generated genesis configuration will +represent the Gossamer default configuration. The `build-spec` subcommand invokes the `buildSpecAction` function defined +in [`main.go`](main.go). + +- `genesis` - path to the human-readable configuration file that should be compiled into a format that Gossamer can + consume + +### Import State Subcommand + +The `import-state` subcommand allows a user to seed [Gossamer storage](../../dot/state) with key-value pairs in the form +of a JSON file. The input for this subcommand can be retrieved from +[the `state_getPairs` RPC endpoint](https://github.com/w3f/PSPs/blob/master/PSPs/drafts/psp-6.md#1114-state_getpairs). +The `importStateAction` function is defined in [`main.go`](main.go). + +- `first-slot` - the first [BABE](https://wiki.polkadot.network/docs/learn-consensus#block-production-babe) slot, which + can be found by checking the [digest](https://docs.substrate.io/v3/getting-started/glossary/#digest) for a chain's + first block _after_ its [genesis block](https://wiki.polkadot.network/docs/glossary#genesis) +- `header` - path to a JSON file that describes the block header corresponding to the given state +- `state` - path to a JSON file that contains the key-value pairs with which to seed Gossamer storage + +### Export Subcommand + +The `export` subcommand transforms a genesis configuration and Gossamer state into a TOML configuration file. This +subcommand invokes the `exportAction` function defined in [`export.go`](export.go). + +- `config` - path to a TOML configuration file (e.g. those defined in [the `chain` directory](../../chain)) +- `basepath` - path to the Gossamer data directory that defines the state to export ## Client Components @@ -58,42 +94,67 @@ a blockchain network. Alongside these services, Gossamer manages [a keystore](#k [monitoring utilities](#monitoring), all of which are described in greater detail below. The entry point to the Gossamer blockchain client capabilities is the `gossamerAction` function that is defined in [main.go](main.go), which in turn invokes the `NewNode` function in [dot/node.go](../../dot/node.go). `NewNode` calls into functions that are defined in -[dot/services.go](../../dot/services.go) and starts the services that power a Gossamer node. What follows is a list that -describes the services and capabilities that inform a Gossamer blockchain client: - -- State - This service is a wrapper around an instance of [`chaindb`](https://github.com/ChainSafe/chaindb), a key-value - database that is built on top of [BadgerDB](https://github.com/dgraph-io/badger) from [Dgraph](https://dgraph.io/). - The state service provides storage capabilities for the other Gossamer services - each service is assigned a prefix - that is added to its storage keys. The state service is defined in [dot/state/service.go](../../dot/state/service.go). -- Network - The network service, which is defined in [dot/network/service.go](../../dot/network/service.go), is built on - top of [the Go implementation](https://github.com/libp2p/go-libp2p) of [the `libp2p` protocol](https://libp2p.io/). - This service manages a `libp2p` "host", a peer-to-peer networking term for a network participant that is providing - both client _and_ server capabilities to a peer-to-peer network. Gossamer's network service manages the discovery of - other hosts as well as the connections with these hosts that allow Gossamer to communicate with its network peers. -- Digest Handler - The digest handler ([dot/digest/digest.go](../../dot/digest/digest.go)) manages the verification of - the [cryptographic digests](https://docs.substrate.io/v3/getting-started/glossary/#digest) that are present in block - headers. -- Consensus - The BABE and GRANDPA services work together to provide Gossamer with its - [hybrid consensus](https://wiki.polkadot.network/docs/learn-consensus#hybrid-consensus) capabilities. The term "hybrid - consensus" refers to the fact that block _production_ is decoupled from block _finalisation_. Block production is - handled by the BABE service, which is defined in [lib/babe/babe.go](../../lib/babe/babe.go); block finalisation is - handled by the GRANDPA service, which is defined in [lib/grandpa/grandpa.go](../../lib/grandpa/grandpa.go). -- Sync - This service is concerned with keeping Gossamer in sync with a blockchain - it implements a "bootstrap" mode, - to download and verify blocks that are part of an existing chain's history, and a "tip-syncing" mode that manages the - multiple candidate forks that may exist at the head of a live chain. The sync service makes use of - [a block verification utility](../../lib/babe/verify.go) that implements BABE logic and is used by Gossamer to verify - blocks that were produced by other other nodes in the network. The sync service is defined - [dot/sync/syncer.go](../../dot/sync/syncer.go). -- RPC - This service, which is defined in [dot/rpc/service.go](../../dot/rpc/service.go), exposes a JSON-RPC interface - that is used by client applications like [Polkadot JS Apps UI](https://polkadot.js.org/apps/). The RPC interface is - used to interact with Gossamer to perform tasks such as key management, as well as for interacting with the "inner" - runtime by querying storage and submitting transactions. -- System - The system service is defined in [dot/system/service.go](../../dot/system/service.go) and exposes metadata - about the Gossamer system, such as the names and versions of the protocols that it implements. -- Core - As its name implies, the core service ([dot/core/service.go](../../dot/core/service.go)) encapsulates a range - of capabilities that are central to the functioning of a Gossamer node. In general, the core service is a type of - dispatcher that coordinates interactions between services, e.g. writing blocks to the database, reloading - [the runtime](#runtime) when its definition is updated, etc. +[dot/services.go](../../dot/services.go) and starts the services that power a Gossamer node. + +### Services & Capabilities + +What follows is a list that describes the services and capabilities that inform a Gossamer blockchain client: + +#### State + +This service is a wrapper around an instance of [`chaindb`](https://github.com/ChainSafe/chaindb), a key-value database +that is built on top of [BadgerDB](https://github.com/dgraph-io/badger) from [Dgraph](https://dgraph.io/). The state +service provides storage capabilities for the other Gossamer services - each service is assigned a prefix that is added +to its storage keys. The state service is defined in [dot/state/service.go](../../dot/state/service.go). + +#### Network + +The network service, which is defined in [dot/network/service.go](../../dot/network/service.go), is built on top of +[the Go implementation](https://github.com/libp2p/go-libp2p) of [the `libp2p` protocol](https://libp2p.io/). This +service manages a `libp2p` "host", a peer-to-peer networking term for a network participant that is providing both +client _and_ server capabilities to a peer-to-peer network. Gossamer's network service manages the discovery of other +hosts as well as the connections with these hosts that allow Gossamer to communicate with its network peers. + +#### Digest Handler + +The digest handler ([dot/digest/digest.go](../../dot/digest/digest.go)) manages the verification of the +[digests](https://docs.substrate.io/v3/getting-started/glossary/#digest) that are present in block headers. + +#### Consensus + +The BABE and GRANDPA services work together to provide Gossamer with its +[hybrid consensus](https://wiki.polkadot.network/docs/learn-consensus#hybrid-consensus) capabilities. The term "hybrid +consensus" refers to the fact that block _production_ is decoupled from block _finalisation_. Block production is +handled by the BABE service, which is defined in [lib/babe/babe.go](../../lib/babe/babe.go); block finalisation is +handled by the GRANDPA service, which is defined in [lib/grandpa/grandpa.go](../../lib/grandpa/grandpa.go). + +#### Sync + +This service is concerned with keeping Gossamer in sync with a blockchain - it implements a "bootstrap" mode, to +download and verify blocks that are part of an existing chain's history, and a "tip-syncing" mode that manages the +multiple candidate forks that may exist at the head of a live chain. The sync service makes use of +[a block verification utility](../../lib/babe/verify.go) that implements BABE logic and is used by Gossamer to verify +blocks that were produced by other other nodes in the network. The sync service is defined in +[dot/sync/syncer.go](../../dot/sync/syncer.go). + +#### RPC + +This service, which is defined in [dot/rpc/service.go](../../dot/rpc/service.go), exposes a JSON-RPC interface that is +used by client applications like [Polkadot JS Apps UI](https://polkadot.js.org/apps/). The RPC interface is used to +interact with Gossamer to perform administrative tasks such as key management, as well as for interacting with the +runtime by querying storage and submitting transactions, and inspecting the chain's history. + +#### System + +The system service is defined in [dot/system/service.go](../../dot/system/service.go) and exposes metadata about the +Gossamer system, such as the names and versions of the protocols that it implements. + +#### Core + +As its name implies, the core service ([dot/core/service.go](../../dot/core/service.go)) encapsulates a range of +capabilities that are central to the functioning of a Gossamer node. In general, the core service is a type of +dispatcher that coordinates interactions between services, e.g. writing blocks to the database, reloading +[the runtime](#runtime) when its definition is updated, etc. ### Keystore @@ -115,7 +176,9 @@ In addition to the above-described services, Gossamer hosts a Wasm execution env upgradeable blockchain runtime. The runtime must be implemented in Wasm, and must expose an interface that is specified in [lib/runtime/interface.go](../../lib/runtime/interface.go). The runtime defines the blockchain's state transition function, and the various Gossamer services consume this capability in order to author blocks, as well as to verify -blocks that were authored by network peers. +blocks that were authored by network peers. The runtime is dependent on a +[Wasm host interface](https://docs.wasmer.io/integrations/examples/host-functions), which Gossamer implements and is +defined in [lib/runtime/wasmer/exports.go](../../lib/runtime/wasmer/exports.go). ### Monitoring From 7d2ff68152d06e1cfcecc93573850fe02b32ffa8 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Fri, 5 Nov 2021 07:10:09 -0700 Subject: [PATCH 4/6] Review comments --- cmd/gossamer/README.md | 46 +++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 21 deletions(-) diff --git a/cmd/gossamer/README.md b/cmd/gossamer/README.md index 193b4f5834..4277ee32a0 100644 --- a/cmd/gossamer/README.md +++ b/cmd/gossamer/README.md @@ -15,37 +15,37 @@ of the Gossamer CLI capabilities, follow [the installation instructions](../../R ### Default Command -This the default Gossamer execution method, which invokes the `gossamerAction` function defined in +This is the default Gossamer execution method, which invokes the `gossamerAction` function defined in [`main.go`](main.go) - it will launch a Gossamer blockchain client. The details of how Gossamer orchestrates a blockchain client are [described below in the Client Components section](#client-components). -- `basepath` - the path to the directory where Gossamer will store its data -- `chain` - specifies the [chain configuration](../../chain) that the +- `--basepath` - the path to the directory where Gossamer will store its data +- `--chain` - specifies the [chain configuration](../../chain) that the [Gossamer host node](https://chainsafe.github.io/gossamer/getting-started/overview/host-architecture/) should load -- `key` - specifies a test keyring account to use (e.g. `--key=alice`) -- `log` - supports levels `crit` (silent), `error`, `warn`, `info`, `debug`, and `trce` (detailed), default is `info` -- `name` - node name, as it will appear in, e.g., [telemetry](https://telemetry.polkadot.io/) +- `--key` - specifies a test keyring account to use (e.g. `--key=alice`) +- `--log` - supports levels `crit` (silent), `error`, `warn`, `info`, `debug`, and `trce` (detailed), default is `info` +- `--name` - node name, as it will appear in, e.g., [telemetry](https://telemetry.polkadot.io/) ### Init Subcommand This subcommand accepts a genesis configuration file and uses it to initialise the Gossamer node and its state. The `init` subcommand invokes the `initAction` function defined in [`main.go`](main.go). -- `genesis` - path to the "compiled" genesis configuration file that should be used to initialise the Gossamer node and - its state +- `--genesis` - path to the "compiled" genesis configuration file that should be used to initialise the Gossamer node + and its state ### Account Subcommand The `account` subcommand provides the user with capabilities related to generating and using `ed25519`, `secp256k1`, and `sr25519` [account keys](https://wiki.polkadot.network/docs/learn-keys), and managing the keys present in the -[Gossamer keystore](#keystore). The `actionAction` function is defined in [account.go](account.go); it is an interface +[Gossamer keystore](#keystore). The `accountAction` function is defined in [account.go](account.go); it is an interface to the capabilities defined in the [`lib/crypto`](../../lib/crypto) and [`lib/keystore`](../../lib/keystore) packages. This subcommand provides capabilities that are similar to [Parity's Subkey utility](https://docs.substrate.io/v3/tools/subkey). -- `generate` - creates a new key pair; specify `--ed25519`, `--secp256k1`, or `--sr25519` (default) -- `list` - lists the keys in the Gossamer keystore -- `password` - allows the user to provide a password to either encrypt a generated key or unlock the Gossamer keystore +- `--generate` - creates a new key pair; specify `--ed25519`, `--secp256k1`, or `--sr25519` (default) +- `--list` - lists the keys in the Gossamer keystore +- `--password` - allows the user to provide a password to either encrypt a generated key or unlock the Gossamer keystore ### Import Runtime Subcommand @@ -62,8 +62,9 @@ Gossamer node can consume. If the `--genesis` parameter is not provided, the gen represent the Gossamer default configuration. The `build-spec` subcommand invokes the `buildSpecAction` function defined in [`main.go`](main.go). -- `genesis` - path to the human-readable configuration file that should be compiled into a format that Gossamer can +- `--genesis` - path to the human-readable configuration file that should be compiled into a format that Gossamer can consume +- `--raw` - when this flag is present, the output will be a raw genesis spec described as a JSON document ### Import State Subcommand @@ -72,19 +73,21 @@ of a JSON file. The input for this subcommand can be retrieved from [the `state_getPairs` RPC endpoint](https://github.com/w3f/PSPs/blob/master/PSPs/drafts/psp-6.md#1114-state_getpairs). The `importStateAction` function is defined in [`main.go`](main.go). -- `first-slot` - the first [BABE](https://wiki.polkadot.network/docs/learn-consensus#block-production-babe) slot, which - can be found by checking the [digest](https://docs.substrate.io/v3/getting-started/glossary/#digest) for a chain's - first block _after_ its [genesis block](https://wiki.polkadot.network/docs/glossary#genesis) -- `header` - path to a JSON file that describes the block header corresponding to the given state -- `state` - path to a JSON file that contains the key-value pairs with which to seed Gossamer storage +- `--first-slot` - the first [BABE](https://wiki.polkadot.network/docs/learn-consensus#block-production-babe) slot, + which can be found by checking the + [BABE pre-runtime digest](https://crates.parity.io/sp_runtime/enum.DigestItem.html#variant.PreRuntime) for a chain's + first block _after_ its [genesis block](https://wiki.polkadot.network/docs/glossary#genesis) (e.g. + [Polkadot on Polkascan](https://polkascan.io/polkadot/log/1-0)) +- `--header` - path to a JSON file that describes the block header corresponding to the given state +- `--state` - path to a JSON file that contains the key-value pairs with which to seed Gossamer storage ### Export Subcommand The `export` subcommand transforms a genesis configuration and Gossamer state into a TOML configuration file. This subcommand invokes the `exportAction` function defined in [`export.go`](export.go). -- `config` - path to a TOML configuration file (e.g. those defined in [the `chain` directory](../../chain)) -- `basepath` - path to the Gossamer data directory that defines the state to export +- `--config` - path to a TOML configuration file (e.g. those defined in [the `chain` directory](../../chain)) +- `--basepath` - path to the Gossamer data directory that defines the state to export ## Client Components @@ -185,6 +188,7 @@ defined in [lib/runtime/wasmer/exports.go](../../lib/runtime/wasmer/exports.go). Gossamer publishes telemetry data and also includes an embedded Prometheus server that reports metrics. The metrics capabilities are defined in the [dot/metrics](../../dot/metrics) package and build on [the metrics library that is included with Go Ethereum](https://github.com/ethereum/go-ethereum/blob/master/metrics/README.md). -The Gossamer telemetry server publishes telemetry data that is compatible with +The default port for Prometheus metrics is 9090, and Gossamer allows the user to configure this parameter with the +`--metrics-port` command-line parameter. The Gossamer telemetry server publishes telemetry data that is compatible with [Polkadot Telemetry](https://github.com/paritytech/substrate-telemetry) and [its helpful UI](https://telemetry.polkadot.io/). From 872fde74e31aee8e0677c5f24f9a4ee487a616bd Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Fri, 5 Nov 2021 07:30:56 -0700 Subject: [PATCH 5/6] chore(ci): Fix PR check (again) (#1960) --- lib/utils/pull_request.go | 39 +++++++++++++++++++++++++--------- lib/utils/pull_request_test.go | 28 +++++++++++------------- 2 files changed, 41 insertions(+), 26 deletions(-) diff --git a/lib/utils/pull_request.go b/lib/utils/pull_request.go index f14f81c988..9050161ade 100644 --- a/lib/utils/pull_request.go +++ b/lib/utils/pull_request.go @@ -11,24 +11,43 @@ const ( bodyRegex = `## Changes.*- .*[A-Za-z0-9].*## Tests.*[A-Za-z].*## Issues.*- .*[A-Za-z0-9].*## Primary Reviewer.*- @.+[A-Za-z0-9].*` ) -// CheckPRDescription matches the PR title and body according to the PR template. +var ( + titleRegexp = regexp.MustCompile(`^[A-Za-z-_]+\([-_/A-Za-z ]+\):.+[A-Za-z]+.+$`) + commentRegexp = regexp.MustCompile(``) +) + +// CheckPRDescription verifies the PR title and body match the expected format. func CheckPRDescription(title, body string) error { match, err := regexp.MatchString(titleRegex, title) if err != nil || !match { return fmt.Errorf("title pattern is not valid: %w match %t", err, match) } - var bodyData string - // Remove comment from PR body. - for { - start := strings.Index(body, "") - if start < 0 || end < 0 { - break + body = commentRegexp.ReplaceAllString(body, "") + body = strings.ReplaceAll(body, "\r", "") + + // Required subheading sections in order + requiredSections := []string{"Changes", "Tests", "Issues", "Primary Reviewer"} + + previousIndex := -1 + previousSection := "" + for i, requiredSection := range requiredSections { + textToFind := "## " + requiredSection + if i > 0 { + // no new line required before the first section + textToFind = "\n" + textToFind } - bodyData = bodyData + body[:start] - body = body[end+4:] + index := strings.Index(body, textToFind) + if index == -1 { + body = strings.ReplaceAll(body, "\n", "\\n") // for error logs in one line + return fmt.Errorf("%w: %q in body: %s", ErrBodySectionNotFound, textToFind, body) + } else if i > 0 && index < previousIndex { + return fmt.Errorf("%w: section %q cannot be before section %q", + ErrBodySectionMisplaced, requiredSection, previousSection) + } + previousIndex = index + previousSection = requiredSection } bodyData = bodyData + body diff --git a/lib/utils/pull_request_test.go b/lib/utils/pull_request_test.go index 8cd7cf4cc5..be7baeee07 100644 --- a/lib/utils/pull_request_test.go +++ b/lib/utils/pull_request_test.go @@ -12,25 +12,21 @@ func Test_PR_Checks(t *testing.T) { body string valid bool }{ - { - title: "", - body: "", - valid: false, + "all empty": { + err: errors.New(`title pattern is not valid: for regular expression ^[A-Za-z-_]+\([-_/A-Za-z ]+\):.+[A-Za-z]+.+$: ''`), }, - { - title: "abc(abc): abc", - body: "", - valid: false, + "invalid title": { + title: "category: something", + err: errors.New(`title pattern is not valid: for regular expression ^[A-Za-z-_]+\([-_/A-Za-z ]+\):.+[A-Za-z]+.+$: 'category: something'`), }, - { - title: `feat(dot/rpc): implement chain_subscribeAllHeads RPC`, - body: `## Changes\n\n\n\n- changes for demo :123\n\n## Tests\n\n\n\n- tests for demo:123{}\n\n## Issues\n\n\n\n- issues for demo:43434\n\n## Primary Reviewer\n\n\n\n- @noot for demo:12`, - valid: true, + "empty body only": { + title: "category(subcategory): something", + err: errors.New("body section not found: \"## Changes\\n\" in body: "), }, - { - title: "abc(): abc", - body: "", - valid: false, + "invalid body": { + title: "category(subcategory): something", + body: "##Change\n## Tests ## Issues ## Primary Reviewer", + err: errors.New("body section not found: \"## Changes\\n\" in body: ##Change\\n## Tests ## Issues ## Primary Reviewer"), }, { title: "(abc): abc", From c39e68915a2405808790c22fff6ed8703407d251 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Fri, 5 Nov 2021 07:43:47 -0700 Subject: [PATCH 6/6] Standarize lists --- cmd/gossamer/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/cmd/gossamer/README.md b/cmd/gossamer/README.md index 4277ee32a0..f71ac24f2f 100644 --- a/cmd/gossamer/README.md +++ b/cmd/gossamer/README.md @@ -167,11 +167,11 @@ network participants use their private keys to sign messages in order to authori informing the Gossamer blockchain client capabilities, the Gossamer keystore is accessible by way of the `account` subcommand. The Gossamer keystore manages a number of key types, some of which are listed below: -- `babe` - This key is used for signing messages related to the BABE block production algorithm. -- `gran` - The GRANDPA key is used for participating in GRANDPA block finalisation. -- `imon` - The name of this key is a reference to "ImOnline", which is an +- `babe` - this key is used for signing messages related to the BABE block production algorithm +- `gran` - the GRANDPA key is used for participating in GRANDPA block finalisation +- `imon` - the name of this key is a reference to "ImOnline", which is an [online message](https://wiki.polkadot.network/docs/glossary#online-message) that Gossamer nodes use to report - liveliness. + liveliness ### Runtime