[Feature] Agent Prometheus Endpoint

[yourbuddyconner]

There has been a request that we implement a Prometheus endpoint in the Agents, this is a thread to enumerate the metrics we should include as well as implementation details like crate dependencies.

[ajcrowe]

Some initial thoughts, some of these will likely be easier to implement than others. I may have missed some but these are what come to mind on a first pass

# gauge for upstream RPC blockheight to check data is current and node in sync/connected
optics_blockheight{rpc_network}
# counter for updater number of signed attestations
optics_attestation_count{rpc_network, component}
# histogram for rpc calls measuring latency, count etc
optics_rpc_duration_ms{rpc_network, rpc_method (maybe)}
# counter for messages received and validated
optics_messages_count{home_network, replica_network}
# counter for transactions issued by wallets within component
optics_transaction_count{wallet_network, address}
# gauge for wallet balances for each network
optics_wallet_balance{wallet_network}

[prestwich]

• blockheight and wallet balances seem very easy
• transaction count could mean "txns from agent boot" or "current nonce for the wallet". which would you look for?
• same for message and attestation count. do we want "from agent boot" or "total"
• rpc latency can be inserted at the Homes and Replicas enum easily

most of these seem pretty doable as long as we have a once_cell for whatever metrics collection

[yourbuddyconner]

"from boot or total"

Probably from boot, but if the agents are rebooting a lot (as they are designed to do when they encounter errors) this might be problematic. Might be prudent to make a "total" counter and a "from boot" counter to help with this.

[prestwich]

we can do both. the total will likely be polled from chain

[ajcrowe]

yeah generally from boot is ok, you're looking for the activity more than the count. But more is definitely better. How would that work with shared wallets across components, they'd just report the same I guess

[yourbuddyconner]

Prometheus Crate: https://docs.rs/prometheus/0.12.0/prometheus/

Helpful Links/Examples:
Example: https://gist.github.com/breeswish/bb10bccd13a7fe332ef534ff0306ceb5
Blog Post: https://blog.logrocket.com/using-prometheus-metrics-in-a-rust-web-service/

[prestwich]

here's the implementation sketch

1. Add a metrics mod to optics-base
2. metrics mod contains once_cell for a CoreMetrics struct
3. CoreMetrics contains a selection of metrics and exposes a gather
   • tx counts
   • wallet balance
   • blockheight
   • wallet address
4. Define an AgentMetrics in each agent with once_cell in a metrics.rs
   • macroify (see decl_settings macro)
5. AgentsMetrics exposes agent-specific metrics and exposes a gather
   • e.g. updater attestations signed and total
   • e.g. current next_to_inspect for processor

[emberian]

Metric numbers are uint64 or f64, according to the protobuf spec, wallet addresses are OOB.

[emberian]

metric naming

Using prometheus terminology, metrics have a standard format of namespace_subsystem_name_units. optics will be the namespace. Should the agents have their own metrics, making subsystem an agent name, or should the particular agent instead be a label? Labels are a way of slicing up datapoints recorded under a single metric along arbitrary dimensions, specified in the opentsdb format inside braces, such as optics_updater_wallet_balance_total vs optics_wallet_balance_total{agent="updater"}.

[yourbuddyconner]

optics_wallet_balance_total{agent="updater"} This is the right way to do it for sure

[emberian]

@yourbuddyconner similar for blockheight etc?

[yourbuddyconner]

Yes indeed, keep the same pattern for all the metrics.

[emberian]

datatype compatibility

prometheus only supports double and uint64, excluding in full generality wallet balance from being a metric datapoint. should we truncate to 64 bits? convert to float?

[yourbuddyconner]

I think we should convert currency representation to whole ETH as opposed to 10e+22, more human readable that way which is what we want in metrics, despite some potential loss in specific digits.

[yourbuddyconner]

@prestwich could you elaborate how you see the wallet_balance gauge working? Specifically speccing out what parts of ethers we'd use to get this functionality. This is the major feature I am looking forward to currently.

[emberian]

from my mild investigation so far i think report_tx! macro callsites are where balance mutations occur, perhaps those?

[yourbuddyconner]

New Metrics:

• Contract State, @prestwich to elaborate