Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - Add validator-manager #3502

Closed
wants to merge 151 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
151 commits
Select commit Hold shift + click to select a range
9503e33
Add JSON deposit data to `create`
paulhauner Aug 15, 2022
5af83be
Add deposit data to recover function
paulhauner Aug 15, 2022
e3b2020
Revert "Add deposit data to recover function"
paulhauner Aug 15, 2022
ec33951
Add incomplete test script
paulhauner Aug 15, 2022
66e24e2
Add changes to test.py
paulhauner Aug 16, 2022
1984c6b
Add initial progress
paulhauner Aug 16, 2022
ff3a025
Add progress
paulhauner Aug 16, 2022
e25526e
Add progress
paulhauner Aug 16, 2022
e463518
Start refactoring into separate commands
paulhauner Aug 17, 2022
fec2969
Remove old command
paulhauner Aug 17, 2022
85d0843
Add function to check against beacon node
paulhauner Aug 17, 2022
ce7f29d
Appease clippy
paulhauner Aug 17, 2022
709598f
Link validator_manager into Lighthouse binary
paulhauner Aug 17, 2022
0cc72bc
Fix mistake in command name
paulhauner Aug 17, 2022
b025185
Avoid pulling env into blocking context
paulhauner Aug 17, 2022
3b324fc
Tidy logging
paulhauner Aug 17, 2022
7b62064
Fix deposit log creation
paulhauner Aug 17, 2022
c27d3dd
Refactor to use `Config` struct
paulhauner Aug 17, 2022
61e0347
Add testing for create command
paulhauner Aug 17, 2022
5edcfb7
Start working on test vectors
paulhauner Aug 17, 2022
79b5c6f
Add working test vector generator
paulhauner Aug 18, 2022
3bab10b
Pass test vectors
paulhauner Aug 18, 2022
ed0f354
Remove keystore json
paulhauner Aug 18, 2022
516e4ee
Add test vectors
paulhauner Aug 18, 2022
2873cb8
Tidy directory structure
paulhauner Aug 18, 2022
5e31754
Expose VC `ApiTester` via `test_utils`
paulhauner Aug 18, 2022
2677443
Fix visibility
paulhauner Aug 18, 2022
391ece9
Migrate VC tests to tokio::test
paulhauner Aug 21, 2022
fa6eba9
Start adding import validator testing
paulhauner Aug 21, 2022
7ed821b
Progress with testing
paulhauner Aug 22, 2022
52e50f5
Progress with testing
paulhauner Aug 22, 2022
3bdddb3
Fix warning
paulhauner Aug 22, 2022
1d4e7ba
Add DumpConfigs
paulhauner Aug 22, 2022
4886e27
Start adding CLI tests
paulhauner Aug 22, 2022
5b86fce
Add tests for validator create
paulhauner Aug 23, 2022
630cb91
Add import tests
paulhauner Aug 23, 2022
bcbafc0
Start adding validator move
paulhauner Aug 23, 2022
c0e9087
Add export endpoint to LH
paulhauner Aug 23, 2022
71409e4
Progress with move command
paulhauner Aug 23, 2022
e98a015
Progress with `move` implementation
paulhauner Aug 24, 2022
adc87f7
Add basic `move` testing
paulhauner Aug 24, 2022
92edb1b
Progress more with testing
paulhauner Aug 24, 2022
d20e65a
Progress with testing
paulhauner Aug 24, 2022
43b16b1
Fix clippy lints
paulhauner Aug 24, 2022
bd00a79
Remove commented-out code
paulhauner Aug 24, 2022
0f5092e
Fix clippy lint
paulhauner Aug 24, 2022
06fa52e
Improve tests
paulhauner Aug 24, 2022
65e0fe2
Fix failing tests
paulhauner Aug 24, 2022
bdffff1
Remove abandoned files
paulhauner Aug 24, 2022
1b40dd7
Revert changes to fork choice tests
paulhauner Aug 24, 2022
328a450
Revert abandoned changes
paulhauner Aug 24, 2022
40b1619
Don't run validator-manager tests in debug
paulhauner Aug 24, 2022
05bafee
Retry rather than saving file to disk
paulhauner Aug 24, 2022
0c32b5a
Remove intermediate file from `move`
paulhauner Aug 25, 2022
64decbc
Handle status from import
paulhauner Aug 25, 2022
0c786c0
Add tests for duplicates
paulhauner Aug 25, 2022
7f89f1e
Add `move` tests, "builder-proposals" takes bool
paulhauner Aug 25, 2022
e5476cd
Fix tests for windows
paulhauner Aug 25, 2022
89b2334
Use common version of regex across workspace
paulhauner Aug 25, 2022
e8da281
Revert "Use common version of regex across workspace"
paulhauner Aug 25, 2022
1f8f125
Add keystore export flag to VC
paulhauner Aug 25, 2022
be84637
Add count to `move`, fix tests
paulhauner Aug 25, 2022
2abde0b
Allow VC to create password files via HTTP API
paulhauner Aug 29, 2022
47eaa50
Shorten "validator-client" to "vc"
paulhauner Aug 29, 2022
d1aff84
Handle a missing command
paulhauner Aug 29, 2022
fdd0311
Strip newlines from VC token
paulhauner Aug 29, 2022
b2887f8
Ensure password dir exists in val dir builder
paulhauner Aug 29, 2022
4ccb7cb
Exit early on keystore upload error
paulhauner Aug 29, 2022
0719680
Remove naughty return Err
paulhauner Aug 29, 2022
da2a8ba
Start adding docs
paulhauner Aug 29, 2022
9539ca2
Add "done" message
paulhauner Aug 29, 2022
4d79002
Add dodgy progress to docs
paulhauner Aug 29, 2022
6af351a
Update docs
paulhauner Aug 30, 2022
9dfea42
Add more to docs
paulhauner Aug 30, 2022
2abb4a7
Add password migrate command
paulhauner Sep 5, 2022
c60b9c6
Revert "Add password migrate command"
paulhauner Sep 5, 2022
6569a5c
Remove comments from deposit `amount`
paulhauner Sep 5, 2022
dbe2fa4
Add comment about `amount` field
paulhauner Sep 5, 2022
958d1e4
Merge branch 'unstable' into validator-manager
paulhauner Oct 31, 2022
95c997b
Merge branch 'unstable' into validator-manager
paulhauner Oct 31, 2022
fb49408
Merge branch 'unstable' into validator-manager
paulhauner Nov 14, 2022
c63fece
Appease clippy
paulhauner Nov 14, 2022
94f97da
Merge branch 'unstable' into validator-manager
paulhauner Nov 30, 2022
53e492e
Fix Cargo.lock
paulhauner Nov 30, 2022
06f7efc
Fix clippy error
paulhauner Nov 30, 2022
0ece17e
Remove `validators` subcommand
paulhauner Nov 30, 2022
b5c463e
Fix cli tests
paulhauner Nov 30, 2022
af6affa
Fix compile errors
paulhauner Nov 30, 2022
0fe8958
Add test for moving validators back again
paulhauner Dec 1, 2022
86c3bcd
Add misc changes
paulhauner Dec 1, 2022
bbcf7b8
Revert "Add misc changes"
paulhauner Dec 5, 2022
6bef2be
Allow export without password
paulhauner Dec 14, 2022
0abb6a3
Read password from user in move command
paulhauner Dec 14, 2022
afed3fb
Add test for supplying password
paulhauner Dec 14, 2022
d096a07
Add more password tests
paulhauner Dec 14, 2022
aeb70ca
Merge branch 'unstable' into validator-manager
paulhauner Dec 14, 2022
8ca7f32
Add stdin-inputs flag to move command
paulhauner Dec 14, 2022
d278bab
Merge branch 'unstable' into validator-manager
paulhauner Feb 14, 2023
fb72975
Merge branch 'unstable' into validator-manager
paulhauner Feb 23, 2023
aafc587
Merge branch 'unstable' into validator-manager
paulhauner Mar 13, 2023
f27908a
Update docs as per removal of `validators`
paulhauner Mar 13, 2023
0de05e9
Merge branch 'unstable' into validator-manager
paulhauner Mar 24, 2023
9c76d45
Clarify help message
paulhauner Mar 27, 2023
1b225e9
Tidy message
paulhauner Mar 27, 2023
707d2a2
Further improve help message
paulhauner Mar 27, 2023
ee430c0
Merge branch 'unstable' into validator-manager
paulhauner May 15, 2023
fb51b09
Merge branch 'stable' into validator-manager
paulhauner Jul 10, 2023
ad65cf6
Fix clippy lints
paulhauner Jul 10, 2023
8d3aa44
Use async tests
paulhauner Jul 10, 2023
4af863a
Fix warnings
paulhauner Jul 10, 2023
fbb844d
Add `--count` flag as per @michaelsproul's comment
paulhauner Jul 11, 2023
66fd861
Update `KeyCache` after removing validator
paulhauner Jul 11, 2023
4c4f227
Check key cache consistency in tests
paulhauner Jul 11, 2023
c7ee1ea
Fix clippy lints
paulhauner Jul 11, 2023
98bcb94
Reduce loglevel for `UnknownPubkey` in the VC
paulhauner Jul 12, 2023
038c7a2
Update help text for move command
paulhauner Jul 12, 2023
856cd7e
Delete password files unless its being used elsewhere
paulhauner Jul 12, 2023
1ec051c
Update docs
paulhauner Jul 13, 2023
31dc520
Avoid using BLS credentials
paulhauner Jul 13, 2023
c3fe9d7
Add more docs
paulhauner Jul 13, 2023
b11ce55
Update error for missing fee recip
paulhauner Jul 13, 2023
e2a2e89
Add suggested fee recip
paulhauner Jul 13, 2023
2341f03
Update docs more
paulhauner Jul 13, 2023
b5027a0
Add docs to summary
paulhauner Jul 13, 2023
392624b
Tidy docs
paulhauner Jul 13, 2023
aa54777
Add new pages to summary
paulhauner Jul 13, 2023
6c9bea8
Fix failing test
paulhauner Jul 13, 2023
95de58e
De-duplicate some code
paulhauner Jul 16, 2023
745f9af
Fix comments
paulhauner Jul 16, 2023
2900b6c
Merge branch 'unstable' into validator-manager
paulhauner Jul 17, 2023
3d8d3de
Update Cargo.lock
paulhauner Jul 17, 2023
7b4b96d
Update validator_manager/src/move_validators.rs
paulhauner Jul 17, 2023
09bf5ab
Fix format string
paulhauner Jul 17, 2023
5c4b837
Update validator_manager/src/move_validators.rs
paulhauner Jul 18, 2023
d749888
Apply suggestions from code review
paulhauner Jul 18, 2023
5bc0f51
Add comment about omitting --beacon-node
paulhauner Jul 18, 2023
9ff1679
Remove references to dupe flag in move
paulhauner Jul 18, 2023
1766dbd
Apply suggestions from code review
paulhauner Jul 18, 2023
db88cd8
Fix clippy error
paulhauner Jul 19, 2023
fac1df7
Apply suggestions from code review
paulhauner Aug 3, 2023
c8317f2
Apply suggestions from code review
paulhauner Aug 3, 2023
a803529
Merge branch 'unstable' into validator-manager
paulhauner Aug 3, 2023
37c7003
Apply suggestions from code review
paulhauner Aug 3, 2023
1689ed6
Remove unencrypted flags
paulhauner Aug 7, 2023
3422225
Fix log message with swapped number
paulhauner Aug 7, 2023
ec9cf1c
Apply suggestions from code review
paulhauner Aug 7, 2023
e86c0e6
Apply suggestions from code review
paulhauner Aug 7, 2023
c07a554
Fix --validators help text
paulhauner Aug 7, 2023
88db492
Update "no validators present" log
paulhauner Aug 7, 2023
5654966
Add deprecation notice to "Key Management"
paulhauner Aug 7, 2023
a8865f9
Update book/src/key-management.md
paulhauner Aug 7, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Add JSON deposit data to create
  • Loading branch information
paulhauner committed Sep 22, 2022
commit 9503e338b560056ce07b5041182781cb6ef8f416
4 changes: 4 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions account_manager/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ safe_arith = {path = "../consensus/safe_arith"}
slot_clock = { path = "../common/slot_clock" }
filesystem = { path = "../common/filesystem" }
sensitive_url = { path = "../common/sensitive_url" }
serde = { version = "1.0.116", features = ["derive"] }
serde_json = "1.0.58"

[dev-dependencies]
tempfile = "3.1.0"
42 changes: 41 additions & 1 deletion account_manager/src/validator/create.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ pub const WALLET_NAME_FLAG: &str = "wallet-name";
pub const WALLET_PASSWORD_FLAG: &str = "wallet-password";
pub const DEPOSIT_GWEI_FLAG: &str = "deposit-gwei";
pub const STORE_WITHDRAW_FLAG: &str = "store-withdrawal-keystore";
pub const JSON_DEPOSIT_DATA_PATH: &str = "json-deposit-data-path";
pub const COUNT_FLAG: &str = "count";
pub const AT_MOST_FLAG: &str = "at-most";
pub const WALLET_PASSWORD_PROMPT: &str = "Enter your wallet's password:";
Expand Down Expand Up @@ -110,6 +111,17 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
.long(STDIN_INPUTS_FLAG)
.help("If present, read all user inputs from stdin instead of tty."),
)
.arg(
Arg::with_name(JSON_DEPOSIT_DATA_PATH)
.long(JSON_DEPOSIT_DATA_PATH)
.value_name("PATH")
.help(
"When provided, outputs a JSON file containing deposit data which \
is equivalent to the 'deposit-data-*.json' file used by the \
staking-deposit-cli tool.",
)
.takes_value(true),
)
}

pub fn cli_run<T: EthSpec>(
Expand Down Expand Up @@ -140,6 +152,9 @@ pub fn cli_run<T: EthSpec>(
let count: Option<usize> = clap_utils::parse_optional(matches, COUNT_FLAG)?;
let at_most: Option<usize> = clap_utils::parse_optional(matches, AT_MOST_FLAG)?;

let json_deposit_data_path: Option<PathBuf> =
clap_utils::parse_optional(matches, JSON_DEPOSIT_DATA_PATH)?;

// The command will always fail if the wallet dir does not exist.
if !wallet_base_dir.exists() {
return Err(format!(
Expand Down Expand Up @@ -212,6 +227,8 @@ pub fn cli_run<T: EthSpec>(
)
})?;

let mut json_deposit_data = Some(vec![]).filter(|_| json_deposit_data_path.is_some());

for i in 0..n {
let voting_password = random_password();
let withdrawal_password = random_password();
Expand Down Expand Up @@ -241,7 +258,7 @@ pub fn cli_run<T: EthSpec>(
)
})?;

ValidatorDirBuilder::new(validator_dir.clone())
let validator_dir = ValidatorDirBuilder::new(validator_dir.clone())
.password_dir(secrets_dir.clone())
.voting_keystore(keystores.voting, voting_password.as_bytes())
.withdrawal_keystore(keystores.withdrawal, withdrawal_password.as_bytes())
Expand All @@ -250,9 +267,32 @@ pub fn cli_run<T: EthSpec>(
.build()
.map_err(|e| format!("Unable to build validator directory: {:?}", e))?;

if let Some(json_deposit_data) = &mut json_deposit_data {
let standard_deposit_data_json = validator_dir
.standard_deposit_data_json(&spec)
.map_err(|e| format!("Unable to create standard JSON deposit data: {:?}", e))?;
json_deposit_data.push(standard_deposit_data_json);
}

println!("{}/{}\t{}", i + 1, n, voting_pubkey.as_hex_string());
}

// If configured, create a single JSON file which contains deposit data information for all
// validators.
if let Some(json_deposit_data_path) = json_deposit_data_path {
let json_deposit_data =
json_deposit_data.ok_or("Internal error: JSON deposit data is None")?;

let mut file = fs::OpenOptions::new()
.write(true)
.create_new(true)
.open(&json_deposit_data_path)
.map_err(|e| format!("Unable to create {:?}: {:?}", json_deposit_data_path, e))?;

serde_json::to_writer(&mut file, &json_deposit_data)
.map_err(|e| format!("Unable write JSON to {:?}: {:?}", json_deposit_data_path, e))?;
}

Ok(())
}

Expand Down
2 changes: 2 additions & 0 deletions common/validator_dir/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ tree_hash = "0.4.1"
hex = "0.4.2"
derivative = "2.1.1"
lockfile = { path = "../lockfile" }
serde = { version = "1.0.116", features = ["derive"] }
eth2_serde_utils = "0.1.1"

[dev-dependencies]
tempfile = "3.1.0"
65 changes: 64 additions & 1 deletion common/validator_dir/src/validator_dir.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,12 @@ use deposit_contract::decode_eth1_tx_data;
use derivative::Derivative;
use eth2_keystore::{Error as KeystoreError, Keystore, PlainText};
use lockfile::{Lockfile, LockfileError};
use serde::{Deserialize, Serialize};
use std::fs::{read, write, File};
use std::io;
use std::path::{Path, PathBuf};
use tree_hash::TreeHash;
use types::{DepositData, Hash256, Keypair};
use types::*;

/// The file used to save the Eth1 transaction hash from a deposit.
pub const ETH1_DEPOSIT_TX_HASH_FILE: &str = "eth1-deposit-tx-hash.txt";
Expand Down Expand Up @@ -41,6 +42,8 @@ pub enum Error {
Eth1DepositRootMismatch,
#[cfg(feature = "unencrypted_keys")]
SszKeypairError(String),
DepositDataMissing,
ConfigNameUnspecified,
}

/// Information required to submit a deposit to the Eth1 deposit contract.
Expand All @@ -54,6 +57,26 @@ pub struct Eth1DepositData {
pub root: Hash256,
}

/// The structure generated by the `staking-deposit-cli` which has become a quasi-standard for
/// browser-based deposit submission tools (e.g., the Ethereum Launchpad and Lido).
///
/// We assume this code as the canonical definition:
///
/// https://github.com/ethereum/staking-deposit-cli/blob/76ed78224fdfe3daca788d12442b3d1a37978296/staking_deposit/credentials.py#L131-L144
#[derive(Debug, PartialEq, Serialize, Deserialize)]
pub struct StandardDepositDataJson {
pub pubkey: PublicKeyBytes,
pub withdrawal_credentials: Hash256,
#[serde(with = "eth2_serde_utils::quoted_u64")]
pub amount: u64,
pub signature: SignatureBytes,
#[serde(with = "eth2_serde_utils::bytes_4_hex")]
pub fork_version: [u8; 4],
pub eth2_network_name: String,
pub deposit_message_root: Hash256,
pub deposit_data_root: Hash256,
}

/// Provides a wrapper around a directory containing validator information.
///
/// Holds a lockfile in `self.dir` to attempt to prevent concurrent access from multiple
Expand Down Expand Up @@ -203,6 +226,46 @@ impl ValidatorDir {
root,
}))
}

/// Calls `Self::eth1_deposit_data` and then builds a `StandardDepositDataJson` from the result.
///
/// The provided `spec` must match the value that was used to create the deposit, otherwise
/// an inconsistent result may be returned.
pub fn standard_deposit_data_json(
&self,
spec: &ChainSpec,
) -> Result<StandardDepositDataJson, Error> {
let deposit_data = self.eth1_deposit_data()?.ok_or(Error::DepositDataMissing)?;

let domain = spec.get_deposit_domain();
let deposit_message_root = deposit_data
.deposit_data
.as_deposit_message()
.signing_root(domain);

let deposit_data_root = deposit_data.deposit_data.tree_hash_root();

let DepositData {
pubkey,
withdrawal_credentials,
amount,
signature,
} = deposit_data.deposit_data;

Ok(StandardDepositDataJson {
pubkey,
withdrawal_credentials,
amount,
signature,
fork_version: spec.genesis_fork_version,
eth2_network_name: spec
.config_name
.clone()
.ok_or(Error::ConfigNameUnspecified)?,
deposit_message_root,
deposit_data_root,
})
}
}

/// Attempts to load and decrypt a Keypair given path to the keystore.
Expand Down