Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Withdrawals without queues #3068

Merged
merged 37 commits into from
Nov 10, 2022
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
d958ed7
Implement withdrawals without queues
potuz Oct 28, 2022
88f4938
remove unnecessary constants
potuz Oct 28, 2022
f506087
rebase on top of develop
potuz Oct 31, 2022
7dbd50e
Reviewers' comments
potuz Oct 31, 2022
49a2519
lint
potuz Oct 31, 2022
c156ea6
linting and typo
potuz Oct 31, 2022
0f74ab5
type annotation
potuz Oct 31, 2022
5b92eae
type annotation
potuz Oct 31, 2022
ff1dd90
fix toc
potuz Oct 31, 2022
7e4d169
update minimal preset
potuz Oct 31, 2022
a14479a
g11tech review
potuz Nov 1, 2022
39e6ec5
lint
potuz Nov 1, 2022
ad36548
Fix auto-rebase errors
hwwhww Nov 3, 2022
12404d0
fix for loop
potuz Nov 3, 2022
329bafa
dapplion's suggestions
potuz Nov 3, 2022
a09d617
right ret
potuz Nov 3, 2022
0b1f32e
Fix capella/block_processing tests
hwwhww Nov 3, 2022
494cefc
Fix sanity block tests
hwwhww Nov 3, 2022
b530dc0
Move old withdrawal epoch_processing tests to block_processing
hwwhww Nov 3, 2022
6e913ec
rename to latest_withdrawal_validator_index
potuz Nov 3, 2022
22f803a
name change and increment operator
potuz Nov 4, 2022
9973e9f
harden tests and add couple of cases
potuz Nov 4, 2022
e15b02d
lint
potuz Nov 4, 2022
8488fb7
Alex Stokes' review
potuz Nov 4, 2022
c8d1614
Apply suggestions from code review
ralexstokes Nov 4, 2022
3fc1ebc
Move some Capella operation tests to sanity/blocks tests
hwwhww Nov 7, 2022
dac756e
Minor clean up
hwwhww Nov 7, 2022
2f89f50
Danny's review 1st pass
potuz Nov 7, 2022
ac670e2
add phase0 sentence regarding process slots
potuz Nov 7, 2022
99e2704
make tests pass
potuz Nov 7, 2022
8f42e48
Add extra %
potuz Nov 8, 2022
710b124
fix last commit
potuz Nov 9, 2022
087f785
PR feedback from @djrtwo
hwwhww Nov 9, 2022
7f266bc
Use next_validator_withdrawal_index
potuz Nov 10, 2022
6179085
Merge branch 'withdrawals_without_queues' of github-potuz:potuz/conse…
potuz Nov 10, 2022
3d82a19
whitespace
potuz Nov 10, 2022
c7d7333
lint
potuz Nov 10, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 0 additions & 12 deletions presets/mainnet/capella.yaml
Original file line number Diff line number Diff line change
@@ -1,23 +1,11 @@
# Mainnet preset - Capella

# Misc
# ---------------------------------------------------------------
# 2**8 (= 256) withdrawals
MAX_PARTIAL_WITHDRAWALS_PER_EPOCH: 256


# State list lengths
# ---------------------------------------------------------------
# 2**40 (= 1,099,511,627,776) withdrawals
WITHDRAWAL_QUEUE_LIMIT: 1099511627776


# Max operations per block
# ---------------------------------------------------------------
# 2**4 (= 16)
MAX_BLS_TO_EXECUTION_CHANGES: 16


hwwhww marked this conversation as resolved.
Show resolved Hide resolved
# Execution
# ---------------------------------------------------------------
# 2**4 (= 16) withdrawals
Expand Down
16 changes: 2 additions & 14 deletions presets/minimal/capella.yaml
Original file line number Diff line number Diff line change
@@ -1,17 +1,5 @@
# Minimal preset - Capella

# Misc
# ---------------------------------------------------------------
# [customized] 16 for more interesting tests at low validator count
MAX_PARTIAL_WITHDRAWALS_PER_EPOCH: 16


# State list lengths
# ---------------------------------------------------------------
# 2**40 (= 1,099,511,627,776) withdrawals
WITHDRAWAL_QUEUE_LIMIT: 1099511627776


# Max operations per block
# ---------------------------------------------------------------
# 2**4 (= 16)
Expand All @@ -20,5 +8,5 @@ MAX_BLS_TO_EXECUTION_CHANGES: 16

# Execution
# ---------------------------------------------------------------
# [customized] Lower than MAX_PARTIAL_WITHDRAWALS_PER_EPOCH so not all processed in one block
MAX_WITHDRAWALS_PER_PAYLOAD: 8
# [customized] 2**2 (= 4)
MAX_WITHDRAWALS_PER_PAYLOAD: 4
159 changes: 46 additions & 113 deletions specs/capella/beacon-chain.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,8 @@
- [Constants](#constants)
- [Domain types](#domain-types)
- [Preset](#preset)
- [Misc](#misc)
- [State list lengths](#state-list-lengths)
- [Max operations per block](#max-operations-per-block)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests/core/pyspec/eth2spec/test/helpers/execution_payload.py

- [Execution](#execution)
- [Configuration](#configuration)
- [Containers](#containers)
- [New containers](#new-containers)
- [`Withdrawal`](#withdrawal)
Expand All @@ -27,17 +24,13 @@
- [`BeaconBlockBody`](#beaconblockbody)
- [`BeaconState`](#beaconstate)
- [Helpers](#helpers)
- [Beacon state mutators](#beacon-state-mutators)
- [`withdraw_balance`](#withdraw_balance)
- [Predicates](#predicates)
- [`has_eth1_withdrawal_credential`](#has_eth1_withdrawal_credential)
- [`is_fully_withdrawable_validator`](#is_fully_withdrawable_validator)
- [`is_partially_withdrawable_validator`](#is_partially_withdrawable_validator)
- [Beacon chain state transition function](#beacon-chain-state-transition-function)
- [Epoch processing](#epoch-processing)
- [Full withdrawals](#full-withdrawals)
- [Partial withdrawals](#partial-withdrawals)
- [Block processing](#block-processing)
- [New `get_expected_withdrawals`](#new-get_expected_withdrawals)
- [New `process_withdrawals`](#new-process_withdrawals)
- [Modified `process_execution_payload`](#modified-process_execution_payload)
- [Modified `process_operations`](#modified-process_operations)
Expand Down Expand Up @@ -75,18 +68,6 @@ We define the following Python custom types for type hinting and readability:

## Preset

### Misc

| Name | Value |
| - | - |
| `MAX_PARTIAL_WITHDRAWALS_PER_EPOCH` | `uint64(2**8)` (= 256) |

### State list lengths

| Name | Value | Unit |
| - | - | :-: |
| `WITHDRAWAL_QUEUE_LIMIT` | `uint64(2**40)` (= 1,099,511,627,776) | withdrawals enqueued in state |

### Max operations per block

| Name | Value |
Expand All @@ -99,8 +80,6 @@ We define the following Python custom types for type hinting and readability:
| - | - | - |
| `MAX_WITHDRAWALS_PER_PAYLOAD` | `uint64(2**4)` (= 16) | Maximum amount of withdrawals allowed in each payload |

## Configuration

## Containers

### New containers
Expand Down Expand Up @@ -241,32 +220,12 @@ class BeaconState(Container):
# Execution
latest_execution_payload_header: ExecutionPayloadHeader
# Withdrawals
withdrawal_queue: List[Withdrawal, WITHDRAWAL_QUEUE_LIMIT] # [New in Capella]
next_withdrawal_index: WithdrawalIndex # [New in Capella]
next_partial_withdrawal_validator_index: ValidatorIndex # [New in Capella]
latest_withdrawal_validator_index: ValidatorIndex # [New in Capella]
```

## Helpers

### Beacon state mutators

#### `withdraw_balance`

```python
def withdraw_balance(state: BeaconState, validator_index: ValidatorIndex, amount: Gwei) -> None:
# Decrease the validator's balance
decrease_balance(state, validator_index, amount)
# Create a corresponding withdrawal receipt
withdrawal = Withdrawal(
index=state.next_withdrawal_index,
validator_index=validator_index,
address=ExecutionAddress(state.validators[validator_index].withdrawal_credentials[12:]),
amount=amount,
)
state.next_withdrawal_index = WithdrawalIndex(state.next_withdrawal_index + 1)
state.withdrawal_queue.append(withdrawal)
```

### Predicates

#### `has_eth1_withdrawal_credential`
Expand Down Expand Up @@ -307,66 +266,6 @@ def is_partially_withdrawable_validator(validator: Validator, balance: Gwei) ->

## Beacon chain state transition function

### Epoch processing

```python
def process_epoch(state: BeaconState) -> None:
process_justification_and_finalization(state)
process_inactivity_updates(state)
process_rewards_and_penalties(state)
process_registry_updates(state)
process_slashings(state)
process_eth1_data_reset(state)
process_effective_balance_updates(state)
process_slashings_reset(state)
process_randao_mixes_reset(state)
process_historical_roots_update(state)
process_participation_flag_updates(state)
process_sync_committee_updates(state)
process_full_withdrawals(state) # [New in Capella]
process_partial_withdrawals(state) # [New in Capella]

```

#### Full withdrawals

*Note*: The function `process_full_withdrawals` is new.

```python
def process_full_withdrawals(state: BeaconState) -> None:
current_epoch = get_current_epoch(state)
for index in range(len(state.validators)):
balance = state.balances[index]
validator = state.validators[index]
if is_fully_withdrawable_validator(validator, balance, current_epoch):
withdraw_balance(state, ValidatorIndex(index), balance)
```

#### Partial withdrawals

*Note*: The function `process_partial_withdrawals` is new.

```python
def process_partial_withdrawals(state: BeaconState) -> None:
partial_withdrawals_count = 0
# Begin where we left off last time
validator_index = state.next_partial_withdrawal_validator_index
for _ in range(len(state.validators)):
balance = state.balances[validator_index]
validator = state.validators[validator_index]
if is_partially_withdrawable_validator(validator, balance):
withdraw_balance(state, validator_index, balance - MAX_EFFECTIVE_BALANCE)
partial_withdrawals_count += 1

# Iterate to next validator to check for partial withdrawal
validator_index = ValidatorIndex((validator_index + 1) % len(state.validators))
# Exit if performed maximum allowable withdrawals
if partial_withdrawals_count == MAX_PARTIAL_WITHDRAWALS_PER_EPOCH:
break

state.next_partial_withdrawal_validator_index = validator_index
```

### Block processing

```python
Expand All @@ -377,23 +276,57 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None:
process_execution_payload(state, block.body.execution_payload, EXECUTION_ENGINE) # [Modified in Capella]
process_randao(state, block.body)
process_eth1_data(state, block.body)
process_operations(state, block.body)
process_operations(state, block.body) # [Modified in Capella]
process_sync_aggregate(state, block.body.sync_aggregate)
```

#### New `get_expected_withdrawals`

```python
def get_expected_withdrawals(state: BeaconState) -> Sequence[Withdrawal]:
epoch = get_current_epoch(state)
withdrawal_index = state.next_withdrawal_index
validator_index = state.latest_withdrawal_validator_index
djrtwo marked this conversation as resolved.
Show resolved Hide resolved
withdrawals: List[Withdrawal] = []
for _ in range(len(state.validators)):
validator_index = ValidatorIndex((validator_index + 1) % len(state.validators))
validator = state.validators[validator_index]
balance = state.balances[validator_index]
if is_fully_withdrawable_validator(validator, balance, epoch):
withdrawals.append(Withdrawal(
index=withdrawal_index,
validator_index=validator_index,
address=ExecutionAddress(validator.withdrawal_credentials[12:]),
amount=balance,
))
withdrawal_index += WithdrawalIndex(1)
elif is_partially_withdrawable_validator(validator, balance):
withdrawals.append(Withdrawal(
index=withdrawal_index,
validator_index=validator_index,
address=ExecutionAddress(validator.withdrawal_credentials[12:]),
amount=balance - MAX_EFFECTIVE_BALANCE,
ralexstokes marked this conversation as resolved.
Show resolved Hide resolved
))
withdrawal_index += WithdrawalIndex(1)
if len(withdrawals) == MAX_WITHDRAWALS_PER_PAYLOAD:
break
return withdrawals
```

#### New `process_withdrawals`

```python
def process_withdrawals(state: BeaconState, payload: ExecutionPayload) -> None:
num_withdrawals = min(MAX_WITHDRAWALS_PER_PAYLOAD, len(state.withdrawal_queue))
dequeued_withdrawals = state.withdrawal_queue[:num_withdrawals]

assert len(dequeued_withdrawals) == len(payload.withdrawals)
for dequeued_withdrawal, withdrawal in zip(dequeued_withdrawals, payload.withdrawals):
assert dequeued_withdrawal == withdrawal

# Remove dequeued withdrawals from state
state.withdrawal_queue = state.withdrawal_queue[num_withdrawals:]
expected_withdrawals = get_expected_withdrawals(state)
assert len(payload.withdrawals) == len(expected_withdrawals)

for expected_withdrawal, withdrawal in zip(expected_withdrawals, payload.withdrawals):
assert withdrawal == expected_withdrawal
decrease_balance(state, withdrawal.validator_index, withdrawal.amount)
djrtwo marked this conversation as resolved.
Show resolved Hide resolved
if len(expected_withdrawals) > 0:
# `withdrawal` holds the last withdrawal object in the payload.
state.next_withdrawal_index = WithdrawalIndex(withdrawal.index + 1)
potuz marked this conversation as resolved.
Show resolved Hide resolved
state.latest_withdrawal_validator_index = withdrawal.validator_index
potuz marked this conversation as resolved.
Show resolved Hide resolved
```

#### Modified `process_execution_payload`
Expand Down
3 changes: 1 addition & 2 deletions specs/capella/fork.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,9 +129,8 @@ def upgrade_to_capella(pre: bellatrix.BeaconState) -> BeaconState:
# Execution-layer
latest_execution_payload_header=latest_execution_payload_header,
# Withdrawals
withdrawal_queue=[],
next_withdrawal_index=WithdrawalIndex(0),
next_partial_withdrawal_validator_index=ValidatorIndex(0),
latest_withdrawal_validator_index=ValidatorIndex(0),
ralexstokes marked this conversation as resolved.
Show resolved Hide resolved
)

return post
Expand Down
7 changes: 0 additions & 7 deletions specs/capella/validator.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,13 +58,6 @@ All validator responsibilities remain unchanged other than those noted below.
expected withdrawals for the slot must be gathered from the `state` (utilizing the
helper `get_expected_withdrawals`) and passed into the `ExecutionEngine` within `prepare_execution_payload`.
potuz marked this conversation as resolved.
Show resolved Hide resolved


```python
def get_expected_withdrawals(state: BeaconState) -> Sequence[Withdrawal]:
num_withdrawals = min(MAX_WITHDRAWALS_PER_PAYLOAD, len(state.withdrawal_queue))
return state.withdrawal_queue[:num_withdrawals]
```

*Note*: The only change made to `prepare_execution_payload` is to call
`get_expected_withdrawals()` to set the new `withdrawals` field of `PayloadAttributes`.

Expand Down
Loading