Original title: On Attestations, Block Propagation, and Timing Games
Written by: Nero_eth
Compiled by: Tia, Techub News
Nowadays, proposer timing games are very common, and many studies have analyzed this phenomenon.
This article will take you through the evolution of the proposer timing game and analyze its impact on witnesses. Through case studies of node operators from Lido, Coinbase, and Kiln, we will delve into the timing game of block proposals and its impact on Ethereum consensus.
As of August 2024, the block building market is largely outsourced, with approximately 90% of blocks being built by mevboost block builders. Of these, Titan Builder and Beaverbuild build approximately 80% of blocks.
Kiln was one of the main entities driving the timing game, delaying block proposals by 3-3.5 seconds within a single slot.
In the current mevboost environment, block propagation is primarily done through relayers. While proposers still propagate a block after receiving it from a relayer, relayers typically have better network connections and can therefore complete propagation faster. However, the timing is still controlled by the proposer, who can delay their "getHeader" call to game the timing.
This chart shows the evolution of the timing game. We can see that over time, the blocks proposed by Kiln validators are relatively lagging within the slot.
This has an impact on the network: blocks proposed by Kiln proposers have significantly higher missed/wrong block header votes.
Previous analysis showed that the longer the wait time, the higher the expected number of missed header votes ("80% of witnesses happen at 5 seconds into the slot"). Kiln proposes blocks very late, causing some witnesses to miss them and vote for the parent block instead. With approximately 32,000 validators assigned to each slot, this would result in a ~10% incorrect block vote rate.
Let’s look at the witness behavior of three large node operators and compare how they react to blocks proposed at different times. The following graph shows the distribution of correct and timely block header votes within slot seconds.
For early blocks, we observe that Lido and Coinbase exhibit a characteristic “U” shape in voting patterns, which may be due to different geographical locations or client software. In contrast, Kiln shows a clear peak that lags slightly behind the first peak of Coinbase and Lido. However, for later blocks, Kiln witnesses also show a “U” shaped pattern.
When the block first appears at 4 seconds in the slot (due to the P2P network, each node receives the block at a different time), the Lido witness witnesses up to 2 seconds earlier than the Kiln or Coinbase witnesses. This pattern does not necessarily indicate that Kiln is executing a "personal strategy". Instead, it may be attributed to different clients or different geographical locations.
Who influenced whom?
In the figure below, we compare the performance of node operators under different proposers. For example, the green part above y=1 shows that when Kiln is the proposer, Lido witnesses will miss block header votes more easily. However, when Lido is the proposer, Lido witnesses are the most timely in witnessing blocks. The dashed line 1 represents the average share of missed block header votes when all entities are proposers. The bars below 1 mean that the specific entity missed fewer block header votes when it was associated with the respective proposer compared to the average.
Notably, node operators perform best when processing blocks that they themselves proposed.
A quick summary of what we saw:
When other operators proposed blocks as proposers, most operators performed relatively stably.
Figment, Lido, Kraken, and EtherFi performed poorly when Kiln proposed blocks as the proposer.
Only Kiln and Binance performed better when Kiln was the proposer.
Kiln performs well as a witness. Early analysis shows that Kiln excels when it comes to high-performance validators. For more details on Kiln's witness performance, see this analysis.
But Kiln caused the pressure. Now we know that Kiln's proposed blocks caused pressure on other witnesses, but not on Kiln's witnesses.
At this point, it is difficult to explain the “how”. One possible explanation is that Kiln’s validators are highly centralized, co-located, or have very dense peer-to-peer connections. Another reason could be that validators connecting them via a custom peer-to-peer network/private network or through other additional communication layers coordinate their behavior. The latter is considered more centralized because it places more emphasis on economies of scale.
We can observe similar patterns when we look at the (correct and timely) witness times for Lido and Coinbase when they each proposed a block.
Interestingly, Kiln develops a “U” shaped distribution from 3.8 seconds to 6.1 seconds for their own late blocks, while Lido sees a peak at 4.2 seconds and Coinbase sees a plateau starting at 4 seconds in the slot and a small peak at 6 seconds.
Prevent your proposed blocks from being reorganized
Let's turn our attention to the reorganized blocks. From the perspective of a node operator, a strategy might be to never vote for a block that reorganizes itself. Simply put, "If the proposer is me, never vote the parent block as the block header."
In the following sections, I will use "local block" to represent "self-proposed block".
The following graph shows the percentage of witnesses that voted for the reorg block compared to the witnesses that voted for the parent block. The red portion shows the percentage of witnesses that voted for the reorg block.
Kiln exhibited unusual behavior. While most witnesses from node operators honestly voted for the correct block header instead of their own block, Kiln witnesses did not do so. More than 10% of Kiln witnesses tried to keep it on-chain by voting for their own block. If such a strategy was adopted, they could incur losses by voting for the wrong block header. However, these strategies are generally despised in the Ethereum community: "Don't game the consensus."
The chart uses 365 days of data. Therefore, if some complex strategies were implemented in the past year, the red portion would be smaller.
But how do we think about collaboration at other levels?
Regarding witness coordination, as a community we seem to have accepted the fact that validators running on the same node vote for the same checkpoints.
We probably don’t want to take any efforts across physical machine boundaries to improve collaboration between validators. This should be something that everyone can build. This collaboration could take different forms:
Level 1 - Fallback mechanism with static peer connections: Provide a central standby/backup node for multiple physical machines. This could also be a circuit breaker, some particularly fault-tolerant machines that act as a private relay of information. Setups with improved peer connections, private networks or similar could also fall into this category.
Level 2 - If-else rules: Hardcoded rules to wait longer in certain slots. Those will be installed on multiple physical machines, allowing them to "collaborate" based on predefined rules.
Level 3 — Botnet: There is a centralized oracle that communicates with all validators and provides checkpoints of votes and timestamps of when they should be published.
In my opinion, the latter two forms of collaboration (levels 2 and 3) are problematic and node operators should be held accountable. Finally, there may be a grey area regarding policies involving static peer connections and private networks.
Such a setup could potentially be used to run (malicious) policies such as:
Make sure to never vote on different checkpoints across multiple physical machines.
Make sure to never vote on a reorg's own proposed block.
Collaborate based on successive proposers (honest reorganization client (y/n)).
Review a party's testimony.
Do not vote for a certain party's blocks.
other.
When discussing collaboration, it is important to distinguish between two types:
Collaborative behavior occurs between validators running from the same physical machine.
Collaborative behavior stems from running the same modified client software or relying on the same centralized oracle.
A potential solution to combat complex collaborative validator behavior is EIP-7716: Anti-correlation Penalties, which proposes adjusting penalties based on correlation between validators.
