The Internet Computer blockchain relies on a peer-to-peer (P2P) protocol that distributes messages (artifacts) between nodes in each subnet, which is a set of protocols that run the Internet Computer blockchain, including the Internet Computer consensus protocol, the DKG (distributed key generation) protocol, or the state synchronization protocol.
Each such protocol generates artifacts and requires the P2P layer to distribute these artifacts to peers in the subnet. We denote each of these protocols as a P2P client, which is implemented as a higher-level component above the P2P layer.
Following the introduction of a separate P2P layer for state synchronization, a new P2P layer is being introduced for all other P2P clients in the Internet Computer protocol stack. This new P2P layer provides improved network performance, ensures the guarantees required by the consensus protocol, and makes it easier to detect misbehavior of nodes.
The newly proposed P2P layer uses the recently launched new transport layer based on QUIC. Therefore, with the migration to this new layer, the P2P layer of the Internet Computer will completely stop using TCP. The transition to QUIC also means a transition to a fully asynchronous implementation of the P2P layer.
Each request is sent as a new QUIC stream and processed independently of other requests, which prevents potential head-of-line blocking issues that, at least in theory, could cause liveness issues.
The new P2P layer introduces the use of a novel abstract data structure (called the slot table), which makes it easier to control the distribution of artifacts to peers while maintaining a sufficient send rate to each peer based on the connection quality and without affecting other peers. It also makes it easier to detect misbehavior of peers.
As the proposed new P2P layer is accepted, each client will use a separate instance of the P2P protocol, where state synchronization will use an instance designed specifically for it and the rest of the clients will use separate instances of the new P2P layer as described in detail in this post.
The rollout of the new proposed P2P layer begins with the adoption of a series of NNS proposals that transition the HTTPS-outcalls artifact distribution to use the new P2P layer, after which all remaining clients will be migrated, including the consensus protocol, eventually making the old P2P layer obsolete.
Background: The P2P Layer of the Internet Computer
At a very high level, the P2P protocol is responsible for distributing any artifacts present in each client's verified artifact pool to peers in the same subnet. The verified artifact pool is populated by clients based on their need to broadcast those artifacts to their peers.
Figure 2 shows the interface between P2P and the client on top of it, where the client can arbitrarily change the artifact pool every time it calls on_state_change().
Each such call ultimately returns a set of ChangeActions corresponding to the addition and removal of artifacts from that client's pool of verified artifacts, and the P2P layer should use this information to propagate the contents of the verified pool to peers.
The Internet Computer's existing P2P protocol is based on the flow of artifacts from each node to its peers. Whenever an artifact is added to the verified pool, an advertisement is broadcast to all peers. An advertisement is a small message describing the artifact.
It is sent instead of the artifact itself as a bandwidth saving measure so that the receiver can decide whether to download the (potentially large) artifact. The sending node maintains a TCP stream to each peer on which it sends the advertisement (and later the artifact upon request).
The P2P layer should guarantee the delivery of artifacts between honest and operational nodes and should be resilient to malicious behavior from potentially malicious nodes.
All consensus-related clients at the P2P layer have two important properties:
P1 - Finite number of active artifacts: The pool of validated artifacts is of finite size at any point in time, and the consensus protocol uses checkpoints to periodically purge artifacts, so the maximum size C of the pool can be calculated as a function of the checkpoint interval (measured in blocks) and the subnet size.
P2 - Explicit expiration of artifacts: if an artifact is removed from the pool (purged), it no longer needs to be propagated to peers, or, from the receiver's point of view, if no peer has a certain artifact, the receiver is guaranteed not to need that artifact, even if it failed to receive it when it was removed by all other peers.
These two properties are emphasized here because they support important design decisions that will be explained shortly.
Network backpressure
In traditional client-server applications, the concept of backpressure is widely used: if the receiver slows down the rate at which it consumes messages, the sender's buffer fills up, and then the sender's network layer must take one of three paths:
Propagate backpressure to the application layer, causing the application to slow down data production
Buffer messages (potentially indefinitely)
Discard egress message
On a blockchain this gets even more complicated. Imagine the sender experiences backpressure from a peer who may or may not be honest. Taking any of the above approaches will lead to serious problems:
Slowing down data production means slowing down the blockchain, and allowing this behavior will become a Denial of Service (DoS) attack vector;
The buffer (which could be indefinite) would also be an attack vector;
Dropping egress messages may mean that message delivery to honest but slow nodes cannot be guaranteed.
Due to the security risks of options 1 and 2, most blockchains chose option 3. However, option 3 carries a risk of blockchain liveness (i.e., if enough messages are dropped, it can get stuck). The new P2P layer proposed for the Internet Computer overcomes this risk without losing messages.
New P2P Layer
The new P2P layer works very differently from the existing ones. First, it does not always use advertisements. If the artifacts are small enough, they will be sent immediately without any advertisements. Second, it does not use a single stream, but multiple streams over the same QUIC connection. Third, because it does not use a single stream, it manages the sending of artifacts in a very different way. Fourth, it introduces a slightly different protocol for communication between peers in the same subnet.
Let’s take a step back and look at the purpose of the P2P layer when serving consensus protocols and other clients with similar needs (i.e. non-state synchronization), the purpose is to, for each honest node, ensure that peers can receive anything that node has in its pool of validated artifacts, while of course keeping everything secure, scalable, and performant.
The new P2P layer achieves this by introducing a novel abstract data structure called a slot table, which is used to track the contents of the pool of verified artifacts and the process of updating peers about that content. The slot table data structure is simple, but it provides exactly what is needed to satisfy the requirement.
Slot table data structure
The slot table is an abstract data structure maintained by every node on the sending side and inferred by every node on the receiving side. The size of the slot table on the sending side corresponds exactly to the number of active artifacts in the verification pool. If you remember property P1 above, this means that the slot table is bounded by some constant C.
Whenever an artifact is added to the validation pool, it is added to an empty slot in the slot table on the sending side. Slot Update messages are being sent to all peers, telling them that the content of the slot has changed. On the receiving side, each peer keeps track of the state of its slot table for each peer based on the arrival of new Slot Update messages.
Note, however, that network congestion and backpressure may cause these updates to be delayed, so that the receiver's view is only eventually consistent with the sender's slot table.
In addition to the artifact information, each slot also has a version number that is globally incremented with each update of the validation pool so that the receiver can know whether it is a new update or an old update when it receives an update message by only accepting updates with version numbers higher than its version number.
Figure 3 shows an example of this process: the sender generates artifacts A through F, it also deletes some of these artifacts in the process, since deletes are not necessarily propagated to peers, they may still have the deleted artifacts in their view of the peer’s slot table, which is correct because the contents of the slots are eventually updated and propagated.
For each slot on the sending side, a new set of asynchronous tasks (i.e. green threads) are generated - one task per slot per peer. Since tasks are very lightweight, they can also scale for larger subnets. Each task is responsible for reliably pushing slot update messages to the corresponding slot and the corresponding peer.
This means that the task will retry pushing updates until it receives an acknowledgement, and whenever the content of a slot changes, the task will stop trying to push the old content and start trying to push the new content instead. Slow peers may be slow to get updates, but they will not interfere with faster peers.
This approach is a bit like a combination of methods 2 and 3 in the back pressure discussion above (buffering messages at the network layer and discarding them), which solves the back pressure problem without giving up resilience and liveness.
The correctness of this approach stems from the previously mentioned client-side property P1, namely the finite number of active artifacts, which guarantees that C slots are sufficient in any case, so slots can be reused.
The above protocol not only allows peers to synchronize the contents of their pool of verified artifacts, but also allows a node to ensure that no more than C artifacts are advertised by its peers at a time. If the slot number of an update message is greater than C, the receiver can immediately infer that the sender is misbehaving.
When a node on the receiving end notices that an artifact is no longer present in the slot table of any of its peers, the node can safely remove the artifact from the pool of unverified artifacts (if it still exists), or stop any attempts to retrieve this artifact if it has not already been retrieved.
Property P2: The aforementioned explicit expiration of artifacts guarantees that no peer needs such an artifact anymore, so the slot table also imposes an implicit limit on the size of the pool of unverified artifacts: the unverified pool can contain at most C artifacts from honest peers (since they should have roughly the same content in the verified pool), and at most C artifacts per malicious peer, since it may spam the IC with C completely different artifacts.
Less than 1/3 of the nodes can be malicious, so the total size of the unverified pool cannot exceed C*4*n/3 artifacts.
You may have noticed that the above design description only refers to artifacts and does not mention advertisements. The reason is that advertisements are just an optimization to improve bandwidth utilization, and the new P2P layer only uses advertisements for large artifacts (the current threshold is set to 1KB).
Artifacts smaller than the threshold are sent in the Slot Update message and therefore do not have to be explicitly requested later by the receiver. For artifacts larger than the threshold, an advertisement is generated and sent in an Update message. The receiver client can then decide whether to request it and from which peer.
Improve performance
Using QUIC's asynchronous implementation and direct push of smaller artifacts improves network performance and therefore consensus performance.
Figure 4 shows the results of an experiment we conducted to compare the performance of the existing P2P layer and the new P2P layer, in which we ran a 60-node subnet in a single datacenter at a load of 200 x 100 KB requests per second and measured the consensus block rate.
The graphs show the block rate over time with a line for each node. We artificially added a link delay of 80 ms round trip to all network links in both experiments to simulate geographically distributed subnets.
The above graph shows the block rate of the existing P2P layer. While the subnet is making progress successfully, the block rate is very unstable when the load is high. Once the load ends, it returns to normal, but during the high load period, this means that the block rate is lower across the subnet, and therefore the user-perceived latency is higher.
The following figure shows the robustness of the block rate of the newly proposed P2P layer. Even under heavy load, the subnet still operates at a very stable block rate.
in conclusion
New P2P layers for consensus and similar clients increase the performance of the Internet Computer, enable better scalability, reduce code complexity, and improve behavior under imperfect network conditions.
It is already enabled on certain subnets, only for artifacts related to HTTPS callouts, and the DFINITY Foundation will soon submit a proposal to enable it for other clients.
If accepted and implemented, the entire P2P layer of the Internet Computer will move to using QUIC instead of TCP, making the Internet Computer's network layer more robust, scalable, and performant.
Learn more about the Internet Computer:
IC content you care about
Technology Progress | Project Information | Global Activities
Collect and follow IC Binance Channel
Get the latest news
