Original title: A Letter To The Hyperliquid Core Team
Original article by Kam Benbrik, Researcher @Chorus One
Original translation: Ashley, BlockBeats
Editor's note: Node operator Chorus One described in detail the multiple problems of the Hyperliquid test network in an open letter on the X platform, including frequent node shutdowns, operational difficulties caused by closed-source code, and single-point failure risks caused by centralized APIs, and proposed a number of improvement suggestions to improve the transparency and decentralization of the chain. In response to these contents, Hyperliquid founder Jeff responded and emphasized that the validator selection criteria have been explained in the announcement; the Hyperliquid official account also published a separate article on the X platform to clarify the issues mentioned in the letter and stated that the node code will be open sourced under safe conditions.
The following is the original content (for readability, the original content has been reorganized):
The following letter is from Chorus One to the @HyperliquidX engineering team, hoping the team can take the time to review this feedback on Hyperliquid chain management.
TL;DR
· Due to closed-source code, lack of documentation, and reliance on a centralized API, validators face significant challenges, leading to frequent node shutdowns (jailing) and unstable performance.
· The testnet incentive mechanism has led to black market trading of HYPE tokens, favoring transactions with large holders instead of fairly selecting validators.
· Low returns for mainnet validators cannot cover high self-staking requirements, limiting decentralization as 81% of the stake is controlled by foundation nodes.
· To compete with major layer 1s, Hyperliquid must improve transparency, decentralize staking, implement a fair validator selection mechanism, and interact more with external validators.
I became acquainted with Hyperliquid in December 2023 and found this application to be excellent. It is easy to use, provides a great user experience, and Hyperliquid offers some unique features not found elsewhere, such as vaults and the well-known HLP. Currently, HLP manages over $350 million in funds and allows anyone to participate in Hyperliquid passively.
After seeing the excellent performance of the platform and learning that Hyperliquid operates independently as its layer 1, I hope Chorus One can participate as an operator on the Hyperliquid chain. I am an employee of Chorus One, one of the largest node operators in the industry. Chorus One has been active in the Proof of Stake industry since 2018. We have collaborated with many excellent teams, participated in the development of various blockchain designs and consensus algorithms, and played a key role in early Proof of Stake chains like Tezos and Cosmos Hub. Currently, Chorus One manages over 50 blockchains with a total staked asset value exceeding $3 billion and has collaborated with all major Proof of Stake blockchains since their early stages.
Chorus One joined the Hyperliquid testnet after being whitelisted on October 17. I would like to share our overall experience on the testnet with the Hyperliquid engineering team, as we still have not had a chance to interact with the team even after nearly 3 months of the testnet running. During this period, we witnessed one of the most successful token launches of 2024—the launch of the HYPE token. At the same time, we also experienced a testnet environment that was both interesting and challenging. I hope to mention some observed points that I hope will be taken into consideration in the coming days, weeks, or months.
Testnet experience.
The experience on the testnet has been very challenging so far. Node operators have almost no information on how to run nodes, with only limited resources available for reference.
Frequent node shutdown issues with unknown reasons.
Initially, we were shut down multiple times without understanding the reasons. Since the code is closed-source, we couldn't properly assess the reasons for the shutdowns. The only way was to communicate with other validators on Discord and jointly speculate on possible reasons. After communicating with several validators, we learned that others were also repeatedly shut down and were not entirely clear on the reasons.
Node location issues.
Later we found that the shutdown issue might be due to our nodes not being deployed in Tokyo. Moving the nodes to Tokyo might help. Unfortunately, the team never explicitly informed us of this, and we only discovered it after encountering multiple issues.
After migrating the nodes to Tokyo, the situation improved. This may be because many testnet nodes with significant stakes are also deployed in Tokyo, allowing our nodes to catch up slightly and reduce missed blocks. However, even after the migration, we still faced shutdown issues, and the specific reasons remain unclear. This lack of understanding is primarily due to the closed-source code.
Relying on automated shutdown lift scripts.
We realized that maintaining good uptime on the Hyperliquid testnet depends on the speed of scripts that automatically lift node shutdowns. The only way to improve uptime is to rely on fast automated shutdown lift scripts. Validators cannot fully understand or resolve potential issues and can only automatically lift node shutdowns without understanding the situation.
Centralized Hyperliquid API as a single point of failure.
There have been several times when our attempts to lift shutdowns have failed due to Hyperliquid's API outages. Since validators must send requests to the Hyperliquid server to lift shutdowns, they cannot independently lift shutdowns when the API is down.
The team may have realized this, but this design needs to be reconsidered as it makes the API a significant single point of failure for the network. If the goal is to build a Byzantine Fault Tolerant system, no node should have special privileges, such as relying on a centralized API.
Validator selection on the mainnet.
Hyperliquid recently selected about 16 validators in the process of decentralizing its validator set. The 4 validators previously managed by the core team have faced a lot of criticism. Hyperliquid has recently taken a significant step by expanding the validator set from 4 to 16.
Regarding the selection of validators, the 4 validators were announced via the following Discord post:
These validators are Validao, Harvest, Hypurrstake, and Prrposefulnode. These validators were selected based on maintaining over 90% uptime in the past 7 or 30 days.
This is a significant achievement in several ways, mainly because validator performance is also affected by external factors such as Hyperliquid API outages, shutdown issues, and ongoing binary crashes, all of which have a non-negligible impact on performance.
In addition to the 4 validators selected based on the testnet performance, 5 validators from the Hyperliquid Foundation are also operating on the mainnet. Additionally, another 7 validators have been selected to participate in the mainnet, but the reasons for their selection have not been disclosed.
Subsequently, a black market for HYPE testnet tokens began to emerge.
The Hyperliquid testnet originally had a set of 50 validators. Initially, specific entities were whitelisted to join the testnet, but since December 12, the validator set has been completely open.
The conditions are simple: you need to register 10,000 HYPE testnet tokens to become a validator. However, to become an active validator, you must also rank in the top 50; otherwise, the validator will remain inactive.
This decision led to a surge in the price of HYPE testnet tokens. The initial price rose to over 3,000 simulated USDC, and a few days later even exceeded 28,000 simulated USDC. As of the writing of this article, the price is about 700 simulated USDC per token.
Unfortunately, the faucet only distributes 100 simulated USDC every 4 hours. To become one of the top 50 validators on the testnet, currently more than 528,747 HYPE testnet tokens are needed. Assuming a price of 700 simulated USDC per token and relying solely on the faucet, the calculation is as follows:
Days = (528,747 × 700) ÷ (100 × 6) = 616,871.5 days.
This means that relying solely on the faucet would take approximately 616,871.5 days, or 1,690 years, to acquire enough HYPE testnet tokens to become an active validator on Hyperliquid.
However, those who received HYPE airdrops on the mainnet also received the same amount of tokens on the testnet. This provides validators an opportunity to collaborate with these community members, allowing validators to ensure entry into the active set by having them stake testnet HYPE tokens.
Meanwhile, this situation also provides another line of thought for those holding testnet HYPE tokens. Given the fierce competition to join the testnet validator set, many validators are eager to acquire as many HYPE testnet tokens as possible. Therefore, a black market emerged where whales holding large amounts of testnet HYPE tokens began selling their testnet tokens to validators in exchange for USDC on the mainnet.
I have never seen such a chaotic situation before. Although the Hyperliquid team clearly disapproves of these practices, they are fully capable of resolving the issue. A potential solution is to implement a proper testnet validator selection process.
In most other PoS networks, the core team typically shares a form that any validator can fill out to express their willingness to run a chain. The team then reviews these applications based on various criteria and makes initial selections, such as the validator's node operating experience, past contributions, community involvement, or other factors.
This group of initially selected validators could then participate in the testnet and work closely with the engineering team to provide feedback and ensure everything runs smoothly. We have made multiple attempts to provide feedback, but so far, we have not been successful.
Mainnet vs. Decentralization
As mentioned earlier, the current Hyperliquid mainnet validator set consists of 16 validators, which can be viewed at: https://app.hyperliquid.xyz/staking
· 5 validators are from the Hyperliquid Foundation.
· 4 validators were selected based on their performance on the testnet, maintaining over 90% uptime in the past 7 days.
· 7 validators were decided by the Hyperliquid team.
Of the 404,495,250 HYPE tokens staked, about 329,578,724 HYPE tokens are staked on foundation nodes, accounting for about 81.4% of the total staked amount. We know very little about HyperBFT, but assuming it operates as a Byzantine Fault Tolerant system, the core assumption of most BFT systems is that no more than 33% of the voting power behaves maliciously. If a single entity controls 1/3 of the stake, they can halt the chain. If they control 2/3 of the stake, they completely control the network.
The Hyperliquid Foundation initially staked 60 million HYPE tokens on each foundation node. However, many HYPE holders also chose to stake on foundation nodes, which is not ideal for decentralization. The team should engage more with the community to encourage a more decentralized staking distribution.
There are three potential solutions:
· Educate the community on the importance of staking with external validators to enhance the security and decentralization of the chain.
· Implement a 100% commission rate for foundation nodes to incentivize users to stake with external validators, promoting decentralization.
· Redistributing foundation staking to external validators is a practice taken by most chains.
Decentralizing staking to external validators will also help them become more economically sustainable. Hyperliquid is a blockchain focused on high throughput, and its infrastructure costs can be quite high, especially when nodes are deployed in Tokyo. Currently, validators at the bottom of the set earn between $3,000 and $5,000 per year, which is not enough to cover costs. This is particularly challenging as they must self-stake the initial 10,000 HYPE tokens (approximately $250,000 at current prices) to validate on the mainnet.
Currently, users interact with Hyperliquid by bridging USDC from Arbitrum to the Hyperliquid chain. By reviewing the contracts of the bridge, it seems that the bridge is still managed by the 4 validators. These validators do not appear to be associated with the chain's consensus or the 16 validators on the mainnet.
Hyperliquid has a great product, but the team still needs to improve several aspects of the infrastructure to truly compete with major layer 1s. Some improvements are quite simple, such as:
Listen to the opinions of experienced validators managing multiple networks. While the team's currently independent approach to building its perpetual product is very effective, validators are a pillar of a layer 1. Listening to their opinions is equally important to ensure everything runs smoothly.
Open-source code. This will help validators better understand the challenges they face when running nodes on Hyperliquid L1, while also helping users trust the product. Open-sourcing the code will also allow validators to better understand the architecture and consensus algorithms. Currently, information about HyperBFT is very limited, and open-sourcing can provide much-needed transparency and understanding. Chorus One has an important online manual about the significance of open-source. Operators should be able to build all the software they operate from the source code: https://handbook.chorus.one/node-software/open-source.html
Create a proper validator selection process to stop the black market trading of HYPE testnet tokens. Selecting validators based on uptime is a fair method, but achieving good uptime should also be fair. It should not depend on whether there are relationships to obtain testnet tokens, purchasing testnet tokens, or external factors (such as relying on Hyperliquid API uptime).
Overall, Hyperliquid does not need to make many changes to compete with major layer 1s. The main focus should be on interacting more with external parties and adopting their feedback. I look forward to seeing changes in the coming weeks and months, and our team is always available to provide help and feedback.
Hyperliquid founder Jeff and the official account responded.
In response to this letter, Hyperliquid founder Jeff made a statement on the X platform.
He emphasized that successfully running a validator is not difficult; the key lies in the validator's own setup and specialization. Additionally, he pointed out that the validator selection criteria have been stated in announcements, based on high uptime performance in the early stages of the testnet. This indicates that Jeff is more inclined to believe that the current issues stem more from the validators' own configurations rather than flaws in system design.
Additionally, Hyperliquid officials also released further clarification, stating that the node code will be open-sourced when it is secure.
· All validators are qualified based on testnet performance and cannot obtain seats through purchase; related false statements undermine the efforts of those who have invested time and energy to understand the system; as the blockchain matures, the validator set will gradually expand.
· As previously announced, a foundation delegation program will be launched to support high-performing validators and further decentralize the network.
· Anyone can run an API server pointing to any node; example client code sends requests to a specific API server, but this is not a fundamental requirement of the network.
· It is unacceptable for users to attempt to create a black market for testnet HYPE; this has been stated multiple times; we will continue to work on improving the onboarding process for the testnet.
· The node code is currently closed source; open sourcing is important, and the project will be open-sourced once it reaches a stable state; Hyperliquid's development speed is several orders of magnitude faster than most projects, and its scope is also several orders of magnitude larger than most projects; the code will be open-sourced when it is secure.
· Currently, there is only one binary file. Even a very mature network like Solana has most validators running a single client.