In this three-part series, we will reveal the technical achievements that can significantly improve data processing for applications running on the Internet Computer Protocol (ICP).
This upgrade is a milestone in the ICP roadmap for Stellarator, currently being promoted across the network. Stellarator is a breakthrough in on-chain data persistence, significantly enhancing the scalability and processing capabilities of data storage, bringing new opportunities for complex data-rich applications that were previously constrained by system limitations.
This advancement enables developers to build complex applications requiring large-scale data processing, bringing new levels of practical utility to blockchain technology.
In the second part of this series, Luc Bläser will walk us through the enhanced orthogonal persistence of Motoko. If you missed the first part about data persistence, you can find it here.
Simple, Safe, Fast: Enhanced Orthogonal Persistence of Motoko
Motoko introduces enhanced orthogonal persistence, a unique feature designed to ease the burden on programmers dealing with stable memory by providing a simple, safe, and fast upgrade mechanism.
Motoko has always been able to automatically retain program state during upgrades without any additional code to handle persistence. Unfortunately, the previous implementation of this feature could not scale to large or deeply nested data.
This functionality has significantly enhanced security, performance, and scalability, with the key innovation being avoiding state transfer to stable memory by simply retaining rather than erasing main memory.
The runtime system now ensures data consistency during upgrades in a very efficient manner, regardless of memory size. Finally, we have switched from 32-bit to 64-bit main memory, ultimately expanding to a large amount of persistent data.
Background
Container upgrades are a challenge on the Internet Computer, often involving considerable complexity, overhead, and some risk of data loss. Although the main memory of the container (also known as WebAssembly memory) is persistent between transactions, in the past it would be erased during upgrades, which is a rather artificial step since memory already has file backups on the Internet Computer.
The reason for this behavior is that mainstream programming language implementations did not consider persistence at design time: they rearranged memory structures in an uncontrolled manner during recompilation or runtime and could not simply use previous versions of memory to restore modified programs.
Therefore, programmers must explicitly use the stable memory API or special stable data structures to save data during upgrades, similar to traditional computer architectures that provide volatile main memory and persistent auxiliary storage, but at the cost of increased complexity in software development (think of object-relational mappers, etc.).
In contrast, with Motoko, we have complete control over the compiler and runtime system, allowing us to fully control the memory layout, which enables support for orthogonal persistence in Motoko and continued usage even during upgrades.
Thus, Motoko programmers can conveniently develop any data structure (first-order types) within standard language concepts without the need for explicit stable memory, dedicated stable data structures, or other database-like abstractions, as the runtime system automatically retains the necessary objects.
Previously, the Motoko runtime system achieved orthogonal persistence by serializing and deserializing persistent object graphs to stable memory, which led to serious scalability and performance issues because the serialization/deserialization process is very expensive and could even exceed instruction limits, ultimately hindering upgrades. These flaws made it impractical for larger applications, which is why we redesigned the orthogonal persistence support in Motoko.
Through changes to the Internet Computer and Motoko runtime system, we have achieved scalable upgrades without the expensive serialization and deserialization in auxiliary stable memory space; instead, we simply retain main memory persistence during the upgrade process, while extending the address space of orthogonal persistence to 64 bits, allowing for future expansion to the same capacity provided by 64-bit stable memory.
Enhanced Orthogonal Persistence
Our goal is to simplify software development on the Internet Computer and ease the burden on programmers dealing with stable memory. To this end, Motoko has been optimized for persistence on the Internet Computer while achieving simple, safe, and fast upgrades:
Simple: With orthogonal persistence (stable variables in Motoko), any first-order type that is transitively reachable will be automatically persisted during the upgrade process without the need for stable memory or stable data structures.
Safety: The runtime system strictly checks type compatibility during upgrades and supports various data changes through implicit migration. Any more complex migration can be achieved through custom code, which prevents any data corruption or misunderstanding at the memory level.
Speed: Upgrades have become extremely fast because the main memory is only retained during the upgrade, eliminating the need to copy to stable memory or copy from stable memory. The main memory has been expanded to 64 bits so that it can be extended in the future to be as large as the stable memory.
Design Overview
As a prerequisite for enhanced orthogonal persistence, the Internet Computer Protocol has been extended to support main memory retention across upgrades and 64-bit main memory based on the WebAssembly Memory64 proposal.
Through customized compiler and runtime system design, Motoko defines a compile-invariant memory layout where all objects are allocated on the dynamic heap and come with sufficient metadata, allowing new program versions to safely access the state of previous versions.
Therefore, passive data segments in WebAssembly have proven useful as they allow for delayed allocation of static program data (such as text) in the runtime system without retaining a static address range.
We will even retain the incremental garbage collector (GC) state during the upgrade process, meaning upgrades can happen at any time without waiting for the GC to finish running.
To ensure strict memory and type safety, the runtime system stores the types of the current program version and uses this information to check memory compatibility when attempting to upgrade to a new program version.
Since this check only depends on the number of types, not the number of objects, upgrades are extremely fast and can scale to arbitrary memory sizes. Certain data migrations (such as adding or removing stable variables, promoting types, adding variant options, etc.) are automatically supported, while any more complex migrations can be programmed manually while continuously protecting type safety.
Motoko implements automatic data migration from classic traditional persistence to enhanced orthogonal persistence. To allow for potential radical changes in memory layout in the future (which we expect to be rare), Motoko also includes an auxiliary persistence mechanism based on stable memory serialization and a graph replication algorithm, as well as an infinite deterministic time slicing that is not subject to any instruction limits of the Internet Computer Protocol.
You can find more detailed information about enhanced orthogonal persistence, its design, implementation, data migration scenarios, and performance evaluation in our recently published paper (Implementing Smarter Contract Upgrades with Orthogonal Persistence).
Production Promotion
As a first step, enhanced orthogonal persistence can be used as an optional feature and can be activated via the compiler option '--enhanced-orthogonal-persistence'. The corresponding parameters can be specified in dfx.json as follows:
Motoko will automatically migrate from the old 32-bit classic persistence to enhanced orthogonal persistence, but please note that the reverse operation of downgrading from EOP to 32-bit classic persistence is not supported.
In the current version, although we have switched to 64 bits, the Internet Computer only provides 4GB of main memory, which is a conservative measure for the first step. We follow a risk-averse release approach, starting small and gradually increasing capacity over time.
In fact, due to the switch to 64 bits and the initial limit of 4GB, the net memory usage in the initial phase is expected to be less than 32 bits (because pointer size has doubled), and the instruction cost of accessing 64-bit memory has also increased to conservatively cover hardware costs, which may decrease in the future.
But most importantly, upgrades will become very fast and no longer hit Internet Computer instruction limits, even at maximum heap utilization (considering that upgrade hooks are no longer needed).
Next, after gathering feedback and potentially improving the system and expanding memory capacity, we plan to make enhanced orthogonal persistence the default mode.
Conclusion
Motoko's enhanced orthogonal persistence allows Internet Computer developers to focus on their core applications without worrying about stable memory.
This not only enables simple and safe persistence but also significantly reduces the cost of upgrades and data access on the Internet Computer.
Enhanced orthogonal persistence can only be achieved through a custom compiler and runtime system, just like we currently have for Motoko.
More Information
Luc Bläser, Claudio Russo, Gabor Greif, Ryan Vandersmith, and Jason Ibrahim, Implementing Smarter Contract Upgrades with Orthogonal Persistence, VMIL 2024:
https://dl.acm.org/doi/10.1145/3689490.3690401
DFINITY Forum Post - Beta Testing Enhanced Orthogonal Persistence:
https://forum.dfinity.org/t/beta-testing-motoko-s-enhanced-orthogonal-persistence-eop/35787
Documentation - Enhanced Orthogonal Persistence of Motoko:
https://internetcomputer.org/docs/current/motoko/main/canister-maintenance/orthogonal-persistence/enhanced
Documentation - Stable Variables, Upgrades, and Data Migration in Motoko:
https://internetcomputer.org/docs/current/motoko/main/canister-maintenance/upgrades
We are pleased to receive your feedback, please share your thoughts in the DFINITY Developers X channel or on the Motoko GitHub repo.
Join us tomorrow for Part 3 of our Stellarator journey with Kamil Popielarz and Yvonne-Anne Pignolet to learn how to improve ingress message throughput.
IC content you care about
Technical Progress | Project Information | Global Activities
Follow the IC Binance Channel
Stay Updated
