Overview

The Root Network is built on top of the open-source blockchain framework Substrate. We have adapted this with many enhancements known as pallets to deliver on our goals of driving better ux and more interoperability for open gaming. Substrate (now known as Polkadot SDK) is maintained by a large open source community lead by Parity Technologies.

The Root Network was initially created using an older Substrate version v0.9.27. The journey to v1.0.0 started a few months back. First, we upgraded the node to v0.9.30 covering 3 main releases with The Root Network release v4.41.0, followed by the largest jump we have ever seen in The Root Network’s history, v1.0.0, covering 14 major releases. 🤯

The Requirement

The Substrate framework is a highly active, fast-paced codebase where developers constantly introduce new features, bug fixes, and improvements. Being on an old Substrate version hinders the ability to introduce these new advancements to The Root Network. Hence we need to transition The Root Network to Substrate v1.0 smoothly, with minimal disruption to the network. And where’s the fun in living in the past without stepping into the future? Working with the cutting-edge technology is much more rewarding.

Benefits

The new upgrade release comes with many benefits.

• The Root Network will get all the new features, bug fixes, security patches, tooling, performance and efficiency improvements and new protocols developed by the upstream Substrate framework. For details refer to Pull Request #821.
• The Root Network will be prepared for much faster and streamlined future upgrades.
• This upgrades supports Shanghai EVM version, a framework the whole eco system will be able to benefit from.
• Update maximum block weight of 1 second, allowing more space within a The Root Network block, increase in transaction throughput.
• Gas benchmarking redone on the standard hardware. An optimized value of 15 million for the block gas limit, up from the previous value of 11 million.
• Open the door for the next stage of The Root Networks long term architecture.

Challenges

Upgrading to a newer Substrate version is a major milestone the team is excited about, but at the same time, many challenges need to be addressed and resolved, especially if we are sitting on an older codebase.

Substrate codebase is a highly active and very dynamic codebase. Sometimes the structures (structs, traits) in the codebase get completely removed or moved to other places between the updates. Other times even the fundamental concepts might get changed as a result of continuous advancements.

At The Root Network codebase level, we use and reuse the framework code and the structures to build new functionality or develop new structures for the custom requirements of The Root Network. One of the heaviest tasks of an upgrade is to make the code buildable with new dependencies. Sometimes an entire feature has to be rewritten and updated to accomplish this. During The Root Network upgrade, we faced a number of such challenges. Some of the major ones are listed below. For the full changeset refer Pull Request #821.

• A big refactor was done to the framework asset handling traits with “introduce holds and freezing into fungible traits” PR - Pull Request #12951. We had our custom AssetExt  pallet and multi asset handling custom code written on top of the old logic/concepts. But with the upgrade we had to rewrite/update most of that code.
• Frame v1 was deprecated with Pull Request #13705, but it was broken already with the removal of the block number in Pull Request #14437. The Root Network had two pallets that were on Frame v1 code, Ethy and Erc20-peg , out of which Ethy is a complex codebase serving many other pallets as well. We had to do a prior release to upgrade these two pallets to Frame v2 before proceed upgrading to Substrate v1.0.0.
• Changes to framework client-side code/services required modifications to The Root Network custom client-side protocols such as ethy-p2p which powers the decentralized bridging and other consensus activities.
• Migrations - Database migrations are one of the major headaches and a deciding factor when it comes to Substrate upgrade planning. When jumping across 14 major releases, one must be very cautious regarding the migrations. Missing an important migration could lead to a far more serious problem from corrupt data to completely bricking the network. Database migrations could be client-side or runtime. With this upgrade we did not find any client-side migrations, but many runtime db migrations. More about how we get around the migrations in the planning section.

Planning and Execution

Prioritizing Migrations and Data Loads

The first step of this journey was to undertake a feasibility study to gather and document all the details. In the first round each major release starting from v0.27.0 to v1.0.0 in both Substrate and Frontier repositories was analyzed. The main focus of this round of migrations was to identify any migrations and list them with as many as details possible.

Then shortlisting of the migrations to those relevant to The Root Network was done. We are mainly interested in migrations of the runtime pallets that are being used by The Root Network runtime, other runtime migrations can be excluded.

The next step is to check how heavy the migration data is. For runtime migrations, the migration would happen at the start of the next block mint once we set the new runtime code. It is important to notice that during this period the nodes have higher resource utilization and we have a limited time to apply the migrations before the next block mint. If the validator nodes are unable to apply the migrations on time, it will cause them to miss the slots and delay the chain. And the nodes will also be on high-stress conditions if that happens. Therefore it is important to get a sense of the load introduced by the amount of data that needed to be migrated over. In this step, both testnet Porcini and mainnet Root relevant data was analyzed and documented.

Any migration that can be excluded for any valid reason has to be removed to reduce the load. It is also important to note that multiple migrations within the same runtime pallet need sequencing so that it won’t cause any issues when the upgrade is underway. Additionally there were extra migrations to upgrade all the accounts in the network as a part of Pull 12951.  These migrations needed to be executed once we finish the runtime upgrade.

Updating The Root Network codebase

The next step is the single most difficult task of the whole upgrade, upgrading The Root Network codebase and making it buildable with new dependencies. Most of the task has to be done by a single person since the opportunity to parallelize this task is very slim. Hence it took a considerable amount of time to complete.

Testing, Testing and more Testing

Next comes the deployment of devnets. We created two devnets Sprout1, Sprout2 each for Porcini and Root with having the forked state of each network. The idea here is to run a new network on a snapshot of Porcini/Root data so that we can do the upgrade and check the status, do functional testing, and post-upgrade monitoring before we go for a testnet/mainnet release. Each important step was documented to ensure we would follow the same plan during testnet/mainnet rollout. Most of the peripheral services such as indexers, explorers, and relayers were pointed to the new devnets in order to achieve complete functional testing. Internal QAs, as well as downstream application teams, were also involved in testing on devnets.

Apart from functional testing on devnets, a separate testing effort was carried out for mixed runtime testing and load testing. Here the idea was to simulate the actual upgrade scenario on devnets. The Substrate upgrade consists of two main segments, A client deployment and a runtime upgrade afterward. The new client rollout can take up to hours depending on how fast it goes. During this time there will be situations like new clients running old runtime and old clients running old runtimes. Also once the runtime upgrade is done, there will be scenarios like new clients running new runtime and old clients running new runtime. Note that The Root Network is a decentralized blockchain and there is no way we can upgrade all the nodes in the network at the same time durations. The idea of this testing is to make sure that under any of these mixed scenarios client-side protocols such as babe, grandpa, and ethy can function nominally. This would make sure the liveliness of the network during and after the upgrading process.

Once the devnet testing was at a satisfactory level, the next step was to go for testnet release. Porcini was upgraded with the new release image and then upgraded to v8.56.0 which essentially includes the Substrate upgrade plus a few more improvements and optimizations at The Root Network codebase. Then more testing on Porcini was carried out.

Once we are satisfied with the Porcini it’s time for the grand finale, the mainnet upgrade, and post-testing and monitoring.

Go time

Substrate upgrade has been one of the major projects that went underway in The Root Network history, thanks to everyone who contributed including the Core Dev team, Devops, QAs, PMs and Management, downstream project Dev and QA teams, external developers and testers, marketing, and comms.

The Future

It is exciting times ahead. We upgraded The Root Network to the landmark Substrate v1.0.0 release. Now it’s time to move forward much faster with the new polkadot-sdk repository, Substrate has moved to this new repository after the v1.0.0 release. Keeping up to date with Substrate and Frontier and developing with the latest technology is going to be very rewarding.

Learn more about the features and custom pallets The Root Network has to offer here. To stay up-to-date with developments and join our growing community, follow us on X and join our Discord.