On June 2nd all Neo 2 testnet consensus nodes were upgraded with “neox-preview1” release of the C# node that brought with it some new features for cross-chain interoperability.
We at NSPCC had also changed our node to run this new version and to support the community rolling out new functionality to the network, but behind the scenes, the work was already underway to bring these new extensions to NeoGo. June 4th NeoSPCC’s consensus node on testnet had been replaced with NeoGo again and we never went back since then. Of course, there were some updates and bug fixes and it wasn’t a stable release yet, but it worked, packing transactions into blocks and serving the network.
But a stable release was needed and at first, we wanted to push some 0.75.1-neox-preview1 version from a separate code branch that would be compatible with version 2.10.3-neox-preview1 of C# node. Then suddenly a better idea appeared and we have released now a whole new 0.76.0 from our regular master-2.x branch. This version is fully compatible both with 2.10.3 and 2.10.3-neox-preview1 versions of C# node so that there is no need to use different node builds for different networks with NeoGo, one binary fits all and all NeoX features are easily configurable with just two options.
But let’s first delve into what this NeoX is all about.
What is NeoX
NeoX is an extension of the original Neo 2 node implemented in neox-2.x branch of C# implementation. It includes the following main changes:
- local state root generation for contract storages based on MPT
- consensus updates for state root exchange between CNs and generation of verified (signed by CNs) state root
- P2P protocol updates for state root distribution
- RPC protocol updates for state status data and proofs generation
- two new key recovery syscalls for smart contracts
Most of these changes are pure extensions to Neo 2 protocol, but consensus changes are incompatible with regular Neo 2 nodes. The idea is that we have now some state reference for each block that can be used by other chains (along with proof paths for individual key-value pairs if needed) and at the same time we’re able to check non-Neo signatures using new key recovery functionality that is available for two curves: Secp256r1 and Secp256k1.
How the local state is being generated and what it covers
Any full node processing blocks can now generate state root information locally using Merkle Patricia Trie (MPT). It’s used for any key-value pairs stored in the database with a prefix of ST_Storage which is used for contracts data storage. Basically, anything contracts save using Neo.Storage.Put syscall gets accounted for.
Each value gets a leaf node in MPT and the key for that value is encoded in branch and extension nodes according to prefix data. Any node in MPT can be hashed and the root node hash naturally depends on every other hash in the trie, so this single hash value represents the current state of the trie and is called state root hash. Any change to the trie state (adding/deleting/changing key-value pairs) changes state root hash.
But even though this state root data can be computed at every full node it can’t be considered authoritative until it’s signed by network-trusted entities which are consensus nodes.
How and why the consensus process was changed in NeoX
Consensus nodes now exchange state root information with PrepareRequest messages, so the Primary node tells everyone its current state root hash (along with the block index that state root corresponds to) and the hash of the previous state root message. This data might also be versioned in case of future updates, so there is a special field reserved for that too, but at the moment it’s always 0. Backups either confirm this data (if it matches their local state) by proceeding with PrepareResponse or request a ChangeView if there is some mismatch detected.
If all goes well CNs generate a signature for this state root data and exchange it with their Commit messages (along with new block signatures). Effectively this creates another signed chain on the network that is always one block behind from the main chain because the process of block N creation confirms the state resulting from processing of block N-1. A separate stateroot message is generated and sent along with the new block broadcast.
How P2P protocol was changed
P2P protocol was extended with getroots, roots and stateroot messages for state root data exchange. Simple stateroot message is what consensus nodes generate to broadcast signed state root data, it’s accepted by all nodes, they check it, verify its signature and save locally (to do that they have to have confirmed state root for the previous block). It’s somewhat similar to block announcement, but as this message is rather small, inv is not being used.
But this message might get lost or some new node may join the network and want to get verification for its state, so there has to be some possibility for state root requests and replies and that’s what getroots/roots pair is for. In general, it’s expected that the node would synchronize state roots the same way it synchronizes blocks, always trying to be up to date with the network. From this synchronization comes the concept of “state height” which represents the latest verified state root known to the node.
How RPC protocol was changed
RPC got extended with four new methods: getproof, getstateheight, getstateroot and verifyproof. getstateheight and getstateroot are easy, the first one allows to get current node’s block and state heights, while the second one returns state root data for the specified (by index or by hash) block. State root data basically mirrors the one exchanged via P2P protocol (version, previous state root message hash, and current state root hash), but also contains an additional flag to specify if the node has a verification (signature) for this state root. If the state is verified then the node also includes witness data for this state root which uses the same format transaction’s witnesses use.
getproof and verifyproof methods are a bit more special as they allow you to prove that some key-value pair exists in Neo state DB without having whole state DB (like when you’re operating on a different chain or when you’re working as a light node). This works via MPT path encoding from the root node to the particular leaf (value) node you’re interested in (that contains some token balance for example). Using this path data it’s easy to regenerate a part of MPT corresponding to that key-value pair locally and recalculate MPT hashes for that trie. If the top-level hash matches verified root hash then you have proof that the key-value pair is a part of the state DB shared by all proper Neo nodes.
So getproof method returns this path from the root node to the given key. It can then be used to verify the proof locally or can be used to send this proof to some trusted RPC node to verify it using verifyproof method that returns value for that key in case of success.
What are these new NeoX syscalls
Two syscalls were added along with other NeoX changes: “Neo.Cryptography.Secp256k1Recover” and “Neo.Cryptography.Secp256r1Recover”, they’re similar in their function and interface but using different elliptic curves for their operation. The first one uses SEC-standardized Koblitz curve widely known for its usage in Bitcoin and the second one operates on a regular SEC-standardized curve that is used by Neo.
Both of these syscalls allow to recover a public key from the given signature (r, s) on the given message hash with the help of a flag denoting Y’s least significant bit in the decompression algorithm. The return value is a byte array representing recovered public key (64 bytes containing 32-byte X and Y) in case of success and zero-length byte array in case of failure.
This functionality allows you to check message signatures in the smart contract, the key recovered can be compared with an expected one or be hashed and compared with an expected key hash (depending on what data is provided by the other blockchain).
How NeoX is supported in NeoGo
NeoGo 0.76.0 has full support for the functionality outlined above. Syscalls are available via interop wrappers in crypto package and RPC client contains methods to work with new RPC protocol extensions. Client-side support is always available, but NeoGo node’s behavior is controlled by two configuration options: EnableStateRoot and StateRootEnableIndex, the first one is boolean and the second one is integer. If not specified in the configuration the first one has a default of false and the second has a default value of 0.
EnableStateRoot controls state root generation and processing functionality. NeoGo is able to operate both on stateroot-enabled and classic networks, so this is the main switch between these two modes.
With EnableStateRoot set to false the node works in classic mode:
- no local state root is being generated
- consensus process operates using classic message formats not including state root data
- stateroot-related P2P messages are ignored
- stateroot-related RPC calls are available, but always return an error
- recovery syscalls are unavailable to contracts
- StateRootEnableIndex setting is ignored
With EnableStateRoot set to true things change and the node operates with full NeoX support, but a StateRootEnableIndex setting may additionally affect its P2P-processing behavior. getroots requests for blocks with height less than StateRootEnableIndex are ignored, roots messages are only processed for blocks higher than StateRootEnableIndex and the node doesn’t actively try to synchronize its state height until its block height reaches StateRootEnableIndex. This setting is made for network upgrades when there are no confirmed state roots for old blocks and they’ll never be properly confirmed.
Things you can do
Running a classic network
It doesn’t require changing anything, just upgrade the node to 0.76.0 and run it, the database format is compatible with 0.75.0.
Running new stateroot-enabled network
Setting EnableStateRoot to true and not setting StateRootEnableIndex is a good choice for a new private network as it gives you all the functionality from block zero. Note that all consensus nodes must be using this settings combination for successful operation.
Adding stateroot functionality to existing network
If you already have some network and you need it to continue working, but want to upgrade it with NeoX functionality you need to:
- prepare a current dump of network’s blocks
- upgrade all consensus nodes to NeoGo 0.76.0
- stop all of them
- change their configuration, setting EnableStateRoot to true and StateRootEnableIndex to some block in the future (not far away from current network’s height)
- remove CNs local databases
- import blocks from the previously generated dump on all CNs
- start all CNs
This can be optimized to reduce network’s downtime by doing block dumps/restores with old CNs still running, but you have to regenerate local databases with stateroot enabled for correct operation.
Computing local state roots for classic networks
You can also compute some local state roots for classic networks like mainnet, to do that you drop your old DB, set EnableStateRoot to true and StateRootEnableIndex to some block index you’re not expecting to reach soon (like 100000000). This way the node will produce local state root data and you’ll be able to play with it via RPC, but it won’t affect P2P interactions. One caveat though is that this node can’t be a consensus node.
Things you shouldn’t do
Randomly changing EnableStateRoot setting
Switching EnableStateRoot on and off without full block resynchronization may lead to unexpected results on any full node (independent of whether it’s a consensus node or not) because with EnableStateRoot set to true an MPT structure is initialized using local DB and if that DB doesn’t have correct MPT state it will fail. If you’re changing this setting in any way— restore the DB from block dump.
Running mixed consensus nodes set
All consensus nodes should agree on the protocol being used, either all of them use state roots, or all of them don’t. Mixing two types of nodes will lead to consensus failures.
Ready to try NeoX?
You can try NeoX features right now on testnet (and our default testnet configuration is NeoX-enabled now) or run some new private network while staying compatible with mainnet or older private networks. Just use NeoGo and it will help you running any kind of Neo network.
