Cutting blockchain tail with NeoGo

The problem of the ever-growing blockchain state is well-known. Even though some people want to always have a complete archive available on their nodes, in many cases it’s not needed, and applications can work just fine using only some subset of recent data. But what and how can be done to provide that depends both on the protocol and implementation. Luckily, NeoGo has everything needed, and in this article we’ll explain what your options are and what to expect from them.

State components and options

  • contract storage
  • block headers
  • block transactions
  • MPT data
  • transfer data
  • auxiliary data (depends on implementation, but negligible in volume)

Contract storage

Block headers

Block transactions

But this doesn’t mean any older transactions can be removed. When a transaction is run, it can access some data from the older blocks/transactions and the depth at which these requests can look is controlled by the MaxTraceableBlocks network setting. Neo Legacy never had that, it wasn’t possible with the UTXO model, but for N3 that’s one of the core settings. On current public networks it’s set to 2102400, which is one full year of blockchain life with 15-seconds blocks.

The key N3 protocol feature coming from the combination of these two parameters is that once a transaction becomes older than MaxValidUntilBlockIncrement and MaxTraceableBlocks relative to the current height, it can be safely deleted from the storage. Given the difference in defaults and the expected usage scenarios, usually it’s the MaxTraceableBlocks setting that is the most important regarding the ability to drop old data.

MPT data

Given that any block does some state changes, MPT also changes with every block. Each key-value pair change affects all MPT nodes from the leaf changed to the top, for the current tries it’s roughly 10 nodes. This means that many new nodes get created with every block and if you don’t do anything, the DB grows very quickly. In fact, when this functionality was introduced into Neo Legacy (0.76.0 version on NeoGo timeline) the DB size exploded more than 2-fold, and very soon everyone realized that storing all of MPT data is quite expensive.

Removing any of this data is not easy, any trie for any block has some nodes of its own and also references a big set of nodes from previous block’s MPT, while some nodes can be reused even in the same MPT. Back then, the NeoGo team did some explorations around garbage collection schemes, but they performed very poorly, rendering the system hardly usable. The main problem was that we needed to do a full MPT traversal during the GC cycle and it can’t be done quickly. C# developers at the same time tried reference counting; and this approach turned out to be more viable, resulting in the implementation of KeepOnlyLatest option (NeoGo introduced it in version 0.78.2). This option makes the node store only one MPT (instead of the whole history of MPTs), in many ways like the node stores only the latest contract storage data. The DB size dropped back almost to where it had been before state tracking was implemented (but Legacy had UTXO, so many transactions never changed contract storage).

The same option is available in the current NeoGo versions, and its logic hasn’t changed, KeepOnlyLatest allows to drop all of old MPT data but the latest one. At the same time, additional logic was implemented in NeoGo 0.98.2, more on that below.

Transfer data

The evolution of RemoveUntraceableBlocks

As you might already have noted, this only allowed removing one part of the old DB data, but we also had the KeepOnlyLatest option to deal with MPT data, so the only thing left in this case was transfer data. While it was kept forever, it at the same time wasn’t very big in volume, at least not requiring a lot of attention for the chains we had back then, so this combination was sufficient in many ways.

Things have changed over time, the chains grew, but more importantly NeoGo has also introduced another extension in version 0.97.3 that allows synchronizing contract storage state over the P2P network (P2PStateExchangeExtensions option). This extension relies on MPT data and another NeoGo extension that puts state root hash into the block header (just to remind everyone — none of these can be used on current public networks). The problem was that this functionality requires keeping at least three MPTs at different heights, so it’s inherently incompatible with the KeepOnlyLatest setting. But, of course, we wanted to drop old MPT data from the storage while still using these extensions.

That spurred another round of experimentation, but the problem turned out to be tougher than originally expected. It was trivial to keep all MPT data, it was very easy to keep just one MPT, but keeping a set of MPTs was a real challenge. They couldn’t be reference-counted (that’d mean traversing the whole MPT for every block to increment all counters), and our previous garbage collection attempts failed to reach adequate performance.

It required us several months of trying various schemes and optimizations to finally find the way to do this. All required DB changes and logic were released in NeoGo version 0.98.2. The base of this approach is still garbage collection, but the changes made to the DB allowed us to avoid MPT traversals even for GC cycle. We’ve also implemented garbage collection for transfer data, so starting from version 0.98.2 NeoGo can now drop all obsolete data in any configuration.

Setting RemoveUntraceableBlocks now activates all of the logic responsible for dropping old data. There is also an associated configuration parameter, GarbageCollectionPeriod that defaults to 10000 and specifies how often (in blocks) garbage collection is performed.

Benchmark

The two machines used are (both Linux-based):

  • Core i7–8565U with 16 GB of memory
  • Ryzen 9 5950X with 64 GB of memory

During our long quest for an optimal GC solution, we’ve also noticed that different databases react differently to variations in loads and schemes. Thus, we’ve performed the test on both databases NeoGo supports: LevelDB and BoltDB.

And we tested the following node configurations:

  • default one (keeps everything)
  • KeepOnlyLatest enabled (only one MPT stored, everything else untouched)
  • RemoveUntraceableBlocks enabled (and default 2M+ MaxTraceableBlocks, for both networks this means that nothing is actually removed, but the DB is prepared for removal at 2M+ heights)
  • MaxTraceableBlocks set to 100K blocks (and RemoveUntraceableBlocks enabled)
  • MaxTraceableBlocks set to 10K blocks (and RemoveUntraceableBlocks enabled)

Warning! Changing MaxTraceableBlocks can make your node incompatible with the network, it’s a network-wide protocol parameter, not something to be changed by a node. Here we’re doing it just for the test; but even in this case it wasn’t possible for us to test 10K MaxTraceableBlocks setting for the testnet, some transaction there looks back 22K blocks, so it fails and then leads to a state that is incompatible with the subsequent transactions.

Mainnet results

  • RemoveUntraceable (RUB)
  • MaxTraceableBlocks (MTB)

LevelDB/Core i7–8565U

BoltDB/Core i7–8565U

LevelDB/Ryzen 9 5950X

BoltDB/Ryzen 9 5950X

Testnet results

LevelDB/Core i7–8565U

BoltDB/Core i7–8565U

LevelDB/Ryzen 9 5950X

BoltDB/Ryzen 9 5950X

Comments

At the same time, there is a clear performance overhead for garbage collection or reference counting which in many cases can be compensated by the effectiveness of the smaller DB. Enabling RemoveUntraceableBlocks with the network settings we have now clearly hits the performance in all cases, but if it was allowed to cut the tail more aggressively it could not only save disk space, but also save time.

Regarding the DB difference, LevelDB is still a good choice for archive nodes that want to keep everything, BoltDB tends to use more disk space and is slower when it comes to write-intensive loads. However, it performs significantly better in any tail-cutting scenario and we expect space-preserving options to be used a lot in production. Also it’s better for read-intensive loads in general and while what we’re doing here is just processing the chain, any real node will also serve a lot of other requests related to P2P/RPC activity and we expect these to be handled faster with BoltDB (like it works better in our TPS benchmarks).

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store