Hi all!
As you know we had identified a memory leak and we had been planning a patch release. Since the tracing and fixing it took quite some time and we had accumulated some actual functional changes, we prepared a minor release. Behold, Aeternity node v.6.7.0
It fixes the memory leak. It was a bug really hard to find and it had been there sleeping for quite some years now. It was only exposed by a combination of small CPU power and the new DB refactoring. In a nutshell it goes like this: Erlang is a GC language and it cleans any unused memory and this works great. It also allows loading external C/C++ code (called NIFs) and it is up to the NIF to clean its own unused memory. We use C code with regards of mining and C++ for the DB connection. Since the cuckoo miner had not been touched, our prime suspect was the C++ DB connectors, especially as we now use them a bit differently. We ran diagnostics and they didn’t expose an issue there. We also replaced them with a Rust alternative and it did not change the unexpected memory growth. To make matters even harder to debug, noone was able to reproduce the growth locally, it only happened in our cloud environment. We even tested it with a home grown C DB binding. Eventually we traced the issue to a race condition in the node’s garbage collection. When the node cleans up the DB of the unused nodes in account’s tree, some cache must be flushed. The old approach was restarting the node itself. This could lead to leaking the NIFs themselves: at this point the BEAM was not aware of them anymore and was unable to clean them up. This race condition happened only on machines with limited CPU power. So a fix was an appropriate handling of states after node’s GC of accounts. This is fixed an in this release.
Further more we ship a handful of small features. The next release is going to be much more interesting as it it will include more user visible features
We had been running this node since the last week and we publish it only now. We had upgraded all our nodes.