Hyperchains development updates

[gorbak25]

Update #9 [9 November - 13 November]

@radrow.chain @erlmachinedev1.chain

• BTC connector
  We made decent progress on connecting to BTC - progress can be tracked here: GitHub - aeternity/aeconnector: Parent chains interface provider (Hyperchains).

• CI fixes
  Some small fixes to our CI pipeline needed to be made: Fix deploys in CI by gorbak25 · Pull Request #3405 · aeternity/aeternity · GitHub

• Consensus Engine
  Test mode consensus was introduced and our testing suite was refactored to use the new engine - due to the newly gained flexibility some test suites decreased their runtime 4 times. The test consensus will later be used for dev mode so developing aepps on AE will be a breeze. Moving block insertion and validation under the new engine is currently underway.

• Erlscripten
  The main focus in this week was getting the Erlscripten compiler to a decent level of compatibility. Right now we made decent progress, we now support multiclause lambda expression, records and many more. We now have a proper CLI utility for transpiling existing erlang projects and a testing framework for testing the output of our transpiler. To be fully compatible we still need to implement 4 things - Binaries, Exceptions, Processes, ETS - Right now we started working on exception support and expect to finish it soon. Currently we managed to get a decent part of Erlang’s STDLIB working:
  

  
  
  And we are capable of properly executing not trivial code like:
  
  
  
  After exception and binary support is finalized we can start transpiling our existing codebase and start making changes in the ecosystem.

[dimitar.chain]

Great progress with Erlscripten! Congrats!

[Vikram]

Yeah I am super excited to collaborate and run nodes in the test phase

[gorbak25]

Update #10 [16 November - 20 November]

@radrow.chain @erlmachinedev1.chain

• BTC connector
  We currently are near finishing our trust-less connector module - we stated publishing commitment transactions on the BTC testnet. We revisited our protocol and optimized it for cost - an amendment to the current whitepaper will be made soon.

• Consensus Engine
  During this week we made progress on refactoring block insertion to use the new consensus engine - the refactoring is still WIP and we expect to finish it next week.

• Erlscripten
  Full support for Erlang exceptions was implemented. We fully support apply/3 apply/2 make_fun/3. TCO support was greatly improved - most(but still not all) tail recursive erlang functions now runs efficiently in javascript. We transpiled real tests from OTP and they actually execute properly:
  
  Initial process emulation is now in place:
  

  
  
  Next week we aim to implement the last missing pieces(binaries, ETS) which will allow us to start transcompiling our existing Aeternity codebase to Purescript.

[gorbak25]

Update #11 [23 November - 27 November]

@radrow.chain @erlmachinedev1.chain

• BTC connector
  Right now we are writing documentation and tests for it. This component is nearly done.

• HC Protocol
  We optimized our protocol with regards to commitment size - right now commitments are only 32 bytes and PoGF is kept locally within the hyperchain. We designed a protocol level delegate registry used for mapping Bitcoin/Ethereum public keys to AE public keys. The hyperchain whitepaper will soon be updated.

• Consensus Engine
  The new consensus engine was finished and is currently under review. Implementing hyperchain consensus using it will be fairly easy. What’s notable is that we now can switch between different consensus algorithms - we can switch from PoW to Dev mode allowing us to flexibly play around with the state from mainnet - Additionally it’s now fairly easy to bootstrap your own hyperchain as it’s possible to temporarily use PoA/PoW to populate the staking contract - establishing the initial set of delegates.

• Erlscripten
  Binaries are now supported, we have a IO server inplace, TCO was further optimized. Right now we reached a point where we can transpile existing projects to Purescript:
  

  
  
  Currently we started the process of transpiling aebytecode, creating a JS friendly library for it and publishing it on NPM. Small fixes still need to be done - bigint support, implementing some missing BIFS etc…

[gorbak25]

Update #12+13 [30 November - 11 Dec]
@radrow.chain @erlmachinedev1.chain
The Recent 51% attack re-prioritized a lot of tasks - due to it here is a biweekly development update - following updates will be posted regularly.

• Coordination with the AE foundation maintenance team
  We coordinated on delivering the next 5.6 maintenance release in December
• Emergency response due to the 51% attack
• Consensus engine
  This large piece of code is still under review - we will include it in the 5.6-rc1 release and test it out on our infrastructure
• BTC integration
  The connectors are ready and right now we are integrating it in the node - implementation of HC consensus is underway.
• HC Protocol + Whitepaper - we plan to amend our whipepaper in the near future with all of the improvements we made to the initial HC protocol - A separate document tailored around BTC and investors will be prepared in the near future.
• DB corruption - blocker
  This is unrelated to the 51% attack - We found that more than half of our production nodes got their DB corrupted on a rare bug - crash due to “missing_hash”. We were notified by one external node operator that this issue also occurred on their private node. We immediately started investigating and trying to find the root cause of this issue - Healing the broken nodes from backups took us some time but now the network is back up and in good shape. We found the root cause of the issue and in the process we found a lot of other issues cripling the sync performance - in the future we will need to migrate our entire DB from “mnesia over rocksdb” to pure rocksdb - the corruption occured in the layer between those two. As the migration is a really big undertaking(we need to touch more than 200 places in the node) we currently derived a quick solution to get the DB back in a consistent state - as a side effect the sync performance improved more than 2 times - I literally was able to sync 10% of the AE mainnet on my laptop in less than an hour… After the DB migration this will become even faster. The proper DB migration is postponed until we get the HC protocol up and running
• Erlscripten
  A lot of progress! We reached a mature state here - we implemented a lot of BIFS, we have bigint support, etc… The project was restructured - it is now hosted under
  GitHub - erlscripten/erlscripten: Erlang to PureScript transpiler. Run Erlang in the browser!
  and some components were split up:
• GitHub - erlscripten/erlps-stdlib: Erlang's STDLIB in Purescript
• GitHub - erlscripten/erlps-core: Core runtime of Erlscripten - ERTS emulation
  The Erlang standard library is now fully working! We managed to get parts of the Erlang compiler fully working! Packages produced by erlscripten are now available for use in various projects.
  Transpilation of Aesophia/Aeserialization/Aebytecode is now underway
  Here is the transpiled Aeserialization library: GitHub - erlscripten/erlps-aeserialization: Purescript bindings for aeternity's aeserialization

[gorbak25]

Update #14 [14 Dec - 18 Dec]
@radrow.chain @erlmachinedev1.chain

• The first release candidate for the 5.6 release was released!
  This includes a lot of work from the hyperchains team and the maintenance team, including(not a full list) the last week DB consistency fix, the new consensus engine, a lot of sync performance improvements, new P2P messages, a full revamp of peer handling. Most notably sync was optimized a LOT - full sync used to take weeks, now it takes only 30 hours! The sync performance will improve a lot in the near future as Beam Sync is currently under development by the maintenance team and we plan to do a full DB refactor Right now 5.6-rc1 passed the full sync test and is deployed on half of our testnet nodes.

• BTC integration
  The connectors are ready and right now we are integrating them in the node - implementation of HC consensus is underway.

• HC Protocol + Whitepaper - we plan to amend our whipepaper in the near future with all of the improvements we made to the initial HC protocol - A separate document tailored around BTC and investors will be prepared in the near future.

• Erlscripten
  The CLI utility was equipped with useful options for transpiling existing erlang projects(recursive transpilation of files, omitting files/directories/functions, etc…) , more BIFs(like binary:split/2…) were implemented, our implementation of the erlang’s IO server was improved. We introduced “function splitting” in order for the purescript compiler to compile complicated pattern matches in a reasonable amount of time. We investigated the long compilation times of purescript projects and discovered the core reason for them - after tuning the Haskell garbage collector the compile times went down to reasonable levels. Aeserialization was transpiled and is fully working - it passes all erlang tests we wrote for it in the past Aebytecode was transpiled and is now partially working - some erlang tests for it pass but we need to tweek 1-2 places in the erlang code for it to fully work(we didn’t implement bitstrings yet - only binaries, aebytecode never uses bitstrings but does make some unaligned matches like <<Sign:1, Abs:6, ?SMALL_INT:1>> - after tweeking the 3 places the entire library should work as a charm). After aebytecode is fully working we will ship out the first production blobs for the Superhero team and start transpiling aesophia.

Right now most of the hyperchains team is on a short holiday break - we will resume regular development in the new year. This is the last update in this year - The HC Team wishes the reader a happy new year Expect the next update in the first week of January.

[Fee.chain]

Great work, looking forward to the January update, I wish a Merry Christmas and Happy New Year in advance.

[Vikram]

Merry christmas guys,

So excited for testing the hyperchains in the near future.

Have a good holiday

Thank You

[tietoujiucai]

Will hc airdrop to Ae holders？

[Baixin.chain]

Come on, believe you

[Fee.chain]

Hello, @gorbak25 will there be an announcement on the delayed launch of Hyperchains? I’m afraid we can no longer follow the roadmap plan.

[Janson]

Hello, @gorbak25 @YaniUnchained at present, the price of bitcoin is close to US $40000. If the hyperchain chooses bitcoin as the parent chain, how much is the daily handling charge of bitcoin? How do you deal with this problem? The purpose of hyperchain is to maintain low cost. If bitcoin is used as the parent chain, the price is not cheap at present.

[gorbak25]

Hi!
We are aware of the issue and it will be addressed in the updated whitepaper.

[Fee.chain]

Hello, shall we give up Bitcoin? But Ethereum is not cheap. I think if professional Bitcoin or Ethereum, no one is willing to be a node. If you do not choose Bitcoin and Ethereum, what is the significance of Hyperchain?

[omega32]

Hello!
Is there a proposed solution to the issue, or are you just going mention it in the whitepaper?

[gorbak25]

Update #15 [01 Jan - 15 Jan]
@radrow.chain @erlmachinedev1.chain

This is a fairly short update despite the time it took. The recent events - most importantly the recent 51% attack re-prioritized a lot of tasks and we were strongly focused towards:

1. mitigating the attack
2. improving the monitoring tools available in the AE community
3. Designing a temporary solution for securing AE mainnet
4. Coordinating with involved parties

Community fork:

To mitigate the recent 51% attack on the network we joined forces with the AE foundation maintenance team and we quickly developed a series of releases which then were applied to affected exchanges and miners. Release Lima community fork · aeternity/aeternity · GitHub We also provided a lot of support for involved parties.

Monitoring improvements:

In order to be better prepared for the future and to further empower the community we developed a series of new features tailored towards new analytic tools:

• The node now can provide detailed statistics on the state of the network - a WIP frontend for it is already deployed
• The node can now provide info on orphans/forks present in the local DB

Bitcoin integration

We focused heavily on cost optimization and found some interesting solutions to the problem at hand. Right now the Bitcoin integration is fully working.

Erlscripten

AeBytecode was transpiled and is working, right now we focus on getting the Sopia compiler fully working. There were a lot of fixes here and unfortunately we encountered some blockers. We aim to wrap up this in the near future.

[gorbak25]

Yes, there are a lot of viable solutions on the table:

1. Use a cheaper but just as secure chain - ETC? Litecoin?
2. Delegates can optionally operate as payment gateways - the BTC fees are shared between various parties reducing the overall cost significantly.
3. Increase the election interval.
4. On hyperchains one might delegate stake to a delegate - staking pools might appear on the network.
5. …
   After we settle on a particular solution(or a mix) we will describe it extensively in the whitepaper.

[Fee.chain]

Hello, is there a timetable for the new white paper?

[gorbak25]

Update #16 [18 Jan - 08 Feb]
@radrow.chain @erlmachinedev1.chain

Here is an update for the past three weeks. A lot of work had been done and we touched a lot of areas.

51% attack postmortem - Hardening the network

It’s clear that due to the low hashrate of the network another 51% attack could happen at any point in time - to protect the network before deploying a long term solution(possibly Hyperchains) the AF maintenance team hardened the node. Right now the newest version of the node finalizes transaction after 100 blocks(5h) regardless of the hashpower - making AE safe again for transacting.
We released 3 new nodes:

• v5.7.0 (In-RAM finality after 100 blocks)
• v5.7.1 (Compatibility fix which sneaked into v5.7.0)
• v5.8.0 (Proper finality after 100 blocks - persistent after restarts)

51% attack postmortem - The community fork outmined the fork from the second 51% attack

On the 28th of January 2021 the community fork accumulated enough difficulty for pre v5.6.1 nodes to switchover to the community fork. No node in sight is on the fork of the attacker.

CI pipeline improvements

As the v5.7.0 release got broken by a simple preventable bug we improved our CI pipeline to catch such bugs early. Fortunately the effort was worth it as it caught issues with upcoming OpenApi 3.0 support.

Reporting tools for validators

In order for validators to operate a hyperchain they need to have insights on their spending, they need to be notified when their funds are running low, when they mined a block, etc… Aeconnector now supports reporting on your expenses and notifying you via various channels such as Telegram

Possible migration of Ae Mainnet to a hyperchain

We designed a nice approach on switching over AE to use Hyperchain consensus - details will be placed in the upcomming whitepaper. The decision on this toppic will be made using a governance vote. Expect more details soon! Possibly you could soon become a delegate and gain rewards from staking

Staking contract deployment and predeployment

The staking contract was nicely integrated with the node - what’s notable is that in the case of switching from PoW to HC, HC nodes will be backwards compatible until the staking contract accumulates enough economic security.

Staking contract UI

This is mostly WIP - the first version will be not elegant but will get the job done. We will later delegate this task to the Superhero team.

Hyperchain consensus and BTC

Right now we are aiming to get a basic version of the HC consensus fully operational. After that’s done and tested we will proceed to deploy the first testnet.

Hyperchain whitepaper V2

No progress on it so far - we first need to sort out lower hanging fruits before revisiting it.

Erlscripten

The JS Sophia compiler works in 95% We will soon publish a bigger article about it. Right now the project in a phase where the interface to Erlscripten is stable and the Superhero team started bundling it up. We made some upstream contributions to the purescript compiler - we are working on decreasing the stack usage and improve the quality of transpiled code.

Stay tuned for the next update!