Hyperchains erlang implementation details

Aeternity node is an implementation of the Aeternity protocol in Erlang. The following is a description of how this can be adapted to build Hyperchains on top of AE blockchain using the aeternity node.

Block validation

Header structure

The plan is to keep the child chain headers as they are. If we can get away without changing them, that would be great. Currently we don’t see a reason to change them.

In order to be able to validate the child chain keyblock, one needs a reference to the parent keyblock that ends the child election cycle. At the moment the structure of the keyblock is:

-record(key_header, {
          height       = 0                                     :: height(),
          prev_hash    = <<0:?BLOCK_HEADER_HASH_BYTES/unit:8>> :: block_header_hash(),
          prev_key     = <<0:?BLOCK_HEADER_HASH_BYTES/unit:8>> :: block_header_hash(),
          root_hash    = <<0:?STATE_HASH_BYTES/unit:8>>        :: state_hash(),
          target       = ?HIGHEST_TARGET_SCI                   :: aeminer_pow:sci_target(),
          nonce        = 0                                     :: non_neg_integer(),
          time         = 0                                     :: non_neg_integer(),
          version                                              :: non_neg_integer(),
          pow_evidence = no_value                              :: aeminer_pow_cuckoo:solution() | no_value,
          miner        = <<0:?MINER_PUB_BYTES/unit:8>>         :: miner_pubkey(),
          beneficiary  = <<0:?BENEFICIARY_PUB_BYTES/unit:8>>   :: beneficiary_pubkey(),
          info         = <<>>                                  :: bin_info()
         }).

In the context of hyperchains, nonce and pow_evidence are unnecessary. In their place we need a parent_prev_key instead

Approach 1: recycle the pow_evidence

Since:

-type solution()          :: [integer()].

We could reuse the structure and store there the prev hash of the parent chain as a list of integers. Then expose a function to “read” the pow_evidence into the correct format. I think this is a wrong approach but it could be explored further.

Approach 2: change the record structure with a plugin approach

Add an extra field in the #key_header{}

          extra        = #{}                                   :: map()

We can change the serialization of the keyblocks to accomodate it. In this map we can store anything and thus - expand it. Some problems that could arise are:

  • we must be extra cautious so a parent chain type of block does not end up in the child chain
  • we must still be able to read both child and parent chain keyblocks

Approach 3: bloat the #key_block{}

Add another record #child_key_block{} and addapt all the module exported functions accordingly. Also expose a new one for the parent_prev_key that works for #child_key_block{} only. This will have a lot of code and test duplication.

parent and child node incorporation

In order for the child node to be able to validate who the elected leader is, it should fetch the latest parent chain keyblock. How to structure this in the code has two different approaches, each with its drawbacks.

Approach 1: separate parent and child nodes

In this scenario we have a parent chain node that is independent running node. It syncs with the parent chain and it has independent DB.
The child chain calls its APIs in order to fetch new keyblocks. It should have a polling mechanism to detect forks on the parent chain. This is especially important with regards of commitments posting. This is the fastest MVP approach.

A big drawback is that the HTTP requests introduce latency to the keyblock validation. This means that if a node receives a new child chain keyblock from the next leader, it should fetch the parent chain’s keyblock as it is the source of entropy. This request will take significantly longer than what is currently the Cuckoo’s cycle validation. This might open the gates for DoS attacks, flooding certain nodes with keyblocks that take time to proove invalid. This is OK for MVP but it might not OK for a real-life scenario. This should be investigated further.

Approach 2: integrate parent and child nodes

In this scenario the child and the parent nodes are one piece of software - one is baked in the other. This results in much faster child chain keyblock validation. This requires duplication of whole applications: two sync-s, two DBs and etc. This will not be a trivial change and will require some heavy testing to make sure the two nodes do not end up entangled.

4 Likes

Hi ,

Personally I like the 2nd approach it will be much faster to sync. And almost feels like we are running the current AE node. Not much major changes.

But I am still learning the system , So I am sure we are going to get many amazing ideas from other members :slight_smile:

1 Like

about what timeframe are we talking when approach 2 is being targeted @dimitar.chain?

this is important for me because I expect that we won’t see another hardfork until this is finished. and this means that I need to wait for certain features (of which some are already implemented) for quite a long time

1 Like

Yes, from user’s perspective it would be just some configuration changes, that’s the idea. From implementation point of view though it would mean a few radical refactorings.

I personally think that we should aim at a MVP as soon as possible, then we can optimize. What is more important: once we have a MVP we can play with it, see where it needs improvements from user’s perspective and adapt accordingly.

Regarding the node incorporation, approach 2 - this should be protocol agnostic so it shouldn’t really matter from outside’s perspective. I’d rather go with the first approach first as the second would likely take a month more to implement. Or more, no proper estimates had been done yet.

2 Likes

My view of the schedule ahead. I am turning this post into a wiki, so you can adapt it.

Tasks

Implement child keyblock structure

Task Id

1

Desription

The fastest MVP approach would certainly be the plugin one. See above

Estimate

Around 14 days. It could be significantly less or even more. The size of it is still a blackbox, we will know once this is started.

Implement modules to talk to parent chain

Task Id

2

Desription

  • to fetch and keep latest keyblock - this would be a polling mechanism so we piggyback on existing HTTP APIs. If we happen to receive a child keyblock that points to a certain keyblock that is not the latest one, we should ask for that one in particular. This would likely be a standalone process that keeps cached the last parent keyblock in case it is needed. This likely should be attached to a gproc pub-sub mechanism so subscriber processes are informed for when a new keyblock is received. This must be fork aware, ex: if a new parent keyblock is received, we should check if it is based on the latest cached one. If not - we should check if the cached keyblock is still in the blockchain or it had been evicted in a fork. If so - we must rollback child chain blockchain to the point of forking and continue from there. We might not have to keep the parent keyblock hashes but rather piggyback on the MPT implementation to go a single microblock back. This might have an impact on GC default settings
  • to track the progress of certain transactions (commitments). From an API this should be something closely resembling the min_depth_watcher but instead of being subject to gproc it would be fed by HTTP API. This can be further improved with exposing a pub-sub from the node
  • list all commitements in a generation

Estimate

5 days

Implement leader election

Task Id

3

Desription

  • check if an account is eligable for delegate
  • check if a commitement on the parent is valid - by an electable leader, following the current top
  • to detect when current node is elected for a leader - this will be triggered by the pub sub mechanism provided by task 2. Once detected, the keyblock producing mechanism shall take over and the leader shall publish microblocks. The same pub sub mechanism could also inform the process that there is another different leader elected. From then on there should not be much changes.
  • to verify a keyblock - that a correct leader had produced it. This will be a part of the keyblock validation process. This is to be used in sync. Note that it wouldn’t be pure in the functional programing meaning of it - it will depend on an external service to validate the block. This should also interupt a current leader not to publish new microblocks.

Estimate

7 days

Commitment for next round of Election process

Task Id

4

Desription

  • post new commitement - Once triggered by the pub sub of task 2, elligable delegates (implemented in task 3) shall produce and post commitements on parent chain. This will enhance a bit the task 2 enriching it with functionality for posting a transaction
  • react on forks (thanks to pub sub in taks 2) on parent chain and repost commitement if needed

Estimate

3 days

Create config setup

Task Id

5

Desription

Expose a child chain network_id to the confg. Make the node differentiate between child and parent chain network_id-s (this might as well get messy).
Define from a user perspective what shall be configurable in a hyperchain and expose child chain rules to the config.

Estimate

7 days

Write black box system tests

Task Id

6

Desription

This becomes a scenario with a lot of moving parts, this needs some heavy system tests that focus on various scenarios and attack vectors:

  • happy path - one node is elected as a leader, all accept their keyblock and microblocks; all delegates make commitements
  • invalid delegate commitements are disregarded
  • malicious leader is properly punished with regards of Bitcoin NG consesus algo
  • invalid keyblocks from the wrong leader are disregarded

A more complex tests are a must, ex:

  • a child node receives a keyblock from the child chain but the corresponding parent chain keyblock arrives after it. At validation time it is not present in the parent chain according to the node we consult. In one second it is. The child chain node must be able to sync.

Estimate

A least 20 days but could be split into smaller tasks implemented in parallel

Not covered in the MVP:

  • Stake delegation in the child chain
  • Proper incentivization problems analysis in child chain consensus
  • impact on latency in block validation analysis
  • investigation of child chain related tools

Sequence

+-------------------------------------------------------------+    +-------------------+                       +-------------------------------------+
|                                                             |    |    task 4         |                       |                                     |
|          task 1                                             |    |commitement posting|                       |                                     |
|      child keyblocks                                        |    |3 days             |                       |             system tests            |
|      14 days                                                |    |                   |                       |             20 days                 |
+-------------------------------------------------------------+    +-------------------+                       |                                     |
                                                                                                               |                                     |
+------------------------+ +-------------------------------+       +---------------------------------------+   |                                     |
|                        | |                               |       |                                       |   |                                     |
|         task 2         | |            task 3             |       |         task 5                        |   |                                     |
|     communication to   | |        leader election        |       |      configuration                    |   |                                     |
|     parent node        | |                               |       |      7 days                           |   |                                     |
|     5 days             | |        7 days                 |       |                                       |   |                                     |
+------------------------+ +-------------------------------+       +---------------------------------------+   +-------------------------------------+
4 Likes

This approach has been implemented in PR #3294, awaiting re-review. The PR’s main feature is a parse transform, which allows internal core APIs to be flagged as modifiable from a plugin application.

2 Likes