[Active] AE maintenance Q2, Q3 and Q4

Application Status

Status: Approved on the 06.05.2021
Last updated: 04.05.2021
**Submited by Dimitar Ivanov @dimitar.chain, Submitted on 04.05.2021
Team: Ulf Wiger, Hans Svensson, Ulf Norell, Dincho Todorov
Approved Budget (in h):
Used Budget (in h):
Planned Delivery:

Specify the funding category

Open Source Development

Application Title

AE maintenance Q2

Applicant

We are the core team.

Value Application

Since now iris hardfork is closing in, We are making plans for the next months. We will be working on the planned roadmap. Those are all pending improvements that help stability of the network, performance or provide new tools to empower developers and node operators

Definition of Terms

Some notable changes on our plate are:

  • A major DB refactoring The initial imlementation to the DB access was impemented for roma release and had not changed much. Since then we’ve acquired a lot of experience and know-how and now we are revisiting the approach grounds up. We are keeping backwards compatibility but the new solution would be far superior than the old one, especially in raw performance but also it would improve stability.
  • Node maintenance mode This is a concept we developed around the 51% attack: we need to be able to disable for a while certain applications running on the node to allow certain operations to be performed on the node itself. This will allow us to build CLI tools. Two such tools would be allowing a node operator to manually pick one fork over another. This is to make fighting 51% attacks easier.
  • AeCanary The monitoring tool AeCanary is already in development and will be released soon. It provides the ability to track the tainted tokens. Also it delivers to the exchanges an exposure at a given point of time
  • FATE improvements - we will gather feedback from FATEv2 and publish plans for further improvements.
  • Beam sync - a radical optimization of fast sync
  • Rosetta API implementation
  • Tests and documentation improvements. It is cleanup time :slight_smile:

Status Quo

The node is safe and stable already, we want to make it faster and easier to use by users.

Required Work

Deliverables would be node releases and AeCanary releases.

Known Limitations

This is quite an ambitious plan. We will grow the team but finding new members will take time and onboarding - even more so. There is the possibility that some of those tasks will spill in Q3.

Outlook

After those tasks are finished, node operators will be better prepared with fighting 51% attacks on the network with both tools and early notification system.

Publishment

Our work will be published under ISC License.

node repo: https://github.com/aeternity/aeternity
AeCanary repo: https://github.com/aeternity/ae_canary

6 Likes

It’s worth celebrating

:grinning:good

1 Like

Any hope for Windows users to be able to run a node more easily as well? Would be probably helpful to have more nodes in Asia, Africa and Latin America :slight_smile:

4 Likes

You can find your updates below.

Week 27 (5 July - 11 July)

Dimitar Ivanov (dimitar.chain)

Dimitar extended the aeternity/swagger-endpoints to allow working with more than one method per URL. This resulted in some simplification of the generated file. This was incorporated in the aeternity/aeternity repo. More work was done on deletion of transactions from the mempool. A total of 35 hours.

Ulf Wiger (uwigeroferlang.chain)

Ulf worked mostly on the structural changes needed for maintenance mode and proper plugin management. Time spent: 36.5 hours

earlyriser99

earlyriser99 worked on AeCanary adding more configuration to allow deployment in different environments and adding partial sync of the blockchain for fork detection.
Time spent: 18 hours

zxq9

Craig was mostly on vacation this week (typhoons!), taking care of a few administrative tasks.
Time spent: 2 hours

Week 28 (12 July - 18 July)

Dimitar Ivanov (dimitar.chain)

More work was done in transaction deletion PR. More transaction pool tests were added and those exposed two small bugs in transaction pool were. Those were fixed. A total of 39.5 hours

Ulf Wiger (uwigeroferlang.chain)

Ulf turned his attention to the changes needed for ‘development mode’ (including instant block production on-demand). This is already mostly implemented and used by automated tests, but the code is intricate and takes some time to get into. Time spent: 35 hours

earlyriser99

earlyriser99 worked on AeCanary replacing the fork detection algorithm with a more robust version and setting up fork detection to trigger email warnings and dashboard display.
Time spent: 25 hours

zxq9

Craig worked on Foundation website updates, mission statement and other slightly more esoteric issues this week.
Time spent: 4 hours

cna0

Sina moved hyperchains tasks from OpenProject to GitHub issues, updated and merged two hyperchains PRs (#3647, #3661), and worked on prettifying hyperchains-master code. Time spent: 10.5 hours

Week 29 (19 July - 25 July)

Ulf Wiger (uwigeroferlang.chain)

Ulf worked on basic support for Maintenance mode and read up on Hyperchains. Time spent: 22 hours

earlyriser99

earlyriser99 worked on AeCanary idleness detection algorithm and Hyperchains discussions
Time spent: 8 hours

cna0

Sina worked on prettifying hyperchains code, documenting hyperchains ambiguities, logging node database info, PR reviews, and hyperchains discussions. Time spent: 17.5

Dimitar Ivanov (dimitar.chain)

Dimitar was inspecting transaciton garbage collection and started onboarding HyperChains. A total of 21.5 hours worked.

Week 30 (26 July - 1 August)

Ulf Wiger (uwigeroferlang.chain)

Ulf was off, giving opera concerts. Time spent: 2.5 hours

earlyriser99

earlyriser99 was away on holiday. No time was devoted to Aeternity this week

cna0

Sina worked on cleaning up some parts of the code, studying the code to find refactoring cases, and hyperchains discussions. Time spent: 16.75 hours

zxq9

Craig worked through some additional platform identification issues, experimented with getting external commands to be managed more easily from within the node, and reduced the complexity of the configuration interface (the complex version seems to make more sense in an “advanced” mode, as most of the options will not be of interest to most users).
Time spent: 13 hours

Dimitar Ivanov (dimitar.chain)

Dimitar was onboarding HyperChains. A total of 24.5 hours worked.

Dincho Todorov (@dincho.chain)

Dincho completed to Terraform modules update to newer improved version and migration of all nodes to Ubuntu 18.04.
Also complted few tasks in the “beta” milestone of Javascript Calldata library.
Time spent: 35h

Week 31 (2 August - 8 August)

Ulf Wiger (uwigeroferlang.chain)

Ulf worked on analysis of Hyperchains and continued working on Maintenance mode. Time spent: 34 hours

earlyriser99

earlyriser99 was away on holiday. No time was devoted to Aeternity this week

cna0

Sina worked on code cleanup, reviewing code, hyperchains discussions. Time spent: 13 hours.

zxq9

Craig continued with the automated build process from the GUI and started separating out platform specific issues, deciding to focus first on the canonical Debian case, getting the automated sourcing and building procedures to work.
Time spent: 12 hours

Dincho Todorov (@dincho.chain)

Dincho finished the “beta” milestone of Javascrip calldata library by adding documentation and publishing the package to NPM.
Started working on applications k8s cluster configuration to migrate current setup (compiler, faucet, etc.) to it.
Time spent: 38h

Dimitar Ivanov (dimitar.chain)

Dimitar introduced a special filter for AeCanary - this is to allow representatives of each exchange to see data relevant only to them. Work was started on transaction strict mode. A total of 33.75 hours worked.

Week 32 (9 August - 15 August)

Ulf Wiger (uwigeroferlang.chain)

Ulf has been testing the Maintenance mode support. Some shortcuts in automated tests need to be addressed. Time spent: 38.5 hours

zxq9

Craig researched use cases for hyperchains, use cases for the main PoW blockchain and existing smart contract utilities for creating a marketplace for a real-world p2p product exchange system (imagine Etsy or eBay, but not subject to political censorship from a payment processor), and started working on the load/start procedure from within the GUI’s own node.
Time spent: 20 hours

Dincho Todorov (@dincho.chain)

Dincho continued working on application cluster and there is some dev environment already running with gitops configuration of argocd, traefik ingress, prometheus, grafana and alertmanager.
Time spent: 30h

Dimitar Ivanov (dimitar.chain)

Dimitar was working on transaction pool strict mode. This required a few nasty refactorings. A total of 43.5 hours worked.

earlyriser99

earlyriser99 worked on adding detection of cases where the chain stops creating blocks/microblocks/transactions, which could indicate some underlying issue, and on understanding Hyperchains. A total of 13.5 hours.

5 Likes

Is the Rosetta API implementation complete?

2 Likes

support you, good job!

If the team always has such a detailed plan and arrangement and communicates with everyone technically, there will be less complaints, at least let us know that the team has been working there.

2 Likes

:slightly_smiling_face:
Is the Rosetta API implementation complete?

1 Like

Keep your original mind; stay patient!

2 Likes

any update?Also the hyperchain seems to stop?

1 Like

You can find our progress reports below:

Week 33 (16 August - 22 August)

Ulf Wiger (uwigeroferlang.chain)

Ulf worked on final testing the Maintenance mode support. Time spent: 22.5 hours

earlyriser99

earlyriser99 continued work on adding detection of cases where the chain stops creating blocks/microblocks/transactions. A total of 17.5 hours.

Dimitar Ivanov (dimitar.chain)

Dimitar was working on transaction pool strict mode. A deep analysis of possible errors to be exposed to the config. Total of 40 h

Week 34 (23 August - 29 August)

Ulf Wiger (uwigeroferlang.chain)

Ulf finished maintenance mode support, updated the PR on chain whitelisting and rollback (to utilize maintenance mode), and then started working on dev mode. Time spent: 38 hours

earlyriser99

earlyriser99 finished work on adding detection of cases where the chain stops creating blocks/microblocks/transactions. A total of 16 hours.

Dimitar Ivanov (dimitar.chain)

Dimitar was working on transaction pool strict mode. A deep analysis of possible errors to be exposed to the config. Tracing an issue reported by a third party. Time spent: 40 h

Week 35 (30 August - 5 September)

Dimitar Ivanov (dimitar.chain)

Dimitar was working mostly on transaction pool strict mode. This required a refactoring of the DB storage. Fixed issues with dialyzer for OTP23. Time spent: 43 h

Ulf Wiger (uwigeroferlang.chain)

Ulf turned his attention to “developer mode”, allowing developer tools more speed and flexibility in their app development. Time spent: 43 h

earlyriser99

earlyriser99 worked on re-factoring AeCanary to support tainted accounts tracking. Time spent 14hrs 40mins

Week 36 (6 September - 12 September)

Dimitar Ivanov (dimitar.chain)

Dimitar was still working mostly on transaction pool strict mode. Improved config utils and those now provide much smarter defaults. Time spent: 41.75 h

Ulf Wiger (uwigeroferlang.chain)

Ulf prepared an initial PR of dev mode, including a new type of support for plugins. He then decided to look into some intermittent failures slowing down the testing process. Time spent: 40.5 h

Hans Svensson (hanssv.chain)

Hans wrote a new Sophia standard library Bitwise.aes with bitwise operations for arbitrary sized signed integers. The library, and these operations, were asked for in an effort to translate a DEX-contract to Sophia.
Time spent: 6h

zxq9

Craig updated the Nix build definitions, completed the build and check procedure for the GUI launcher utility, and continued analysis and assessment of global economic conditions.
Time spent: 14h

earlyriser99

earlyriser99 continued work on re-factoring the AeCanary database sync to use the nodes directly instead of MDW. The goal is to support tainted accounts tracking. Time spent 19h

Week 37 (13 September - 19 September)

Dimitar Ivanov (dimitar.chain)

Dimitar finished the transaction pool strict mode. A new SUITE is provided to test thw whole thing. Testing on integration network. Time spent: 30.5 h

Ulf Wiger (uwigeroferlang.chain)

Ulf identified one race condition in the handling of State Channel conflict messages. The problem is unlikely to appear in real-life use, but had a higher likelihood of triggering in automated testing. He then went on to analyse another intermittent test failure. Time spent: 36 h

Hans Svensson (hanssv.chain)

Hans started working on a more efficient algorithm for packing transactions into micro-blocks. Hans also spent time reviewing the TX-pool strict mode.
Time spent: 12h

Week 38 (20 September - 26 September)

Dimitar Ivanov (dimitar.chain)

Dimitar was working on limiting tx pool reentry. He also fixed an HTTP API issue with int-as-strings flag. Spent some time on a new HTTP endpoint for getting a name entry by a name hash but this is not ready yet. Tracing an intermittent test failure in system tests. Time spent: 43.75 h

Ulf Wiger (uwigeroferlang.chain)

Ulf worked on implementing suggestions gathered from initial testing of the dev mode plugin. Improvements include support for configuration data schemas for plugins, and simpler parameterization. Time spent: 39.5 h

Hans Svensson (hanssv.chain)

Hans finished the more efficient algorithm for packing transactions into micro-blocks, PR is in review. Hans, also improved the calldata/result decoder in the Sophia compiler - this enables the decoding of complex builtin types such as AENS.name and Chain.base_tx.
Time spent: 12h

zxq9

Craig finised the startup procedure of the GUI launcher, added statistics reporting from the local node within the GUI, and began work on graphing GUI stats reporting.
Time spent: 20h

earlyriser99

earlyriser99 started the work to resolve the various hyperchains branches and to get the hyperchains playground working, as well as continued work on re-factoring the AeCanary database sync. Time spent 21h55m

4 Likes

We had some internal discussions and we are planning to implement the Rosetta API as a plugin. Now we have plugin support, that we plan on expanding further and moving all non-essential core functionalities as plugins. We are really excited about this and I will write a forum post with more details.

There had been some progress with HCs, now we are combining changes from competing PRs and revisiting the playground. I hope we can show some functionality soon. Still the really hard edge cases are not implemented yet.

4 Likes

Thank you for your response; praise the team’s work

2 Likes

好的谢谢 :+1: :+1:

Hi,
Our report for the past few weeks is below. As you can see we are making some great progress in various directions. We also have 2 new team members onboard. We are looking into expanding the team even further.

Week 39 (27 September - 3 October)

Ulf Wiger (uwigeroferlang.chain)

Ulf finished the work on preparing the dev mode plugin, and corresponding support, for release. He also refreshed and merged an improvement of minimum-depth confirmation for State Channels. Time spent: 44.5 h

Hans Svensson (hanssv.chain)

Hans looked into multisig - in theory it should be possible to use (n-out-of-n) multisig directly on chain without modifications. After making some additions to enacl a proof-of-concept transaction was made together with Dimitar - all summarized here.
Time spent: 18h

Sean Hinde

Sean picked up the work to resolve the various hyperchains branches and to get the hyperchains playground working. Time spent: 07h15m

Dimitar Ivanov (dimitar.chain)

Dimitar finished the transaction pool reentry filter. This actually exposed an intermittent test failure in the pool test suite, which was fixed as well. Dimitar spent some time identifying what is present and what is missing in HyperChains playground. At the end of the week he supported Hans with the proof-of-concept multisig.
Time spent: 37 hours

Week 40 (4 October - 10 October)

Ulf Wiger (uwigeroferlang.chain)

Ulf mainly worked on debugging (and fixing some) intermittent failures in the Continuous Integration environment. Time spent: 36 h

Hans Svensson (hanssv.chain)

Wrapping up multisig proof-of-concept, unfortunately it requires an updated enacl version - in preparation for that (it is problematic since the LTS version of Ubuntu, 18.04, we support have a too old version of libsodium) there are tests for enacl to ensure a new version still does the right thing.
Time spent: 6h

Sean Hinde

Sean continued the work to resolve the various hyperchains branches and to get the hyperchains playground working. Time spent: 07h30m

Dimitar Ivanov (dimitar.chain)

Dimitar started working on CLI tooling - providing the needed infrastructure and test suites. He also produced release 6.3.0.
Time spent: 43.75 hours

Week 41 (11 October - 17 October)

Ulf Wiger (uwigeroferlang.chain)

Ulf fixed some issues with the dev mode plugin, then started experimenting with a HTLC (Hash Time Locked Contract) for “State Channel markets”. Time spent: 32 h

Hans Svensson (hanssv.chain)

Little time available. As part of the Q4 planning, Hans started looking into FATE/Sophia to write down what additional features and fixes could make it even stronger.
Time spent: 2h

Sean Hinde

Sean started work on re-factoring of AeCanary to support following tainted accounts, tested the upcoming GUI on macOS, fixed a number of AeCanary issues and started work on cleaning up build warnings and deprecations across the Erlang codebase. Time spent: 33h25m

Dimitar Ivanov (dimitar.chain)

Dimitar was working on CLI tools. He was adding support for tx pool interactions: add and delete a tx from the pool, inspect the tx pool and modify the minimum required gas price.
Time spent: 25 hours

Week 42 (18 October - 24 October)

Ulf Wiger (uwigeroferlang.chain)

Ulf made a first proof-of-concept testcase of token transfers between State Channels via an intermediary. He also fixed a bug in the Db rollback function, which caused transactions to be orphaned. Time spent: 44.3 h

Hans Svensson (hanssv.chain)

Spent some time on reviews, mainly the pattern guards in Sophia #339, also started working on Sophia/FATE extensions for the Ceres protocol upgrade. What is planned is: bitwise integer operations, a MULMOD instruction, and a ZK/SNARK friendly hash function Poseidon hash. Also fixed some minor documentation issues in Sophia.
Time spent: 6h

Sean Hinde

Sean continued work on cleaning up build warnings and deprecations across the Erlang codebase.
Time spent: 10h23m

Dimitar Ivanov (dimitar.chain)

Dimitar finished the tx pool CLI. he debuged some tests and started working on CLI for local peer lists.
Time spent: 41 hours

Week 43 (25 October - 31 October)

Hans Svensson (hanssv.chain)

Working on Sophia/FATE extensions for Ceres. This also includes introducing version 3 of the FATE VM, which is a bit of work in the repository. Work-in-progress branch ceres_fate_extension. Also prepared a new version of enoise that is ready for OTP-24.
Time spent: 14h

Ulf Wiger (uwigeroferlang.chain)

Ulf worked on the State Channel “market” PoC and addressed some issues with chain database rollback for dev mode. Time spent: 40 hrs

Sean Hinde

Sean continued work on cleaning up build warnings and deprecations preparing for OTP24 and Mac m1 support across the Erlang codebase.
Time spent: 26h20m

Dimitar Ivanov (dimitar.chain)

Dimitar was working on peer CLI tools. He was adding support listing connected, verified, unverified and blocked peers. Also added functionality for adding and removing peers and blocking/unblocking them.
Time spent: 39 hours

Week 44 (1 Nov - 7 Nov)

Ulf Wiger (uwigeroferlang.chain)

Ulf worked on the State Channel market PoC, did some work on the database plugin rewrite and rollback support for dev mode. Time spent: 39 hrs

Sean Hinde

Sean worked on cleaning up build warnings and deprecations preparing for OTP24 and Mac m1 support across the Erlang codebase.
Time spent: 40 hrs

Hans Svensson (hanssv.chain)

Hans assisted Sean with OTP24 and Mac M1 support, fixed a bug in aesophia_cli w.r.t. UTF-8 encoded arguments. Also prepared proper PRs for Sophia/FATE extensions across aebytecode, aesophia, and aeternity repositories. While preparing the PRs also fixed some test cases.
Time spent: 17h

Dimitar Ivanov (dimitar.chain)

Dimitar was moslty off. He started tracking a test issue.
Time spent: 2.5 hours

Week 45 (8 Nov - 14 Nov)

Ulf Wiger (uwigeroferlang.chain)

Ulf got the State Channel market test suite to work in an initial form, and also improved the REST interface of the dev mode plugin. Time spent: 42 hrs

Sean Hinde

Sean worked on OTP24 and Mac m1 support across the Erlang codebase, investigated potential future consensus ideas, and dug into Sync performance.
Time spent: 40 hrs

Hans Svensson (hanssv.chain)

Limited availability - reviewed a bunch of OTP24 and Mac m1 PRs.
Time spent: 2h

Dimitar Ivanov (dimitar.chain)

Dimitar was mostly off. He fixed the test issue.
Time spent: 11.5 hours

Week 46 (15 Nov - 21 Nov)

Ulf Wiger (uwigeroferlang.chain)

Ulf merged dev mode plugin improvements, and returned to the database plugin, as Sean solved some outstanding problems with the underlying library. Time spent: 34 hrs

Fabian Krol

Fabian worked on ae_canary refactoring and removal of middleware’s API usage.
Time spent: 30.5 hrs

Sean Hinde

Sean finished OTP24 support and continued work on Mac m1 support across the codebase, worked on fixes and additional API support to the upstream erlang rocksDB C++ binding, and dug further into Sync performance.
Time spent: 39 hrs

Hans Svensson (hanssv.chain)

Limited availability - reviewed a bunch of OTP24 and Mac m1 PRs.
Time spent: 1h

Dimitar Ivanov (dimitar.chain)

Dimitar caught up with progress the past few weeks and documented some ideas regarding the hybrid consensus algorithm. He fixed a bug in devmode to allow it to be run on various protocols. He polished a bit more the peers CLI.
Time spent: 43 hours

2 Likes

thanks for update。The team did not see the hyperchain work in

report。But in the original plan; the hyperchain is most important。

@lydia write:

Q4 Roadmap

The node will be continously improved but the focus is put on the HyperChains research, design and development.

:infinity: thanks the team for their hard work and timely update of the project. :+1::+1::+1: Looking forward to the superchain update :muscle: :muscle:

1 Like

Hi @charles, please read the report carefully, several tasks are also important for Hyperchains development as far as I understand.

1 Like