Had worked 39 hours. Mainly worked on Issue #3283 - Expose chain events in contract calls. A Work-In-Progress PR has been pushed. Events are collected and can be subscribed to as internal events. Some work is still required in order to debug the HTTP endpoint for dry-run, where chain events can now be optionally reported. Also, some review and discussion is needed regarding the format of events, and whether some additional event types should be reported.
Had worked 25 hours. He prepared infrastructure for a release and healed the nodes, he increased their disk size. He investigated the infrastructure alerts and he spent some time on GitHub issue 3301 - HTTP cache tests
Had worked 44 hours. Those were mostly spent on adding more and more tests to the aesc_utils_tests suite. I’ve identified some improvement points and added missing function specs. I’ve also prepared the 5.5.5 release PR.
Regarding maintenance the priorities of the Hyperchain projects are as follows. Maintenance tasks are grouped in 3 categories:
Required for hyperchains(order of decreasing priorities):
Onchain protocol fixes:
If a issue/bug is discovered which affects the onchain protocol then this should be fixed ASAP. This includes bugs in the FATE VM etc…
Fast synchronization:
Right now all nodes operate as “archive” nodes, most people want to quickly get the latest state so optional fast synchronization algorithms need to be implemented and provided as an option for users/node operators - after fast sync is done it would be nice to optionally sync older states(configurable policies as in geth)
Dead peer eviction:
Currently only 1% of the peers in our peer pool is alive - we need to quicker evict dead peers and validate newly gossiped peers - a queue of unchecked peers to be validated seems like a good idea.
Peer pool persistance:
Currently after the node restarts we start retrieving the peer pool from scratch - this is bad, we need to persist the peer pool and possibly after restarting revalidate the peers
Client endpoint for retrieving the peer pool status
Currently there is no easy way for a node maintainer to retrieve the list of peers in the peer pool besides attaching a console to the erlang node and writing some code… This essentially means that it is hard to analyze the status of the network and provide people with seed nodes not affiliated with the Ansalt
Decrease the reliance on centralized seed nodes for bootstraping the node - maintain and provide the community with the peer list(possibly posting the list in the forum, a smart contract etc…)
Nice to have
Regulating the naming system
Deprecate AEVM (maybe remove existing aevm contracts after a public governance vote?)
Querying remote nodes for their version
Not necessary
All bugs in the client software bundled inside the node - stratum, SC FSM, etc…
Thank you @gorbak25 for the feedback! The idea of this proposal is to help the AE Ecosystem and since the Hyperchain project is one of the most promising ones out there, we are to put your set of required issues to be with highest priority. Once we clear those we will proceed with the rest of the tasks in the proposal. I’ve created issues accordingly and included them in the maintenance project scope. Please note that the dead peers’ one is already included.
@Lydia and @Tina please consider those tasks as part of the proposal above as well.
Hi @dimitar.chain@gorbak25
Thank you for the proposal and sharing the ideas here. The foundation can prolong the maintenance contract for the next month. Please discuss and finalize the proposal which Hyperchains project can benefit from the maintenance work.
@gorbak25 for completeness regarding your bullet points from your required section:
Yes, those must be fixed ASAP, please consult the bigger chunk of the proposal above.
Yes, this would indeed be great to have. Since this is a non-technical issue, I can not track this in GitHub. Note that this depends heavily on the other peers’ tasks - the dead peers one and the endpoint sone. I consider this to be dangerous before that and we will not start doing those forum posts with seed peers before those are in place.
@Tina from the post above, the proposal is for 2 months:
As stated, I am afraid we won’t be able to solve some bigger issues, esp. regarding dead peers and the fast sync. I think the dead peers is currently the most important issue out there and it could hurt the network. We didn’t touch it in the previous grant period because I am concerned that it could easily spill over one month.
Really looking forward to get fast sync implemented - otherwise without hosting a trustful and centralized DB backup service HC stakers will need to wait weeks for sync to complete
Yup onchain bugs (with exception to deprecated features as AEVM need to be fixed ASAP).
Decentralizing the seed nodes can indeed only be done after we fix all sync related issues - I wonder how the central part of our node ended up in such bad state
One more thing I forget to add to the above list of tasks (it is “a nice to have” but not required):
upgrade Rocksdb -> This is a low hangling fruit which slows down Q/A significantly.
This will be especially true now, since we have some other non-trivial high-priority issues to tackle as well. It’s also not just just a question of man-hours: we will want to utilize the (considerable) expertise of @hanssv.chain, and he has limited availability, so this will add lead time.
Well, it’s hard to call something “working suboptimally” when in fact the node is incapable to sync with alive peers other than our seed nodes as our “ping a random node” has only a 1% chance of succeeding…
This is not fully true, as there is garbage collector that can be enabled to purge old records in the database. However, nodes cannot sync in “light” mode.
Indeed it is a bug and there had been attempts in the past to fix it. There had been some issues found both in peer pool and sync and possibly those had some positive effects but they didn’t solve the main issue there. It is not going to be trivial to validate a probable fix and this would require some involvement of @dincho.chain as well. One knows an issue is hard when you need Dincho’s assistance.
If you configure your node properly (or not at all) you will hit the seed nodes immediately, they are generously configured so I’m not sure it isn’t such a big issue as you pretend it to be. Just for the sake of this discussion I restarted my own node (had 23 inbound and 24 outbound connections)… Within seconds I had 15+ inbound connections and within a couple of minutes I had close to the numbers that I had before shutting down.
That said, it surely is high time for some improvements, it has been on the agenda several times before but other things got higher priority at the time…
@dimitar.chain and the team,
please send the foundation your new proposal with the task list (please make sure the requests from the Hyperchains are throughly discussed, prioritized).
Thank you.
I have updated PR #3353 so that it seems to pass with some very simplistic additions to the aehttp_contracts_SUITE. See the comment in the PR, and please chime in on whether you would like to see some different format in the HTTP API.
The active PR on exposing chain events in contract calls now passes tests locally. Some caches probably need updating for CI to pass. Awaiting comments on which.
A first iteration of presenting connected peers in the REST API has been pushed. Some initial comments received from @gorbak25 will be addressed in further revisions.
Also started looking at fast synch. This is a complex issue, and will take time. In parallel, some initial work was done on fixing the error arising when starting from a DB backup and then synching. The problem may actually lie in the synch code - awaiting some feedback from @hanssv.chain. Finally, some work was also done on peer pool persistence. This is also a complex issue, since a bad implementation could well make nodes more - not less - vulnerable to certain kinds of attack.
Had been working on dead peers issue. I did a through analysis and ran some tests. I’ve identified two distinct issues that could be causing the bug.
Time spent: 36.25 hours
@hanssv.chain’s contract is still not approved by the Foundation and he had not worked on this last week. @dincho.chain hadn’t contributed either as his time was consumed by another project.
can’t give feedback about the format right now. but in general I think we should be able to discover all transactions that can be performed within contract calls.
e.g. we will probably perform lots of AENS related transactions within a smart contract in future versions of aenalytics and it would be nice to be able to see what’s going on “behind the scenes” (without having to rely on contract events which can be used as workaround right now).
nice to see that this topic is finally being tackled!
The key is the so far part.