[Active] Middleware December - March

Application Status

Status: Approved on the 30.11.2022, submitted on the 25.11.2022

Last updated: 25.11.2022

Submitted by Philipp

Team: Sebastian, Rogerio, Philipp

Approved Budget (in h):

Used Budget (in h):

Planned Delivery:

Specify the funding category

Open Source Development

Application Title

Middleware team application December 2022 - March 2023

Applicant

Aeternity Middleware Development Team

Value Application

During December and the first quarter of 2023 the middleware team will continue to maintain and improve the publicly accessible and open source elixir based middleware as used by the explorer, superhero wallet, dex and various other aeternity projects.

Definition of Terms

The middleware is an indexing api for the aeternity core node, it exposed data in ways that the core node can not or is not supposed to be.

It is heavily used by various projects in the aeternity ecosystem such as the explorer, superhero wallet and dex, but available for everyone to consume or host themselves.

The overall goal of middleware is to expose all data in aeternity blockchain in a structured, filterable and developer friendly way.

Status Quo

The middleware consists of various endpoints as documented in GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir, such as:

Middleware is in a transition phase from deprecated v1 to up to date and recommended v2 endpoints, providing better structure, faster response times and more information.

Other Notable Status Quo descriptions are:

  • Rising test coverage, established CI processes
  • Releases using semantic versioning and changelog generated from conventional commits
  • Fully available documentation in Readme and OpenApi Spec
  • Instant sync for each received micro-block

Required Work

Our planned work consists of finalizing the endpoints and features that are under active development, adding some new functionality, tackling refactorings to enable better sync and future features, as well as some tasks to explore new ideas and collaborate with the core team.

One major topic will be to plan our future support for the upcoming hyperchains proposal and hard fork.

Features

  • differentiation for different “account roles” in activities endpoint

Currently, the activities endpoint just exposes any occurrence of one account without differentiation of the account role, e.g. if it is the sender, recipient of that transaction or fulfills any other role, the goal is to group by semantic roles for each type of activity and be able to filter by them

  • more filters in activities endpoint

Currently filtering in the activities endpoint is quite limited, we want to allow the client to be able to filter e.g. by activity type and maybe other properties.

  • state channels balance updates

Currently, we are missing indexation of channel withdraw internal transactions, this requires a update to the core node to achieve. This needs to be added for overall consistency.

  • minor, yet unspecified, endpoint adjustments

Other teams are actively working features consuming middleware, smaller change requests may come in and should be discussed and shipped fairly soon.

  • resolve name for accounts in activities, e.g. spend to name should be indexed for pointer account at height

Refactorings

  • removal of v1 endpoints

Non-v2 endpoints have various inconsistencies and performance issues, this have been deprecated in the past and fully replaced by v2 endpoints, but are still used.

  • measure and profile sync, improve sync

Currently, we are unaware of specific delays in our syncing process, here we can profile our sync in detail and eventually discover quick wins and/or need for bigger improvements.

  • move to using newly introduced “rosetta” apis

The core node has introduced some new internal apis to cater to the need to match the rosetta endpoints, potentially we can use those to simplify code in the middleware for some cases.

  • use newly exposed events from core node

The core node exposes some new events, we should handle them to potentially simplify middleware or expose additional data.

  • unify reference of transactions internally

Currently, not all, especially inner/function call transactions can’t be referenced in middleware in a consistent manner, this leads to some missing data in endpoints that depend on that. This includes changes to presentation and storage/indexing improvements.

  • finalize AEX-141 support

Support for NFTs/AEX-141 is mostly finished, minor improvements may remain, there haven’t been consumers of those endpoints yet, thus we must consider minor change requests that require immediate improvements.

  • add development instrumentation to support observability and monitoring efforts

Documentation

  • finish swagger file definitions, make sure everything is consistent, handle old .json file correctly

  • ensure our readme documentation is consistent

  • improve communication on releases, e.g. what is released vs what is deployed

Operations/Planning Tasks

  • plan structure for v3 endpoints, long term support for v2

  • plan improvements discussed and requested from core team

  • plan scalability improvements, cluster deployment to decouple sync and endpoints

  • plan longer term documentation improvements

  • plan steps needed for hyperchains support, e.g. index hyperchains contract like AEX-N

SRE (out of scope for proposed dev team)

  • automatic production deployment, automatic staging/sync testing deployment

  • add monitoring and observability

Allow to monitor production app metrics, bugs, performance by developers, also automatic notification to developers for any issues.

Estimate

We would be working full time on this, although some of the team (Philipp) already have other commitments and would help us depending on their availability.

Known Limitations

Although we aim at delivering all proposed features as outlined, there may be some higher priority tasks coming up that need to be worked on by the middleware team, thus not every proposed item is expected to be delivered.

Outlook

Future work may include:

  • introduce websocket events for the activities endpoint

  • top x account endpoint

  • contract observability endpoint

  • continuously track and expose forks in chain

  • transaction pool indexing and information

  • autogeneration and verifying swagger documentation

  • documentation and eventually exposing data source in response

  • autogeneration of other documentation

  • activities endpoint for single key/micro-blocks

  • unify url and response structure for v3, as discussed above, eventually consider using graphql for some data

  • configurability for features

  • convert to plugin approach

  • allow users to use middleware user-side for state channel transactions

  • consider adding metadata handling for NFTs

  • scalability and deployment improvements, cluster deployment, decouple sync, endpoint, eventually split big db

Publishment

During development there will be new releases published and deployed continuously.

We publish our work in the official AE repositories.

Maintenance

Maintenance is part of the proposal and included for the approved timeframe.

8 Likes

Dear @philipp.chain, thank you very much for the middleware application. This Application is approved by ACF.

6 Likes

Week 1 (Dec 1 - Dec 3)

Sebastian

  • New page for some channel details that can be accessed by /v2/channels/:channel_id .
    Time spent: 16h

Rogerio

  • Housekeeping for CI and docs updating GH workflow and swagger docs
    Time spend: 7h
  • Handle nft contract and template limits
    Time spend: 7h

Week 2 (Dec 4 - Dec 10)

Sebastian

  • New page for some channel details that can be accessed by /v2/channels/:channel_id .
    Time spent: 32h
  • V2 websockets implementation with middleware aggregated data.
    Time spent: 8h

Rogerio

  • AEX-9 /v2/aex9/:contract_id/balances and /aex9/balances/:contract_id use event-based indexed balances;
  • /v2/aex9/:contract_id/balances/:account_id gets the balance from chain by calling the contract.
    Time spent: 38h

Since there is no longer a need to call AEX-9 contracts while indexing the balances, which now are event-based, a conflict on optimistic lock issue was also solved by removing the async database transaction.

2 Likes

Week 3 (Dec 11 - Dec 18)

Rogerio

  • Add new endpoint /aex141/:contract_id/templates/:template_id/tokens and update /aex141/:contract_id/templates to allow tracking NFT template edition supply in contrast with edition limits
    Time spent 28h
  • Add node info details to /v2/channels/:channel_id exposing the channel rounds (round and solo_round), locking info and participants amounts.
    Time spent 12h

Sebastian

  • Add missing model typespecs and typespec for state put/next/prev/delete calls. Time spent 15h
  • Add additional index to locate internal contract calls for Name claims/transfers/updates and Oracle register/extend 25h
3 Likes

Week 4 (Dec 19 - Dec 25)

Sebastian

  • Add additional index to locate internal contract calls for Name claims/transfers/updates and Oracle register/extend 16h
  • Include additional information on aexn activities, as per wallet’s team requests 16h
  • Add block hash to all activities 8h

Rogerio

  • Docker image build and running (compose) was improved by separating into different environments 14h
  • Contract logs returns event name when the event matches an AEX-N event hash 4h
  • Format binary pointers for inner NameUpdateTx from GaMetaTx and PayingForTx 5h
  • Update AEX-9 event balance when adding liquidity 17h
3 Likes

Week 5 (Dec 26 - Jan 1)

Sebastian

  • Remove tx hashes handling on int contract calls and include source tx_hash on nested names endpoints. 20h
  • Use master instead of latest to pull docker image. 4h
  • Started working on filtering activities by roles. 16h
2 Likes

As I was on vacation during most of December my update was left out in previous reports:

December

Philipp (15,5h)

  • Release Coordination and Sync Testing new Releases
  • Pull Request Reviews
  • Planning Next Development Steps, Coordination with Stakeholders
1 Like

Week 6 (Jan 1 - Jan 7)

Rogerio

  • Query node channel rounds and balances and expose it on channels list and details page 18h
  • Render inner transactions from ga_meta and paying_for like chain ones 16h
  • Simplify and update Mdw setup instructions according to different environments 4h
  • Code reviewing and validations 2h

Sebastian

  • Allow filtering activities by ownership only 32h

Philipp

  • Preparations and research for v1 deprecation 9,75h
  • Other development support, release coordination, sync testing 9,5h
2 Likes

Week 7 (Jan 8 - Jan 15)

Rogerio

  • Add offchain rounds to channel transactions on /txs 8h
  • Cleanup unused code and library dependencies 4h
  • Improve aex9 test coverage considering a DEX flow 4h
  • Add channel participants to channel transactions on /txs 9h
  • Fix formatting of call return composed by tuple values 8h
  • Skip importing hardfork accounts for custom networks 4h
  • Sync with core team 3h

Philipp

  • General Development Support, Coordination, Reviews, Sync Test 11h
  • V1 Deprecation 0,5h

Sebastian

  • Ignore gen-based internal transfers for txi indexed activities. 12h
  • Update node to version 6.7.0. 28h
2 Likes

Week 7 (Jan 16 - Jan 22)

Sebastian

  • Add fixes and support for Oracle and Channel internal contract calls. 14h
  • Remove no longer needed oracle queries nonce fix. 4h
  • Allow cursors with names with dashes on them on pagination. 8h
  • Adapt claim activities to use the new txi_idx stored format. 8h

Philipp

  • General Development Support, Coordination, Reviews, Sync Test 4h

Rogerio

  • Refactor websockets v2 to improve broadcasting delivery timing and use less memory 28h
  • Look for AENS name pointee also on previous expired records 2h
  • Discuss enabling tests on development environment within docker container 2h
  • Investigate Elixir resources for APM mainly considering Datadog and OpenTelemetry 8h
1 Like