[Active] Middleware December - March

Application Status

Status: Approved on the 30.11.2022, submitted on the 25.11.2022

Last updated: 25.11.2022

Submitted by Philipp

Team: Sebastian, Rogerio, Philipp

Approved Budget (in h):

Used Budget (in h):

Planned Delivery:

Specify the funding category

Open Source Development

Application Title

Middleware team application December 2022 - March 2023

Applicant

Aeternity Middleware Development Team

Value Application

During December and the first quarter of 2023 the middleware team will continue to maintain and improve the publicly accessible and open source elixir based middleware as used by the explorer, superhero wallet, dex and various other aeternity projects.

Definition of Terms

The middleware is an indexing api for the aeternity core node, it exposed data in ways that the core node can not or is not supposed to be.

It is heavily used by various projects in the aeternity ecosystem such as the explorer, superhero wallet and dex, but available for everyone to consume or host themselves.

The overall goal of middleware is to expose all data in aeternity blockchain in a structured, filterable and developer friendly way.

Status Quo

The middleware consists of various endpoints as documented in GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir, such as:

Middleware is in a transition phase from deprecated v1 to up to date and recommended v2 endpoints, providing better structure, faster response times and more information.

Other Notable Status Quo descriptions are:

  • Rising test coverage, established CI processes
  • Releases using semantic versioning and changelog generated from conventional commits
  • Fully available documentation in Readme and OpenApi Spec
  • Instant sync for each received micro-block

Required Work

Our planned work consists of finalizing the endpoints and features that are under active development, adding some new functionality, tackling refactorings to enable better sync and future features, as well as some tasks to explore new ideas and collaborate with the core team.

One major topic will be to plan our future support for the upcoming hyperchains proposal and hard fork.

Features

  • differentiation for different “account roles” in activities endpoint

Currently, the activities endpoint just exposes any occurrence of one account without differentiation of the account role, e.g. if it is the sender, recipient of that transaction or fulfills any other role, the goal is to group by semantic roles for each type of activity and be able to filter by them

  • more filters in activities endpoint

Currently filtering in the activities endpoint is quite limited, we want to allow the client to be able to filter e.g. by activity type and maybe other properties.

  • state channels balance updates

Currently, we are missing indexation of channel withdraw internal transactions, this requires a update to the core node to achieve. This needs to be added for overall consistency.

  • minor, yet unspecified, endpoint adjustments

Other teams are actively working features consuming middleware, smaller change requests may come in and should be discussed and shipped fairly soon.

  • resolve name for accounts in activities, e.g. spend to name should be indexed for pointer account at height

Refactorings

  • removal of v1 endpoints

Non-v2 endpoints have various inconsistencies and performance issues, this have been deprecated in the past and fully replaced by v2 endpoints, but are still used.

  • measure and profile sync, improve sync

Currently, we are unaware of specific delays in our syncing process, here we can profile our sync in detail and eventually discover quick wins and/or need for bigger improvements.

  • move to using newly introduced “rosetta” apis

The core node has introduced some new internal apis to cater to the need to match the rosetta endpoints, potentially we can use those to simplify code in the middleware for some cases.

  • use newly exposed events from core node

The core node exposes some new events, we should handle them to potentially simplify middleware or expose additional data.

  • unify reference of transactions internally

Currently, not all, especially inner/function call transactions can’t be referenced in middleware in a consistent manner, this leads to some missing data in endpoints that depend on that. This includes changes to presentation and storage/indexing improvements.

  • finalize AEX-141 support

Support for NFTs/AEX-141 is mostly finished, minor improvements may remain, there haven’t been consumers of those endpoints yet, thus we must consider minor change requests that require immediate improvements.

  • add development instrumentation to support observability and monitoring efforts

Documentation

  • finish swagger file definitions, make sure everything is consistent, handle old .json file correctly

  • ensure our readme documentation is consistent

  • improve communication on releases, e.g. what is released vs what is deployed

Operations/Planning Tasks

  • plan structure for v3 endpoints, long term support for v2

  • plan improvements discussed and requested from core team

  • plan scalability improvements, cluster deployment to decouple sync and endpoints

  • plan longer term documentation improvements

  • plan steps needed for hyperchains support, e.g. index hyperchains contract like AEX-N

SRE (out of scope for proposed dev team)

  • automatic production deployment, automatic staging/sync testing deployment

  • add monitoring and observability

Allow to monitor production app metrics, bugs, performance by developers, also automatic notification to developers for any issues.

Estimate

We would be working full time on this, although some of the team (Philipp) already have other commitments and would help us depending on their availability.

Known Limitations

Although we aim at delivering all proposed features as outlined, there may be some higher priority tasks coming up that need to be worked on by the middleware team, thus not every proposed item is expected to be delivered.

Outlook

Future work may include:

  • introduce websocket events for the activities endpoint

  • top x account endpoint

  • contract observability endpoint

  • continuously track and expose forks in chain

  • transaction pool indexing and information

  • autogeneration and verifying swagger documentation

  • documentation and eventually exposing data source in response

  • autogeneration of other documentation

  • activities endpoint for single key/micro-blocks

  • unify url and response structure for v3, as discussed above, eventually consider using graphql for some data

  • configurability for features

  • convert to plugin approach

  • allow users to use middleware user-side for state channel transactions

  • consider adding metadata handling for NFTs

  • scalability and deployment improvements, cluster deployment, decouple sync, endpoint, eventually split big db

Publishment

During development there will be new releases published and deployed continuously.

We publish our work in the official AE repositories.

Maintenance

Maintenance is part of the proposal and included for the approved timeframe.

8 Likes

This post was flagged by the community and is temporarily hidden.

Dear @philipp.chain, thank you very much for the middleware application. This Application is approved by ACF.

5 Likes