Status: Approved on the 30.11.2022, submitted on the 25.11.2022
Last updated: 25.11.2022
Submitted by Philipp
Team: Sebastian, Rogerio, Philipp
Approved Budget (in h):
Used Budget (in h):
Open Source Development
Middleware team application December 2022 - March 2023
Aeternity Middleware Development Team
During December and the first quarter of 2023 the middleware team will continue to maintain and improve the publicly accessible and open source elixir based middleware as used by the explorer, superhero wallet, dex and various other aeternity projects.
The middleware is an indexing api for the aeternity core node, it exposed data in ways that the core node can not or is not supposed to be.
It is heavily used by various projects in the aeternity ecosystem such as the explorer, superhero wallet and dex, but available for everyone to consume or host themselves.
The overall goal of middleware is to expose all data in aeternity blockchain in a structured, filterable and developer friendly way.
The middleware consists of various endpoints as documented in GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir, such as:
- Transactions, filterable by many different parameters, such as per account GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Transaction Counts, filterable by various parameters, e.g. during certain generations, or of specific types, or addresses GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Micro/Key-Blocks, filterable by generation, such as per account GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- AENS, queryable in many ways, e.g. names in auction, names for accounts, names by pointer, name history, search GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Contract Call Logs, filterably by contract, event, data or function GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Contract Calls, exposing internal contract predefined function calls, e.g.
Chain.spendGitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Internal transfers, exposing implicit balance changes, such as name fees, oracle fees, block/bri rewards and tokens migrated GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Oracles, including queries and responses, filterable by various parameters GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- State channel, overview exposed, transactions filterable via txs endpoint GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- AEX-9 Fungible Tokens, filterable and queryable in various forms, exposing all in-contract information, e.g. transaction list per accounts, tokens held per account, overview of existing tokens, historic balances GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- AEX-141 Non-Fungible Tokens, filterable and queryable in various forms, exposing all in-contract information, e.g. owners overview, collections overview, transfers by account or nft GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Statistics, grouped by delta between generations, total numbers, miner statistics GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Activities, all activities of a certain account or contract on chain in a single powerful endpoint GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Websockets, subscribable for various global or specific data, instant availability of new data GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
Middleware is in a transition phase from deprecated v1 to up to date and recommended v2 endpoints, providing better structure, faster response times and more information.
Other Notable Status Quo descriptions are:
- Rising test coverage, established CI processes
- Releases using semantic versioning and changelog generated from conventional commits
- Fully available documentation in Readme and OpenApi Spec
- Instant sync for each received micro-block
Our planned work consists of finalizing the endpoints and features that are under active development, adding some new functionality, tackling refactorings to enable better sync and future features, as well as some tasks to explore new ideas and collaborate with the core team.
One major topic will be to plan our future support for the upcoming hyperchains proposal and hard fork.
- differentiation for different “account roles” in activities endpoint
Currently, the activities endpoint just exposes any occurrence of one account without differentiation of the account role, e.g. if it is the sender, recipient of that transaction or fulfills any other role, the goal is to group by semantic roles for each type of activity and be able to filter by them
- more filters in activities endpoint
Currently filtering in the activities endpoint is quite limited, we want to allow the client to be able to filter e.g. by activity type and maybe other properties.
- state channels balance updates
Currently, we are missing indexation of channel withdraw internal transactions, this requires a update to the core node to achieve. This needs to be added for overall consistency.
- minor, yet unspecified, endpoint adjustments
Other teams are actively working features consuming middleware, smaller change requests may come in and should be discussed and shipped fairly soon.
- resolve name for accounts in activities, e.g. spend to name should be indexed for pointer account at height
- removal of v1 endpoints
Non-v2 endpoints have various inconsistencies and performance issues, this have been deprecated in the past and fully replaced by v2 endpoints, but are still used.
- measure and profile sync, improve sync
Currently, we are unaware of specific delays in our syncing process, here we can profile our sync in detail and eventually discover quick wins and/or need for bigger improvements.
- move to using newly introduced “rosetta” apis
The core node has introduced some new internal apis to cater to the need to match the rosetta endpoints, potentially we can use those to simplify code in the middleware for some cases.
- use newly exposed events from core node
The core node exposes some new events, we should handle them to potentially simplify middleware or expose additional data.
- unify reference of transactions internally
Currently, not all, especially inner/function call transactions can’t be referenced in middleware in a consistent manner, this leads to some missing data in endpoints that depend on that. This includes changes to presentation and storage/indexing improvements.
- finalize AEX-141 support
Support for NFTs/AEX-141 is mostly finished, minor improvements may remain, there haven’t been consumers of those endpoints yet, thus we must consider minor change requests that require immediate improvements.
- add development instrumentation to support observability and monitoring efforts
finish swagger file definitions, make sure everything is consistent, handle old .json file correctly
ensure our readme documentation is consistent
improve communication on releases, e.g. what is released vs what is deployed
plan structure for v3 endpoints, long term support for v2
plan improvements discussed and requested from core team
plan scalability improvements, cluster deployment to decouple sync and endpoints
plan longer term documentation improvements
plan steps needed for hyperchains support, e.g. index hyperchains contract like AEX-N
automatic production deployment, automatic staging/sync testing deployment
add monitoring and observability
Allow to monitor production app metrics, bugs, performance by developers, also automatic notification to developers for any issues.
We would be working full time on this, although some of the team (Philipp) already have other commitments and would help us depending on their availability.
Although we aim at delivering all proposed features as outlined, there may be some higher priority tasks coming up that need to be worked on by the middleware team, thus not every proposed item is expected to be delivered.
Future work may include:
introduce websocket events for the activities endpoint
top x account endpoint
contract observability endpoint
continuously track and expose forks in chain
transaction pool indexing and information
autogeneration and verifying swagger documentation
documentation and eventually exposing data source in response
autogeneration of other documentation
activities endpoint for single key/micro-blocks
unify url and response structure for v3, as discussed above, eventually consider using graphql for some data
configurability for features
convert to plugin approach
allow users to use middleware user-side for state channel transactions
consider adding metadata handling for NFTs
scalability and deployment improvements, cluster deployment, decouple sync, endpoint, eventually split big db
During development there will be new releases published and deployed continuously.
We publish our work in the official AE repositories.
Maintenance is part of the proposal and included for the approved timeframe.