Application Status
Status: Completed on the 31.03.2022, approved on the 30.11.2022, submitted on the 25.11.2022
Last updated: 25.11.2022
Submitted by Philipp
Team: Sebastian, Rogerio, Philipp
Approved Budget (in h):
Used Budget (in h):
Planned Delivery:
Specify the funding category
Open Source Development
Application Title
Middleware team application December 2022 - March 2023
Applicant
Aeternity Middleware Development Team
Value Application
During December and the first quarter of 2023 the middleware team will continue to maintain and improve the publicly accessible and open source elixir based middleware as used by the explorer, superhero wallet, dex and various other aeternity projects.
Definition of Terms
The middleware is an indexing api for the aeternity core node, it exposed data in ways that the core node can not or is not supposed to be.
It is heavily used by various projects in the aeternity ecosystem such as the explorer, superhero wallet and dex, but available for everyone to consume or host themselves.
The overall goal of middleware is to expose all data in aeternity blockchain in a structured, filterable and developer friendly way.
Status Quo
The middleware consists of various endpoints as documented in GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir, such as:
- Transactions, filterable by many different parameters, such as per account GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Transaction Counts, filterable by various parameters, e.g. during certain generations, or of specific types, or addresses GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Micro/Key-Blocks, filterable by generation, such as per account GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- AENS, queryable in many ways, e.g. names in auction, names for accounts, names by pointer, name history, search GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Contract Call Logs, filterably by contract, event, data or function GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Contract Calls, exposing internal contract predefined function calls, e.g.
Chain.spend
GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir - Internal transfers, exposing implicit balance changes, such as name fees, oracle fees, block/bri rewards and tokens migrated GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Oracles, including queries and responses, filterable by various parameters GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- State channel, overview exposed, transactions filterable via txs endpoint GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- AEX-9 Fungible Tokens, filterable and queryable in various forms, exposing all in-contract information, e.g. transaction list per accounts, tokens held per account, overview of existing tokens, historic balances GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- AEX-141 Non-Fungible Tokens, filterable and queryable in various forms, exposing all in-contract information, e.g. owners overview, collections overview, transfers by account or nft GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Statistics, grouped by delta between generations, total numbers, miner statistics GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Activities, all activities of a certain account or contract on chain in a single powerful endpoint GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
- Websockets, subscribable for various global or specific data, instant availability of new data GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir
Middleware is in a transition phase from deprecated v1 to up to date and recommended v2 endpoints, providing better structure, faster response times and more information.
Other Notable Status Quo descriptions are:
- Rising test coverage, established CI processes
- Releases using semantic versioning and changelog generated from conventional commits
- Fully available documentation in Readme and OpenApi Spec
- Instant sync for each received micro-block
Required Work
Our planned work consists of finalizing the endpoints and features that are under active development, adding some new functionality, tackling refactorings to enable better sync and future features, as well as some tasks to explore new ideas and collaborate with the core team.
One major topic will be to plan our future support for the upcoming hyperchains proposal and hard fork.
Features
- differentiation for different “account roles” in activities endpoint
Currently, the activities endpoint just exposes any occurrence of one account without differentiation of the account role, e.g. if it is the sender, recipient of that transaction or fulfills any other role, the goal is to group by semantic roles for each type of activity and be able to filter by them
- more filters in activities endpoint
Currently filtering in the activities endpoint is quite limited, we want to allow the client to be able to filter e.g. by activity type and maybe other properties.
- state channels balance updates
Currently, we are missing indexation of channel withdraw internal transactions, this requires a update to the core node to achieve. This needs to be added for overall consistency.
- minor, yet unspecified, endpoint adjustments
Other teams are actively working features consuming middleware, smaller change requests may come in and should be discussed and shipped fairly soon.
- resolve name for accounts in activities, e.g. spend to name should be indexed for pointer account at height
Refactorings
- removal of v1 endpoints
Non-v2 endpoints have various inconsistencies and performance issues, this have been deprecated in the past and fully replaced by v2 endpoints, but are still used.
- measure and profile sync, improve sync
Currently, we are unaware of specific delays in our syncing process, here we can profile our sync in detail and eventually discover quick wins and/or need for bigger improvements.
- move to using newly introduced “rosetta” apis
The core node has introduced some new internal apis to cater to the need to match the rosetta endpoints, potentially we can use those to simplify code in the middleware for some cases.
- use newly exposed events from core node
The core node exposes some new events, we should handle them to potentially simplify middleware or expose additional data.
- unify reference of transactions internally
Currently, not all, especially inner/function call transactions can’t be referenced in middleware in a consistent manner, this leads to some missing data in endpoints that depend on that. This includes changes to presentation and storage/indexing improvements.
- finalize AEX-141 support
Support for NFTs/AEX-141 is mostly finished, minor improvements may remain, there haven’t been consumers of those endpoints yet, thus we must consider minor change requests that require immediate improvements.
- add development instrumentation to support observability and monitoring efforts
Documentation
-
finish swagger file definitions, make sure everything is consistent, handle old .json file correctly
-
ensure our readme documentation is consistent
-
improve communication on releases, e.g. what is released vs what is deployed
Operations/Planning Tasks
-
plan structure for v3 endpoints, long term support for v2
-
plan improvements discussed and requested from core team
-
plan scalability improvements, cluster deployment to decouple sync and endpoints
-
plan longer term documentation improvements
-
plan steps needed for hyperchains support, e.g. index hyperchains contract like AEX-N
SRE (out of scope for proposed dev team)
-
automatic production deployment, automatic staging/sync testing deployment
-
add monitoring and observability
Allow to monitor production app metrics, bugs, performance by developers, also automatic notification to developers for any issues.
Estimate
We would be working full time on this, although some of the team (Philipp) already have other commitments and would help us depending on their availability.
Known Limitations
Although we aim at delivering all proposed features as outlined, there may be some higher priority tasks coming up that need to be worked on by the middleware team, thus not every proposed item is expected to be delivered.
Outlook
Future work may include:
-
introduce websocket events for the activities endpoint
-
top x account endpoint
-
contract observability endpoint
-
continuously track and expose forks in chain
-
transaction pool indexing and information
-
autogeneration and verifying swagger documentation
-
documentation and eventually exposing data source in response
-
autogeneration of other documentation
-
activities endpoint for single key/micro-blocks
-
unify url and response structure for v3, as discussed above, eventually consider using graphql for some data
-
configurability for features
-
convert to plugin approach
-
allow users to use middleware user-side for state channel transactions
-
consider adding metadata handling for NFTs
-
scalability and deployment improvements, cluster deployment, decouple sync, endpoint, eventually split big db
Publishment
During development there will be new releases published and deployed continuously.
We publish our work in the official AE repositories.
Maintenance
Maintenance is part of the proposal and included for the approved timeframe.