Application Status
Status: Approved on the 04.04.2023, submitted on the 29.03.2023
Last updated: 29.03.2023
Submited by Sebastian ([email protected])
Team: Rogerio, Sebastian
Approved Budget (in h):
Used Budget (in h):
Planned Delivery:
Specify the funding category
Open Source Development
Application Title
Middleware team application April 2023 - June 2023
Applicant
AEternity Middleware Development Team
Value Application
The upcoming grant will build upon the foundations established by the previous one. We will be leveraging the progress and insights gained from the previous grant to further advance our research and development efforts. This will involve building upon existing changes and exploring new ideas to continue improving Middleware.
Definition of Terms
The middleware is an indexing api for the aeternity core node, it exposed data in ways that the core node can not or is not supposed to be.
It is heavily used by various projects in the aeternity ecosystem such as the explorer, superhero wallet and dex, but available for everyone to consume or host themselves.
The overall goal of middleware is to expose all data in aeternity blockchain in a structured, filterable and developer friendly way.
Status Quo
Our work done during the previous grant includes the following:
- New channels endpoint (/v2/channels/:channel_id) to retrieve a single channel information, including round details.
- List state channels with balance updates.
- Handle NFT contract and template limits.
- New template tokens endpoint on /aex141/:contract_id/templrates/:template_id/tokens to allow tracking template NFTs, including edition, owner and token ID.
- Add typespecs for database Models.
- Improve indexes to include local indexes and internal contract function calls for better referencing. Use this index to include tx_hash and source_tx_type when it corresponds.
- Handle liquidity addition on AEx9 balances.
- Filter activities by roles and types by sending additional ?type= and ?is_owner=1 query parameters.
- Render ga_meta and paying_for inner transactions likewise chain/regular ones…
- Improved test coverage, including aex9, tx routes and channels.
- Profile and restructure websocket subscriptions and make them more efficient when it comes to broadcasting messages.
- Improve docker release building by using mix releases.
- Add metrics and observability using telemetry and other profiling tools.
- Allow filtering contract calls by function name prefix.
- Adapt middleware behaviour for Hyperchains and newer versions of the node.
- Update documentation to cover for some of the requests made by users.
- Several bug fixes on the syncing logic and endpoints.
Required Work
Some of the upcoming work to be done is due to some changes on the node. These changes require a thorough analysis and implementation to ensure the smooth functioning of the system. Additionally, we will be covering some of the new features of æscan as they become available. This will involve exploring the new functionalities and providing a detailed overview of how they can be used to improve the efficiency and effectiveness of the system.
- Adapt to HyperChains and index related data like commitments, parent chain info, AEX-N contracts, etc.
This will require Hyperchains to be fully implemented on the node and some coordination with the Node team to fully cover every new aspect provided by the Hyperchain features.
- Decouple the middleware into two applications to scale the endpoints without syncing in every machine.
This can be done by separating the apps and spawning each one depending on configuration settings.
- Full name history endpoint
Users would have the ability to track all changes (including proclaims and claims) made to a name, to be able to track the state of any given name. (Github issue #1202)
- Improve endpoint performance after metrics provided by APM
Some of the metrics that are domain-specific would be helpful to track to provide a better understanding at what the syncing bottlenecks might be and how to improve them.
- Sync new events and/or improve performance to fetch them.
There’s new delta events now provided by the latest node version which we’re currently ignoring. These events contain internal account transfers and other things that are not directly linked to function calls.
-
V3 endpoints
- Removal of /blocks endpoint (in favour of /key-blocks and /micro-blocks)
- Filtering of AExN tokens by query parameters
- Fix display and nesting of names/auctions
- Remove transaction index (txi) support from routes and responses
- Tag the encoding of non-text binary response fields using the same encoding as the node
- Add tx-encoded transactions to /v2/txs (Github issue #1187)
-
Removal of V1 endpoints
Users were notified about removing support for V1 endpoints, which had consistency and performance issues. Getting rid of these will involve some previous work of making sure no known-users are currently using it.
- Add support for creating transactions on integration tests + using SDK for better integration tests
Currently, integration tests only test existing mainnet transactions. Adding support for creating our own chain with different kinds of transactions and edge-cases would improve the accuracy of tests with data directly returned by the node.
- Review architecture documentation and include it on the repo documentation
The architecture docs are from legacy systems. We would update this, including diagrams of how the middleware syncing works.
- Contract detail endpoints
Similar to how it was recently created with channels, we would include a channel detail endpoint. We would also add a nested paginated endpoint that would include all of the contract calls (both internal and external).
- Move to newly exposed api module provided by the node
Since we’ve updated the Middleware to 6.8.0 version of the node, we can now use the new aeapi module, which provides a subset of functions that can be more easily maintained by the node team to avoid any breaking changes on external applications like Middleware.
- Wealth tracking for getting the accounts with the highest balance
This might involve asynchronous tasks running to avoid impacting the performance when getting account balances.
Ability to filter activities by micro-blocks
Right now the endpoint only allows filtering by a range of generations, but not by a single micro-block.
Estimate
We would be working full time on this, both Rogerio and Sebastian.
Known Limitations
Although we aim at delivering all proposed features as outlined, there may be some higher priority tasks coming up that need to be worked on by the middleware team, thus not every proposed item is expected to be delivered.
Outlook
- MDW client (JS/Elixir/Erlang)
Right now there’s no SDK/libraries available for using the middleware. Users have to build one themselves by interacting with the endpoints directly. Our plan would be to design and implement new clients on JS, Elixir or Erlang. This way we can also include new features or fixes faster, without having to notify external users directly.
- Productization of MDW (users and request limits)
We would need to discuss a new way for users to access the middleware, by establishing some business plan that would charge those users that consume the most MDW requests.
- GraphQL as an interface
At some point it would be worth reviewing alternative ways of returning the Middleware data. GraphQL was considered as a potentially good alternative for returning very specific subsets of aggregated and nested data.
Publishment
During development there will be new releases published and deployed continuously.
We publish our work in the official AE repositories.
Maintainance
Maintenance is part of the proposal and included for the approved timeframe.
