Application Status
Status: New
Last updated: 02.08.2023
Submited by Sebastian Borrazas ([email protected])
Team: Rogerio, Sebastian
Approved Budget (in h):
Used Budget (in h):
Planned Delivery:
Specify the funding category
Open Source Development
Application Title
Middleware team application August 2023 to October 2023
Applicant
AEternity Middleware Development Team
Value Application
The upcoming grant will build upon the foundations established by the previous one. We will be leveraging the progress and insights gained from the previous grant to further advance our research and development efforts. This will involve building upon existing changes and exploring new ideas to continue improving Middleware.
The middleware is an indexing api for the aeternity core node, it exposed data in ways that the core node can not or is not supposed to be.
It is heavily used by various projects in the aeternity ecosystem such as the explorer, superhero wallet and dex, but available for everyone to consume or host themselves.
The overall goal of middleware is to expose all data in aeternity blockchain in a structured, filterable and developer friendly way.
Status Quo
Our work done during the previous grant includes the following:
- Add oracles /responses and /queries nested endpoints
- Track aex9 token holders count per contract, contract supplies and logs count
- List contracts and provide detail pages through /contracts and /contracts/:id to list contracts, including contracts created via PayingFor and GAMeta
- Create /wealth endpoint to list accounts with the most supply of tokens
- Add ability to filter logs by contract and event to allow better use of contract events
- Include counters to block and object broadcasts on websockets
- Encode custom event arguments
- Add ability to filter internal calls by contract and function
- Support new node 6.8.1 version
- Add channels endpoints and allow filtering them by active/inactive
- Document architecture and internal implementation of middleware to help new collaborators
- Display approximate time expiration for names and oracles
- Improve swagger docs and add support for OA3
- Add full integration tests by using devmode and calling contracts via dry-runs
- Cache in-memory mutations to greatly improve syncing speed
- Several bug fixes on the syncing logic and endpoints.
Throughout July, significant progress was made on several fronts, building upon the previously mentioned grant objectives. Notably, newer endpoints were successfully implemented, and new NFT metadata features were developed. Additionally, efforts were dedicated to improving the syncing process, which had experienced notable slowdowns due to the increased complexity of multiple contract calls in newer generations. These achievements represent important milestones in our ongoing efforts to enhance our project’s capabilities and deliver a more efficient and robust system.
Required Work
The goal of this grant is to further enhance the progress achieved through the previous grant by integrating several statistics and providing new, valuable information for client applications and users in general. Additionally, our main emphasis will be on significantly strengthening the support for AEx141 functionality, while implementing additional modifications to facilitate the seamless integration of HyperChains.
- Statistics
We will introduce statistics endpoints that efficiently filter data and provide easily accessible information, which can be readily utilised for generating informative charts.
- Adapt and display further HyperChains information and related data like commitments, parent chain info, AExN contracts, etc.
This will require Hyperchains to be fully implemented on the node and some coordination with the Node team to fully cover every new aspect provided by the Hyperchain features.
- Decouple the middleware sync/endpoints
This involves splitting Middleware into two applications to scale the endpoints without syncing in every machine. This can be done by separating the apps and spawning each one depending on configuration settings.
-
Add new V3 endpoints
- Removal of /blocks endpoint (in favour of /key-blocks and /micro-blocks)
- Filtering of AExN tokens by query parameters
- Fix display and nesting of names/auctions
- Remove transaction index (txi) support from routes and responses
- Tag the encoding of non-text binary response fields using the same encoding as the node
- Add tx-encoded transactions to /v2/txs (#1187)
-
Removal of V1 endpoints
Considering the users’ inability to remove support for V1 endpoints, we have made the decision to postpone the removal of V1 endpoints for this new grant. It is crucial to completely eliminate the consistency and performance issues associated with V1. Moreover, the process of removing these endpoints will require prior work to ensure that no known users are currently dependent on them.
- Move to newly exposed api module provided by the node
With the update of the Middleware to version 6.8.1 of the node, we now have the opportunity to utilise the new aeapi module. This module offers a subset of functions that can be more effectively managed by the node team, mitigating the risk of any disruptive changes on external applications such as Middleware. Additionally, we will ensure proper handling of the new events emitted in the latest version.
Estimate
We would be working full time on this, both Rogerio and Sebastian.
Known Limitations
Although we aim at delivering all proposed features as outlined, there may be some higher priority tasks coming up that need to be worked on by the middleware team, thus not every proposed item is expected to be delivered.
Outlook
- Indirect websocket subscription to contract logs
When a contract is executed, it emits events that can be accessed through the Middleware. To enhance this functionality, we propose to publish meta-events through a websocket endpoint. It would enable the streaming of real-time messages allowing the user to know every event emitted by a specific contract through a contract subscription. By implementing this feature, users will have seamless access to up-to-date event data directly through the websocket connection.
- MDW client (JS/Elixir/Erlang)
Right now there’s no SDK/libraries available for using the middleware. Users have to build one themselves by interacting with the endpoints directly. Our plan would be to design and implement new clients on JS, Elixir or Erlang. This way we can also include new features or fixes faster, without having to notify external users directly.
- Productization of MDW (users and request limits)
We would need to discuss a new way for users to access the middleware, by establishing some business plan that would charge those users that consume the most MDW requests.
- GraphQL
At some point it would be worth reviewing alternative ways of returning the Middleware data. GraphQL was considered as a potentially good alternative for returning very specific subsets of aggregated and nested data.
- Security auditing
In this role, the primary focus is on investigating potential exploits in Middleware systems caused by the presence of large data. This involves conducting thorough vulnerability assessments. Additionally, the task includes exploring data compaction alternatives specifically for contract logs, aiming to reduce data size while maintaining integrity.
Publishment
During development there will be new releases published and deployed continuously.
We publish our work in the official AE repositories.
Maintenance
Maintenance is part of the proposal and included for the approved timeframe.