[Completed] Middleware Grant July 2023

Application Status

Status: New
Last updated: 04.07.2023
Submited by Sebastian Borrazas ([email protected])
Team: Rogerio, Sebastian
Approved Budget (in h):
Used Budget (in h):
Planned Delivery:

Specify the funding category

Open Source Development

Application Title

Middleware team application July 2023


AEternity Middleware Development Team

Value Application

The upcoming grant will build upon the foundations established by the previous one. We will be leveraging the progress and insights gained from the previous grant to further advance our research and development efforts. This will involve building upon existing changes and exploring new ideas to continue improving Middleware.

Definition of Terms

The middleware is an indexing api for the aeternity core node, it exposed data in ways that the core node can not or is not supposed to be.

It is heavily used by various projects in the aeternity ecosystem such as the explorer, superhero wallet and dex, but available for everyone to consume or host themselves.

The overall goal of middleware is to expose all data in aeternity blockchain in a structured, filterable and developer friendly way.

Status Quo

Our work done during the previous grant includes the following:

  • Add oracles /responses and /queries nested endpoints
  • Track aex9 token holders count per contract, contract supplies and logs count
  • List contracts and provide detail pages through /contracts and /contracts/:id to list contracts, including contracts created via PayingFor and GAMeta
  • Create /wealth endpoint to list accounts with the most supply of tokens
  • Add ability to filter logs by contract and event to allow better use of contract events
  • Include counters to block and object broadcasts on websockets
  • Encode custom event arguments
  • Add ability to filter internal calls by contract and function
  • Support new node 6.8.1 version
  • Add channels endpoints and allow filtering them by active/inactive
  • Document architecture and internal implementation of middleware to help new collaborators
  • Display approximate time expiration for names and oracles
  • Improve swagger docs and add support for OA3
  • Add full integration tests by using devmode and calling contracts via dry-runs
  • Cache in-memory mutations to greatly improve syncing speed
  • Several bug fixes on the syncing logic and endpoints.

Required Work

The goal of this grant is to further enhance the progress achieved through the previous grant by integrating several statistics and providing new, valuable information for client applications and users in general. Additionally, our main emphasis will be on significantly strengthening the support for AEx141 functionality, while implementing additional modifications to facilitate the seamless integration of HyperChains.

  • Statistics

We will introduce statistics endpoints that efficiently filter data and provide easily accessible information, which can be readily utilised for generating informative charts.

  • NFT Metadata

Presently, there exist multiple methods for retrieving NFT metadata, and MDW has the potential to offer a simplified approach to accessing NFT URLs.

  • Adapt and display further HyperChains information and related data like commitments, parent chain info, AExN contracts, etc.

This will require Hyperchains to be fully implemented on the node and some coordination with the Node team to fully cover every new aspect provided by the Hyperchain features.

  • Decouple the middleware sync/endpoints

This involves splitting Middleware into two applications to scale the endpoints without syncing in every machine. This can be done by separating the apps and spawning each one depending on configuration settings.

  • Add new nested endpoints

Users will gain the capability to track all changes, encompassing proclaims and claims, made to a name, enabling them to monitor the current state of any specific name. Furthermore, they will have the ability to query all contract calls made to a contract through these nested endpoints, facilitating comprehensive access to contract-related information.

  • Add new V3 endpoints

    • Removal of /blocks endpoint (in favour of /key-blocks and /micro-blocks)
    • Filtering of AExN tokens by query parameters
    • Fix display and nesting of names/auctions
    • Remove transaction index (txi) support from routes and responses
    • Tag the encoding of non-text binary response fields using the same encoding as the node
    • Add tx-encoded transactions to /v2/txs (#1187)
  • Removal of V1 endpoints

Considering the users’ inability to remove support for V1 endpoints, we have made the decision to postpone the removal of V1 endpoints for this new grant. It is crucial to completely eliminate the consistency and performance issues associated with V1. Moreover, the process of removing these endpoints will require prior work to ensure that no known users are currently dependent on them.

Move to newly exposed api module provided by the node

With the update of the Middleware to version 6.8.1 of the node, we now have the opportunity to utilise the new aeapi module. This module offers a subset of functions that can be more effectively managed by the node team, mitigating the risk of any disruptive changes on external applications such as Middleware. Additionally, we will ensure proper handling of the new events emitted in the latest version.


We would be working full time on this, both Rogerio and Sebastian.

Known Limitations

Although we aim at delivering all proposed features as outlined, there may be some higher priority tasks coming up that need to be worked on by the middleware team, thus not every proposed item is expected to be delivered.


  • Indirect websocket subscription to contract logs

When a contract is executed, it emits events that can be accessed through the Middleware. To enhance this functionality, we propose to publish meta-events through a websocket endpoint. It would enable the streaming of real-time messages allowing the user to know every event emitted by a specific contract through a contract subscription. By implementing this feature, users will have seamless access to up-to-date event data directly through the websocket connection.

  • MDW client (JS/Elixir/Erlang)

Right now there’s no SDK/libraries available for using the middleware. Users have to build one themselves by interacting with the endpoints directly. Our plan would be to design and implement new clients on JS, Elixir or Erlang. This way we can also include new features or fixes faster, without having to notify external users directly.

  • Productization of MDW (users and request limits)

We would need to discuss a new way for users to access the middleware, by establishing some business plan that would charge those users that consume the most MDW requests.

  • GraphQL

At some point it would be worth reviewing alternative ways of returning the Middleware data. GraphQL was considered as a potentially good alternative for returning very specific subsets of aggregated and nested data.

  • Security auditing

In this role, the primary focus is on investigating potential exploits in Middleware systems caused by the presence of large data. This involves conducting thorough vulnerability assessments. Additionally, the task includes exploring data compaction alternatives specifically for contract logs, aiming to reduce data size while maintaining integrity.


During development there will be new releases published and deployed continuously.

We publish our work in the official AE repositories.


Maintenance is part of the proposal and included for the approved timeframe.


Week 1 (Jul 3 - Jul 7)


  • Profile persistent sync and disable Rocksdb cache 20h
  • Reuse node logic to get contract code 4h
  • Profile in-memory sync 16h


  • Add channels last updated time. 16h
  • Add further block times to different endpoints16h
  • Stream generations mutations. 8h

Week 2 (Jul 10 - Jul 14)


  • Improve in-memory sync performance 40h


  • Compact type increment counts mutations into a single one. 8h
  • Add block time docs to newly added places. 4h
  • Add v3 initial endpoints for names/auctions. 20h
  • Move nested name records to individual tables. 8h

Week 3 (Jul 17 - Jul 21)


  • Improve in-memory sync garbage collection and validate with 600 contract calls per sec 20h
  • Isolate test fixtures to avoid flaky results 8h
  • Fetch blocks optimization for block time 12h


  • Move nested name records to individual tables. 16h
  • Add missing fname contract creation call. 4h
  • Add devmode test for Chain.clone calls. 16h

Week 4 (Jul 24 - Jul 29)


  • Upgrade Mdw Node to 6.11 8h
  • Decode NFT metadata 8h
  • Filter owned NFTs by collection 8h
  • Detect AEX-n meta info arrangement 6h
  • Maps and streams refactored with high order functions 6h
  • Allow NFT owner call for older contracts 4h


  • Add auction-only claims v3 endpoint. 12h
  • Handle empty stats. 4h
  • Always use JSON format for swagger docs. 4h
  • Include last tx hash on delta/total stats. 8h

Week 5 (Jul 31)


  • Add statistics and /statistics/transactions endpoint. 8h