[Active] Middleware Grant August 2023 - October 2023

Application Status

Status: New
Last updated: 02.08.2023
Submited by Sebastian Borrazas ([email protected])
Team: Rogerio, Sebastian
Approved Budget (in h):
Used Budget (in h):
Planned Delivery:

Specify the funding category

Open Source Development

Application Title

Middleware team application August 2023 to October 2023


AEternity Middleware Development Team

Value Application

The upcoming grant will build upon the foundations established by the previous one. We will be leveraging the progress and insights gained from the previous grant to further advance our research and development efforts. This will involve building upon existing changes and exploring new ideas to continue improving Middleware.

The middleware is an indexing api for the aeternity core node, it exposed data in ways that the core node can not or is not supposed to be.

It is heavily used by various projects in the aeternity ecosystem such as the explorer, superhero wallet and dex, but available for everyone to consume or host themselves.

The overall goal of middleware is to expose all data in aeternity blockchain in a structured, filterable and developer friendly way.

Status Quo

Our work done during the previous grant includes the following:

  • Add oracles /responses and /queries nested endpoints
  • Track aex9 token holders count per contract, contract supplies and logs count
  • List contracts and provide detail pages through /contracts and /contracts/:id to list contracts, including contracts created via PayingFor and GAMeta
  • Create /wealth endpoint to list accounts with the most supply of tokens
  • Add ability to filter logs by contract and event to allow better use of contract events
  • Include counters to block and object broadcasts on websockets
  • Encode custom event arguments
  • Add ability to filter internal calls by contract and function
  • Support new node 6.8.1 version
  • Add channels endpoints and allow filtering them by active/inactive
  • Document architecture and internal implementation of middleware to help new collaborators
  • Display approximate time expiration for names and oracles
  • Improve swagger docs and add support for OA3
  • Add full integration tests by using devmode and calling contracts via dry-runs
  • Cache in-memory mutations to greatly improve syncing speed
  • Several bug fixes on the syncing logic and endpoints.

Throughout July, significant progress was made on several fronts, building upon the previously mentioned grant objectives. Notably, newer endpoints were successfully implemented, and new NFT metadata features were developed. Additionally, efforts were dedicated to improving the syncing process, which had experienced notable slowdowns due to the increased complexity of multiple contract calls in newer generations. These achievements represent important milestones in our ongoing efforts to enhance our project’s capabilities and deliver a more efficient and robust system.

Required Work

The goal of this grant is to further enhance the progress achieved through the previous grant by integrating several statistics and providing new, valuable information for client applications and users in general. Additionally, our main emphasis will be on significantly strengthening the support for AEx141 functionality, while implementing additional modifications to facilitate the seamless integration of HyperChains.

  • Statistics

We will introduce statistics endpoints that efficiently filter data and provide easily accessible information, which can be readily utilised for generating informative charts.

  • Adapt and display further HyperChains information and related data like commitments, parent chain info, AExN contracts, etc.

This will require Hyperchains to be fully implemented on the node and some coordination with the Node team to fully cover every new aspect provided by the Hyperchain features.

  • Decouple the middleware sync/endpoints

This involves splitting Middleware into two applications to scale the endpoints without syncing in every machine. This can be done by separating the apps and spawning each one depending on configuration settings.

  • Add new V3 endpoints

    • Removal of /blocks endpoint (in favour of /key-blocks and /micro-blocks)
    • Filtering of AExN tokens by query parameters
    • Fix display and nesting of names/auctions
    • Remove transaction index (txi) support from routes and responses
    • Tag the encoding of non-text binary response fields using the same encoding as the node
    • Add tx-encoded transactions to /v2/txs (#1187)
  • Removal of V1 endpoints

Considering the users’ inability to remove support for V1 endpoints, we have made the decision to postpone the removal of V1 endpoints for this new grant. It is crucial to completely eliminate the consistency and performance issues associated with V1. Moreover, the process of removing these endpoints will require prior work to ensure that no known users are currently dependent on them.

  • Move to newly exposed api module provided by the node

With the update of the Middleware to version 6.8.1 of the node, we now have the opportunity to utilise the new aeapi module. This module offers a subset of functions that can be more effectively managed by the node team, mitigating the risk of any disruptive changes on external applications such as Middleware. Additionally, we will ensure proper handling of the new events emitted in the latest version.


We would be working full time on this, both Rogerio and Sebastian.

Known Limitations

Although we aim at delivering all proposed features as outlined, there may be some higher priority tasks coming up that need to be worked on by the middleware team, thus not every proposed item is expected to be delivered.


  • Indirect websocket subscription to contract logs

When a contract is executed, it emits events that can be accessed through the Middleware. To enhance this functionality, we propose to publish meta-events through a websocket endpoint. It would enable the streaming of real-time messages allowing the user to know every event emitted by a specific contract through a contract subscription. By implementing this feature, users will have seamless access to up-to-date event data directly through the websocket connection.

  • MDW client (JS/Elixir/Erlang)

Right now there’s no SDK/libraries available for using the middleware. Users have to build one themselves by interacting with the endpoints directly. Our plan would be to design and implement new clients on JS, Elixir or Erlang. This way we can also include new features or fixes faster, without having to notify external users directly.

  • Productization of MDW (users and request limits)

We would need to discuss a new way for users to access the middleware, by establishing some business plan that would charge those users that consume the most MDW requests.

  • GraphQL

At some point it would be worth reviewing alternative ways of returning the Middleware data. GraphQL was considered as a potentially good alternative for returning very specific subsets of aggregated and nested data.

  • Security auditing

In this role, the primary focus is on investigating potential exploits in Middleware systems caused by the presence of large data. This involves conducting thorough vulnerability assessments. Additionally, the task includes exploring data compaction alternatives specifically for contract logs, aiming to reduce data size while maintaining integrity.


During development there will be new releases published and deployed continuously.

We publish our work in the official AE repositories.


Maintenance is part of the proposal and included for the approved timeframe.


Week 1 (Aug 1 - Aug 4)


  • Add statistics and /statistics/transactions endpoint. 28h


  • Count names and oracles from keys 16h
  • Analyse AEX-N contract classification race condition 16h
  • Optimize fetching block height through the header 3h
  • Call AEX-N metainfo on contract creation block 5h

Week 2 (Aug 7 - Aug 13)


  • Add v2 yaml file back to the static directory. 8h
  • Disable cache for DB block mutations. 4h
  • Fix auction bids expiring index. 4h
  • Dedup txs activities when present on several fields. 4h
  • Add week/month interval filter on statistics. 8h
  • Add oracle swagger v2 docs endpoints and resources. 4h


  • Use aexn extensions from byte code 8h
  • Avoid sync timeout after long migrations 4h
  • Add drop tables config 8h
  • Add AEX-N type to contracts response 4h
  • Remove name cache (not needed after memory sync optimization) 8h
  • Fetch only oracles tree and remove oracle cache (not needed after memory sync optimization) 8h

Week 3 (Aug 14 - Aug 20)


  • Put revoked name key for counting 2h
  • Chore test isolating async tests and cleaning up mocked tests 2h
  • Optimization to count object keys only from memory 4h
  • Expire memory stores based on v1 heavy endpoints 8h
  • Add name history endpoint 20h
  • Return bad request for input exceptions 4h


  • Ignore field/transaction counts when they are duplicated. 12h
  • Unify convert_params usage into util function. 6h
  • Add last_tx_hash to swagger docs. 2h
  • Include aexn meta info on aexn activties. 6h
1 Like

Week 4 (Aug 21 - Aug 27)


  • Include aexn meta info on aexn activties. 6h
  • Load aex141 contract for aex141 activities. 2h
  • Add block statistics endpoint. 28h


  • Use extracted tx mod and name 2h
  • Query active entities (e.g auctions) 22h
  • Use inner tx type to render contract creation 2h
  • Prepare wealth to be updated on block basis 2h
  • Enable async migrations 8h
  • Count holders based on events 4h
1 Like

Week 5 (Aug 28 - Sep 3)


  • NFT Marketplace frontend. 8h
  • Handle aex141 templates without tokens count. 2h
  • Add approximate_auction_end_time to auctions. 6h


  • NFT Marketplace frontend. 8h
  • Aeternity Universe conference 32h
1 Like

Week 6 (Sep 4 - Sep 10)


  • Index AEX9 transfers by contract and account 4h
  • Index AEX-N contracts by creation 12h
  • Ensure suffix on name history 4h
  • Handle rendering of logs encoded on previous format 4h
  • Render allowance and approval events 8h
  • Upgrade to Node 6.11 8h
1 Like

Week 7 (Sep 11 - Sep 17)


  • Generalize txs fetching and count microblock txs 8h
  • Add aexn_type to contract txs 2h
  • Index and query aex9 transfers by contract and account 24h
  • Reverse call logs 2h


  • Display only the current auction bids for a name. 4h
1 Like

Week 8 (Sep 18 - Sep 24)


  • Offload startup by async keys loading 12h

Week 9 (Sep 25 - Oct 1)


  • Move previous names into separate table. 6h


  • Constant counting of persisted contracts 12h
1 Like

Right now there’s no SDK/libraries available for using the middleware

Actually, in @aeternity/aepp-sdk there are two classes to connect to middleware over HTTP and WebSocket.

The only issue is that types are not accurate and big numbers are not handled correctly.