[Completed] Middleware December - March 2023

Application Status

Status: Completed on the 31.03.2022, approved on the 30.11.2022, submitted on the 25.11.2022

Last updated: 25.11.2022

Submitted by Philipp

Team: Sebastian, Rogerio, Philipp

Approved Budget (in h):

Used Budget (in h):

Planned Delivery:

Specify the funding category

Open Source Development

Application Title

Middleware team application December 2022 - March 2023

Applicant

Aeternity Middleware Development Team

Value Application

During December and the first quarter of 2023 the middleware team will continue to maintain and improve the publicly accessible and open source elixir based middleware as used by the explorer, superhero wallet, dex and various other aeternity projects.

Definition of Terms

The middleware is an indexing api for the aeternity core node, it exposed data in ways that the core node can not or is not supposed to be.

It is heavily used by various projects in the aeternity ecosystem such as the explorer, superhero wallet and dex, but available for everyone to consume or host themselves.

The overall goal of middleware is to expose all data in aeternity blockchain in a structured, filterable and developer friendly way.

Status Quo

The middleware consists of various endpoints as documented in GitHub - aeternity/ae_mdw: Aeternity Middleware in Elixir, such as:

Middleware is in a transition phase from deprecated v1 to up to date and recommended v2 endpoints, providing better structure, faster response times and more information.

Other Notable Status Quo descriptions are:

  • Rising test coverage, established CI processes
  • Releases using semantic versioning and changelog generated from conventional commits
  • Fully available documentation in Readme and OpenApi Spec
  • Instant sync for each received micro-block

Required Work

Our planned work consists of finalizing the endpoints and features that are under active development, adding some new functionality, tackling refactorings to enable better sync and future features, as well as some tasks to explore new ideas and collaborate with the core team.

One major topic will be to plan our future support for the upcoming hyperchains proposal and hard fork.

Features

  • differentiation for different “account roles” in activities endpoint

Currently, the activities endpoint just exposes any occurrence of one account without differentiation of the account role, e.g. if it is the sender, recipient of that transaction or fulfills any other role, the goal is to group by semantic roles for each type of activity and be able to filter by them

  • more filters in activities endpoint

Currently filtering in the activities endpoint is quite limited, we want to allow the client to be able to filter e.g. by activity type and maybe other properties.

  • state channels balance updates

Currently, we are missing indexation of channel withdraw internal transactions, this requires a update to the core node to achieve. This needs to be added for overall consistency.

  • minor, yet unspecified, endpoint adjustments

Other teams are actively working features consuming middleware, smaller change requests may come in and should be discussed and shipped fairly soon.

  • resolve name for accounts in activities, e.g. spend to name should be indexed for pointer account at height

Refactorings

  • removal of v1 endpoints

Non-v2 endpoints have various inconsistencies and performance issues, this have been deprecated in the past and fully replaced by v2 endpoints, but are still used.

  • measure and profile sync, improve sync

Currently, we are unaware of specific delays in our syncing process, here we can profile our sync in detail and eventually discover quick wins and/or need for bigger improvements.

  • move to using newly introduced “rosetta” apis

The core node has introduced some new internal apis to cater to the need to match the rosetta endpoints, potentially we can use those to simplify code in the middleware for some cases.

  • use newly exposed events from core node

The core node exposes some new events, we should handle them to potentially simplify middleware or expose additional data.

  • unify reference of transactions internally

Currently, not all, especially inner/function call transactions can’t be referenced in middleware in a consistent manner, this leads to some missing data in endpoints that depend on that. This includes changes to presentation and storage/indexing improvements.

  • finalize AEX-141 support

Support for NFTs/AEX-141 is mostly finished, minor improvements may remain, there haven’t been consumers of those endpoints yet, thus we must consider minor change requests that require immediate improvements.

  • add development instrumentation to support observability and monitoring efforts

Documentation

  • finish swagger file definitions, make sure everything is consistent, handle old .json file correctly

  • ensure our readme documentation is consistent

  • improve communication on releases, e.g. what is released vs what is deployed

Operations/Planning Tasks

  • plan structure for v3 endpoints, long term support for v2

  • plan improvements discussed and requested from core team

  • plan scalability improvements, cluster deployment to decouple sync and endpoints

  • plan longer term documentation improvements

  • plan steps needed for hyperchains support, e.g. index hyperchains contract like AEX-N

SRE (out of scope for proposed dev team)

  • automatic production deployment, automatic staging/sync testing deployment

  • add monitoring and observability

Allow to monitor production app metrics, bugs, performance by developers, also automatic notification to developers for any issues.

Estimate

We would be working full time on this, although some of the team (Philipp) already have other commitments and would help us depending on their availability.

Known Limitations

Although we aim at delivering all proposed features as outlined, there may be some higher priority tasks coming up that need to be worked on by the middleware team, thus not every proposed item is expected to be delivered.

Outlook

Future work may include:

  • introduce websocket events for the activities endpoint

  • top x account endpoint

  • contract observability endpoint

  • continuously track and expose forks in chain

  • transaction pool indexing and information

  • autogeneration and verifying swagger documentation

  • documentation and eventually exposing data source in response

  • autogeneration of other documentation

  • activities endpoint for single key/micro-blocks

  • unify url and response structure for v3, as discussed above, eventually consider using graphql for some data

  • configurability for features

  • convert to plugin approach

  • allow users to use middleware user-side for state channel transactions

  • consider adding metadata handling for NFTs

  • scalability and deployment improvements, cluster deployment, decouple sync, endpoint, eventually split big db

Publishment

During development there will be new releases published and deployed continuously.

We publish our work in the official AE repositories.

Maintenance

Maintenance is part of the proposal and included for the approved timeframe.

9 Likes

Dear @philipp.chain, thank you very much for the middleware application. This Application is approved by AF.

7 Likes

Week 1 (Dec 1 - Dec 3)

Sebastian

  • New page for some channel details that can be accessed by /v2/channels/:channel_id .
    Time spent: 16h

Rogerio

  • Housekeeping for CI and docs updating GH workflow and swagger docs
    Time spend: 7h
  • Handle nft contract and template limits
    Time spend: 7h

Week 2 (Dec 4 - Dec 10)

Sebastian

  • New page for some channel details that can be accessed by /v2/channels/:channel_id .
    Time spent: 32h
  • V2 websockets implementation with middleware aggregated data.
    Time spent: 8h

Rogerio

  • AEX-9 /v2/aex9/:contract_id/balances and /aex9/balances/:contract_id use event-based indexed balances;
  • /v2/aex9/:contract_id/balances/:account_id gets the balance from chain by calling the contract.
    Time spent: 38h

Since there is no longer a need to call AEX-9 contracts while indexing the balances, which now are event-based, a conflict on optimistic lock issue was also solved by removing the async database transaction.

2 Likes

Week 3 (Dec 11 - Dec 18)

Rogerio

  • Add new endpoint /aex141/:contract_id/templates/:template_id/tokens and update /aex141/:contract_id/templates to allow tracking NFT template edition supply in contrast with edition limits
    Time spent 28h
  • Add node info details to /v2/channels/:channel_id exposing the channel rounds (round and solo_round), locking info and participants amounts.
    Time spent 12h

Sebastian

  • Add missing model typespecs and typespec for state put/next/prev/delete calls. Time spent 15h
  • Add additional index to locate internal contract calls for Name claims/transfers/updates and Oracle register/extend 25h
3 Likes

Week 4 (Dec 19 - Dec 25)

Sebastian

  • Add additional index to locate internal contract calls for Name claims/transfers/updates and Oracle register/extend 16h
  • Include additional information on aexn activities, as per wallet’s team requests 16h
  • Add block hash to all activities 8h

Rogerio

  • Docker image build and running (compose) was improved by separating into different environments 14h
  • Contract logs returns event name when the event matches an AEX-N event hash 4h
  • Format binary pointers for inner NameUpdateTx from GaMetaTx and PayingForTx 5h
  • Update AEX-9 event balance when adding liquidity 17h
3 Likes

Week 5 (Dec 26 - Jan 1)

Sebastian

  • Remove tx hashes handling on int contract calls and include source tx_hash on nested names endpoints. 20h
  • Use master instead of latest to pull docker image. 4h
  • Started working on filtering activities by roles. 16h
2 Likes

As I was on vacation during most of December my update was left out in previous reports:

December

Philipp (15,5h)

  • Release Coordination and Sync Testing new Releases
  • Pull Request Reviews
  • Planning Next Development Steps, Coordination with Stakeholders
1 Like

Week 6 (Jan 1 - Jan 7)

Rogerio

  • Query node channel rounds and balances and expose it on channels list and details page 18h
  • Render inner transactions from ga_meta and paying_for like chain ones 16h
  • Simplify and update Mdw setup instructions according to different environments 4h
  • Code reviewing and validations 2h

Sebastian

  • Allow filtering activities by ownership only 32h

Philipp

  • Preparations and research for v1 deprecation 9,75h
  • Other development support, release coordination, sync testing 9,5h
2 Likes

Week 7 (Jan 8 - Jan 15)

Rogerio

  • Add offchain rounds to channel transactions on /txs 8h
  • Cleanup unused code and library dependencies 4h
  • Improve aex9 test coverage considering a DEX flow 4h
  • Add channel participants to channel transactions on /txs 9h
  • Fix formatting of call return composed by tuple values 8h
  • Skip importing hardfork accounts for custom networks 4h
  • Sync with core team 3h

Philipp

  • General Development Support, Coordination, Reviews, Sync Test 11h
  • V1 Deprecation 0,5h

Sebastian

  • Ignore gen-based internal transfers for txi indexed activities. 12h
  • Update node to version 6.7.0. 28h
2 Likes

Week 7 (Jan 16 - Jan 22)

Sebastian

  • Add fixes and support for Oracle and Channel internal contract calls. 14h
  • Remove no longer needed oracle queries nonce fix. 4h
  • Allow cursors with names with dashes on them on pagination. 8h
  • Adapt claim activities to use the new txi_idx stored format. 8h

Philipp

  • General Development Support, Coordination, Reviews, Sync Test 4h

Rogerio

  • Refactor websockets v2 to improve broadcasting delivery timing and use less memory 28h
  • Look for AENS name pointee also on previous expired records 2h
  • Discuss enabling tests on development environment within docker container 2h
  • Investigate Elixir resources for APM mainly considering Datadog and OpenTelemetry 8h
1 Like

Week 8 (Jan 23 - Jan 29)

Rogerio

  • Research observability alternatives 16h
  • Add instrumentation for metrics and errors 20h
  • Analyse some integration tests results 4h

Sebastian

  • Adapt claim actvities to use the new txi_idx stored format. 8h
  • Use proper names for transaction types for events and transactions. 12h
  • Add support for channels local index reference. 16h
  • Fix some of dialyzer overspec errors PR#1146. 4h

Philipp

  • General Development Support, Coordination, Meetings, Final Sync Test 6h
1 Like

Week 9 (Jan 30 - Feb 5)

Sebastian

  • Allow filtering by prefix and scoping contract calls. 16h
  • Update websocket integration tests. 8h
  • Allow retrieving the latest txs by hash 16h

Rogerio

  • Validate and replace json encoder 8h
  • Websockets v1 and v2 profiling and enhancements 20h
  • Analyse json logging alternatives and setup 4h
  • Docs review and update for v1 removal 6h
  • Sync with core team 2h

Philipp

  • General Development Support, Coordination, Meetings 4,25h
2 Likes

Week 10 (Feb 6 - Feb 12)

Sebastian

  • Encode blocks on the /v2/blocks endpoint using formatter. 16h
  • Add activities integration tests for the new activity types. 8h
  • Allow filtering activities by type. 16h

Philipp

  • General Development Support, Coordination, Meetings 4,75h

Rogerio

  • Profile and specialize websockets subscriptions 24h
  • Handle invalid channel update format for internal tx reference 4h
  • Investigate and handle call not found errors 8h
2 Likes

Week 11 (Feb 13 - Feb 19)

Sebastian

  • Add activities integration tests for the new activity types. 8h
  • Use mix releases for prod docker images. 32h

Rogerio

  • Add human readable encoding for events args according to AEX-N spec 12h
  • Profile websockets connections and subscriptions 8h
  • Analyse Mdw + Node behaviour on HC setup 4h
  • Review and update NFT event handling 14h

Philipp

  • General Development Support, Coordination, Meetings 3,75h
1 Like

Week 12 (Feb 20 - Feb 26)

Rogerio

  • Profile websockets connections and subscriptions 9h
  • Analyse Mdw + Node behaviour on HC setup 6h
  • Fix and improvement of release process issues 8h
  • Improve test coverage and fixes derived from that 15h
  • Sync with core team 2h

Philipp

  • General Development Support, Coordination, Meetings 1,25h

Sebastian

  • Debug oracles queries/responses on testnet/mainnet. 24h
  • Remove oracle query response check. 16h
1 Like

Week 13 (Feb 27 - Mar 5)

Sebastian

  • Avoid deleting oracle queries for later use of them. 16h
  • Set config to serve endpoints on prod. 8h
  • Automate swagger file and mdw swagger version. 12h
  • Requested docs updates. 4h

Rogerio

  • Use top height hash when fetching AEX9 balances for an account 8h
  • Release candidate tests and add fallback for contracts endpoint 8h
  • Analyse testability of mix releases and update endpoints healthcheck 8h
  • Investigate endpoints availability/vulnerabilities and cowboy setups 8h
  • General review on error handling and set proper error code 8h

Philipp

  • General Development Support, Coordination, Meetings 3,5h
1 Like

Week 14 (Mar 6 - Mar 12)

Sebastian

  • Requested docs updates. 8h
  • Make web config runtime config. 8h
  • Ignore failed ga_meta_tx sync processing. 24h

Rogerio

  • Aggregate request metrics per route 8h
  • Unify docker user with Node’s setup 8h
  • Adjust dev build to proper use of git 8h
  • Review transactions endpoint for Hyperchains 8h
1 Like

Week 15 (Mar 13 - Mar 19)

Rogerio

  • Limit ws subscriptions per connection 16h
  • Count total pending tasks without cache 3h
  • Wealth ranking 17h

Philipp

  • General Development Support, Coordination, Meetings 5,5h

Sebastian

  • Add unit tests to Db.Channel module. 12h
  • Add unit tests to TxController. 12h
  • Handle return type for failed GAMeta txs as well. 8h
2 Likes

Week 16 (Mar 20 - Mar 26)

Rogerio

  • Precompute pubkey field positions 8h
  • Count transactions id filtering per type or type group 6h
  • Migrate GaMeta to index by type 6h
  • Wealth ranking 12h
  • APM setup 2h
  • Sync with core team 2h

Philipp

  • General Development Support, Coordination, Meetings 2h

Sebastian

  • Add /oracles/:id/queries to list an oracle queries. 24h
  • Avoid reverting events and use Call.clone fnames for contracts 16h
3 Likes

Week 17 (Mar 27 - Mar 31)

Rogerio

  • Filter inner txs fields with ga_meta and paying_for type 22h
  • Sum hardfork account mintings according to the network 16h
  • Wealth ranking 2h

Philipp

  • General Development Support, Coordination, Meetings 9,25h

Sebastian

  • Avoid reverting events and use Call.clone fnames for contracts. 8h
  • Ignore oracle responses that don’t have associated query. 16h
  • Display int transfers source tx. 16h
1 Like