[Proposal] Refactor some core backend logic to allow for faster backend development

Hello everyone,

after some discussions in various calls I am creating this post so we can prioritize & approve the pending refactoring topics in the backend:

1. Refactor api to fixed style

Currently the api has some inconsistencies when it comes to naming of routes or parameters. Making a more streamlined approach should make it easier for new & current developers to consume the api and discover errors in their code.

Scope

  • Do some research on best practices
  • Issues right now

Estimates

  • 2 days for research
  • changes should take 1 week for backend, 1 week for consumers

2. Restructure route files

Right now route files are only providing the endpoint and documentation, this results in a lot of duplication when it comes to basic functions like sanity checking a request. It also means that a lot of route related code lands in the logic files.

Scope

  • Generalize some common route tasks in a route util
  • Streamline error responses & logging of said errors
  • Clear logic files of route code

Estimates

1 week for everything

3. Internal refactoring of the cache logic

Initially the caching was a temporary solution to assist the middleware in its tasks until these would be fully moved back there again. Over time the caching has grown to much more than simply a temporary cache. Unfortunately by now the CacheLogic file contains all related logic although a lot of this would be better placed in the tip logic, event logic or other places.

Scope

  • Basically remove “CacheLogic”
  • Use cache.getOrSet(key, Logic.function()) everywhere we want to cache something
  • Define a constants file to keep track of all static cache keys
  • Move the related logic to existing and new controllers

Estimate

2 weeks for everything (lots of tests need to be adjusted)

4. Create event based system

The side-effects have become quite numerous over the last months. With the introduction of a vast caching infrastructure and many services depending on this we need to execute a long list of actions when a new tip is made (invalidate caches, recalculate statistics, fetch link preview, guess language etc.).

Scope

  • We need to do some research / comparison of different event buses for node.js
  • Instead of depending on all other services directly emit an event if something happens
  • Logic can listen for events
  • Right now we are NOT planning for a full message queue system, just events

Estimate

  • Research: 3 days (needs some additional scoping as well)
  • Implementation: 2 weeks
  • Testing: TBD --> At least 1 week, probably more

5. Save Blockchain data as jsonb in postgres

The data the backend fetches from the blockchain is only stored temporarily in the backend and has to be fully fetched again when we redeploy (or have to flush the cache for other reasons). Most of the data is immutable anyways unless a major fork occurs. So we would like to permanently store the events & tips in the database for super fast search & storage.

Scope

  • Research jsonb handling in sequelize ORM
  • Build a simple test setup to verify search & pagination & invalidation
  • Save tips to database
  • Save chain events to database

Estimates

  • 2 days for research & test setup
  • 2 weeks for implementation

6. Add a chain listener for cache invalidation

Currently we provide endpoints to the wallet / ui to invalidate caches when a user takes an action that alters the state of a contract. The new middleware has an endpoint to listen to on-chain events to automatically trigger these actions without relying for user input (and closing the attack vector).

Scope

  • Verify function of middleware endpoint
  • Create chain listener in backend
  • Trigger Actions based on the input

Estimate

1-2 weeks based on state of middleware & ease of implementation

Notes

Would make a lot of sense in combination with the event refactoring.

7. Refactor backend to typescript

As the backend grows more complex it becomes harder for other devs to contribute to the backend & for the maintainers to verify that changes are not breaking anything. A transition to a strongly typed backend would these tasks a lot easier. This can save us a lot of time in the future.

Scope

  • Setup initial build pipeline to convert typescript to js
  • Add basic types
  • Restructure the whole backend to work with typescript

Estimate

  • 1 week for initial setup
  • 4 weeks for full conversion

8. Use graphQL

As noted by various developers the backend proves a lot of unnecessary information to the clients as it just serves as a relay in many cases to get cached information from the blockchain. This generates unnecessary traffic on the client and therefor slows down the rendering & loading times. A possible solution would be to provide a graphQL endpoint so that the clients can request only the information they require.

Scope

  • Research about graphQL implementation
  • Create a graphQL endpoint
  • Allow partial requests & build appropriate resolvers

Estimate

  • Research: 1 week
  • Implementation: Uncertain, can be estimated after research + previous tasks

These are the topics the backend team would like to tackle in the midterm future (now - Q1 2021) to build a strong foundation for future changes & features. Of course we can’t simply stop developing features during this time so I suggest we fit some of these in the suggested time frame but still spend at least 50% on the development of new features.

CC: @milenradkov.chain @davidyuk @philipp.chain @maxkpower

As I heavily discussed all of these points with @keno.chain I can only agree with them.

They are summarizing the technical dept that we accumulated by a quick and feature oriented development approach and got too big to fix quietly.

1 Like