Hello everyone,
after some discussions in various calls I am creating this post so we can prioritize & approve the pending refactoring topics in the backend:
1. Refactor api to fixed style
Currently the api has some inconsistencies when it comes to naming of routes or parameters. Making a more streamlined approach should make it easier for new & current developers to consume the api and discover errors in their code.
Scope
- Do some research on best practices
- Issues right now
- some routes use query params, other path params
- routes are differently named, some have an /api prefix, others do not
- some routes might be under confusing endpoints (eg. /claim/ should be /payfortx)
- and more Proposals for improving endpoint names · Issue #156 · aeternity/tipping-community-backend · GitHub
Estimates
- 2 days for research
- changes should take 1 week for backend, 1 week for consumers
2. Restructure route files
Right now route files are only providing the endpoint and documentation, this results in a lot of duplication when it comes to basic functions like sanity checking a request. It also means that a lot of route related code lands in the logic files.
Scope
- Generalize some common route tasks in a route util
- Streamline error responses & logging of said errors
- Clear logic files of route code
Estimates
1 week for everything
3. Internal refactoring of the cache logic
Initially the caching was a temporary solution to assist the middleware in its tasks until these would be fully moved back there again. Over time the caching has grown to much more than simply a temporary cache. Unfortunately by now the CacheLogic file contains all related logic although a lot of this would be better placed in the tip logic, event logic or other places.
Scope
- Basically remove “CacheLogic”
- Use cache.getOrSet(key, Logic.function()) everywhere we want to cache something
- Define a constants file to keep track of all static cache keys
- Move the related logic to existing and new controllers
Estimate
2 weeks for everything (lots of tests need to be adjusted)
4. Create event based system
The side-effects have become quite numerous over the last months. With the introduction of a vast caching infrastructure and many services depending on this we need to execute a long list of actions when a new tip is made (invalidate caches, recalculate statistics, fetch link preview, guess language etc.).
Scope
- We need to do some research / comparison of different event buses for node.js
- Instead of depending on all other services directly emit an event if something happens
- Logic can listen for events
- Right now we are NOT planning for a full message queue system, just events
Estimate
- Research: 3 days (needs some additional scoping as well)
- Implementation: 2 weeks
- Testing: TBD → At least 1 week, probably more
5. Save Blockchain data as jsonb in postgres
The data the backend fetches from the blockchain is only stored temporarily in the backend and has to be fully fetched again when we redeploy (or have to flush the cache for other reasons). Most of the data is immutable anyways unless a major fork occurs. So we would like to permanently store the events & tips in the database for super fast search & storage.
Scope
- Research jsonb handling in sequelize ORM
- Build a simple test setup to verify search & pagination & invalidation
- Save tips to database
- Save chain events to database
Estimates
- 2 days for research & test setup
- 2 weeks for implementation
6. Add a chain listener for cache invalidation
Currently we provide endpoints to the wallet / ui to invalidate caches when a user takes an action that alters the state of a contract. The new middleware has an endpoint to listen to on-chain events to automatically trigger these actions without relying for user input (and closing the attack vector).
Scope
- Verify function of middleware endpoint
- Create chain listener in backend
- Trigger Actions based on the input
Estimate
1-2 weeks based on state of middleware & ease of implementation
Notes
Would make a lot of sense in combination with the event refactoring.
7. Refactor backend to typescript
As the backend grows more complex it becomes harder for other devs to contribute to the backend & for the maintainers to verify that changes are not breaking anything. A transition to a strongly typed backend would these tasks a lot easier. This can save us a lot of time in the future.
Scope
- Setup initial build pipeline to convert typescript to js
- Add basic types
- Restructure the whole backend to work with typescript
Estimate
- 1 week for initial setup
- 4 weeks for full conversion
8. Use graphQL
As noted by various developers the backend proves a lot of unnecessary information to the clients as it just serves as a relay in many cases to get cached information from the blockchain. This generates unnecessary traffic on the client and therefor slows down the rendering & loading times. A possible solution would be to provide a graphQL endpoint so that the clients can request only the information they require.
Scope
- Research about graphQL implementation
- Create a graphQL endpoint
- Allow partial requests & build appropriate resolvers
Estimate
- Research: 1 week
- Implementation: Uncertain, can be estimated after research + previous tasks
These are the topics the backend team would like to tackle in the midterm future (now - Q1 2021) to build a strong foundation for future changes & features. Of course we can’t simply stop developing features during this time so I suggest we fit some of these in the suggested time frame but still spend at least 50% on the development of new features.