[PROPOSAL][RFC] Running the most essential AEPP tooling in the browser

Hi!

The main inspiration for this post is that I was capable of finding an easy way to bring down the hosted sophia compiler service… The compiler recovered after some time and was scaled up but unfortunately it is still easy to bring down this service.

The main problem with our current ecosystem is that although our core nodes are fully decentralized and trustless, AEPP’s rely on a centralized compiler service in order to interact with sophia contracts. This has several implications for users of Aeternity, Superhero, Hyperchains:

  • AEPP users either need to trust the provided compiler http service or they need to host their own compiler in order to maintain privacy. Hosting a compiler on windows/android/ios won’t be an easy task
  • Usually AEPPS don’t allow you to change the compiler URL. This essentially means that even if you have your own compiler then you still need to trust a operator.
  • If you host your own compiler my attack vector still applies and it is easy to bring it down your compiler service…
  • It’s impossible to host an AEPP as a static site while maintaining trustlessness…
  • Wallets are incapable of decoding contract calldata in order to introspect interactions with contracts
  • There is no way to verify whether the compiler is lying to us without actually encoding call data on the client side
  • The compiler can track users, modify calldata easilly etc…

This thread proposes a new nice approach to finally fix those issues once and for all. Our saving grace is that both Aesophia and Aebytecode do not require sophisticated OTP features so they are feasible for transpilation to other languages. Previously we have anticipated that GitHub - GetFirefly/firefly: An alternative BEAM implementation, designed for WebAssembly will reach a state where aesophia and aebytecode can be compiled to WASM, but it will take Lumen still a lot of time to reach this state.
Creating an alternative “competing” implementation of aebytecode/aesophia isn’t the way to go as those implementations would need to be kept in sync…

After discussing this problem with some members of the Aeternity team we came up with a nice new approach on how to tackle this problem. PureScript is a functional language nearly identical to Haskell which can be compiled to BOTH Erlang and Javascript. PureScript is syntactically really similar to Erlang which means most existing code of Aesophia/Aebytecode can be Copy & Pasted and slightly adjusted to be valid Purescript code. Purescript also supports quickcheck which means that existing QuickCheck models can be used for testing. Because Purescript can be compiled to Erlang we can still use Purescript code in the aeternity node. My proposal is:

  1. We port the existing core parts(aeserialization, aebytecode, aesophia) of the codebase to PureScript
  2. We transpile the core parts to both JS and Erlang (and C if you want more performance)
  3. We deprecate the old Erlang only implementation and the Purescript one becomes the main source of truth

In order to show that the approach is feasible it would be nice to port at least aeserialization and aeb_fate_encoding.erl from aebytecode to Purescript, which then can be used immediately by the SDK, Superhero project etc…

I would like to see as many people as possible voicing their opinion on this topic: @yani.chain @contributor
If the ansalt agrees this proposal might become part of either the Hyperchains or Superhero project.

Best Regards,
Grzegorz Uriasz

12 Likes

I am extremely happy that this topic has been brought up.

The current state is unacceptable – most of aepps are using our provided centralized compiler service that is apparently not only responsible for compiling smart contracts, but also encoding and decoding calldata. Beside mentioned trust-point issues, @gorbak25 has created a trivial smart contract which essentially blows up the typechecker causing OOM error and effectively shuts down the compiler service which in the end makes other services like aestudio or even superhero completely frozen.

The mentioned DOS attack is not a problem with the compiler itself, but rather in the fact that our design makes it the biggest choke point of the aeternity ecosystem. At this point anybody with a little will of having fun is be able to stall most of the aepp development. We need to change that if we want to be treated seriously by our users.

Grzegorz has suggested making use of another programming language – Purescript. This was initially my idea tho. We were considering Rust and even C or JS as well, but there are some pros of PS that are extremely promising and stand as strong advantages of it;

  • Strong and static type system. Most of the bugs in the current Erlang implementation have been caused by errors that would be instantly caught by a proper typechecker. Moreover, ADT-driven type system is very much natural for the compiler implementation (abstract syntax trees, type unification, data flow controls). It is also way easier to get onboarded to a codebase that controls types. You always know what to expect from each function.
  • Pure functional – just like Erlang, but even more. PS uses almost the same paradigm and programming style as Erl, which will make it quite easy (compared to alternatives) to perform a change.
  • Rich and modern syntax – PS is very similar to Haskell, which has a lot of expressiveness. It is going to be readable and, after a little training, writable by every Haskell, Elm, Scala and (ofc) PS dev. It supports syntactic sugars for monads (the “do” notation) which will make the code look way, way better as currently we are heavily using patterns that are most naturally done via monadic stacks.
  • We are not using Erlang’s concurrency models either
  • Compiles to JS by default and exposes FFI that allows direct calls from it. SDKs will be able to access it without any overhead. Moreover, it will be trivially shipped to anyone’s PC as it doesn’t require Erlang runtime to be used.
  • There is a project that transpiles PS to Erlang. If we manage to make use of it we could get rid of current implementation and integrate it with node. Last commit is from 2 weeks, so it looks like it is maintained.

The thing that I enjoy is that it doesn’t have much dependencies – actually the only one causing some problems is aebytecode, but for decentralization reasons we want to have it rewritten as well anyway. The setup is going to be some challenge and time investment (but not very big I believe), but will free the ecosystem of several of our biggest problems – centralization, low performance and non-user-friendliness. This topic has been discussed for quite a long time – I remember I met people talking about moving compiler to the browser even a year ago, during aeternity universe event.

One concern that may rise is what are we going to do with the current code and is it going to go on waste. The truth is that even if it we deprecate it, it won’t get useless at all – it is going to be a base for the new implementation and will be useful for initial testing.

Also, it is a good opportunity to get rid of AEVM from the compiler.

8 Likes

This topic was taken on a few times without much results, I’d be happy for a solution!

6 Likes

I think this is a good initiative, but I see a couple of risks:

  • Rewriting the code in PureScript is more work than what the proposal makes it out to be. PureScript is strongly typed and Erlang is not. Copy-pasting the code and changing the syntax is not going to work.
  • Maturity of the PS to Erlang backend (I haven’t looked at it, so I don’t know its status). aeserialization and aebytecode are used in critical places in the node, so rewriting these and using a backend that generates buggy or inefficient code would be very bad.

A more modest and less risky alternative would be to implement a standalone compiler in PS and leave the node alone. This wouldn’t have to generate exactly the same bytecode as the existing compiler, so I don’t think keeping them in sync is a big problem.

6 Likes

@gorbak25 and @radrow.chain thanks for bringing that up!

This indeed is a big problem. Basically what we expect is that behind every aepp there must be a compiler that at the very least must encode and decode call data. Being slow is its least problem. As this approach introduces trust into the system (that the provided compiler is honest), it is a design flaw of all aepps out there. They are all trustful and the users have no means of verifying what call data they’re signing. This could be easily fixed by providing a JavaScript or TypeScript library that encodes and decodes call data and remove the trust from the system. That would be significantly easier than rewriting the whole compiler but it would not allow users to compile contracts, so it is a tradeoff.

We can’t expect users to host their own compilers but this shouldn’t be a show stopper for developers. Simply exposing a means to point the AEStudio to a different URL where a compiler lives should be good enough for them.

The client must be able to produce and inspect the call data, this is a no-brainer. There might be valid use cases in which the client should compile a contract, although I can not think of any on the top of my head. What we had been discussing ever since Roma release is not porting the compiler to the client but rather the VM itself. That’s what we needed the Lumen for. This would have allowed us to move State Channels in the client and that would have been great. There had been two big problems with this:

  • we need the VM to be translated to JS, WASM or something else that could be used client side. Not an impossible task but rather a big one. There had been different ideas for doing this - as a separate project (that must be kept in sync with the Erlang one), a tool to compile it to JS (that must do exactly the same, any bug in the translator would be critical) and etc.
  • contracts are executed in the context of the blockchain’s state. In order to run contracts locally one must have the latest state locally. There had been different approaches to do that in a trustless manner, none of which seemed feasible. IMO this is why we abandoned this effort.

Rewriting the compiler is significantly easier task than this and I think it is a great jump forward for the whole ecosystem. A smaller but yet really important step would be providing means for encoding and decoding the call data.

5 Likes

The JS calldata encoding/decoding project by @dincho.chain is very relevant here I think:

[ACTIVE-PROPOSAL]: Javascript contract calldata encoding

7 Likes

Client side calldata encoding and decoding is the minimum effort required to remove our accidental centralization of aepps. Developers can always compile contracts and deploy them ahead of time and users can verify the bytecode against the contract source.
The most critical part would be to finish the library started in [PROPOSAL]: Javascript contract calldata encoding or do it with PureScript.

Yesterday in my free time I did an initial investigation of PureScript:

  • The lack of binary pattern matching will make some translations not straightforward
  • The purerl backend has different type mappings than the JS backend - this means that when changing the backend we should at least change the Algebraic Data Types used
  • Algebraic data types in Purescript map really easy to JS objects - it is really easy to integrate PureScript code with JS code
  • Being a strongly typed language it is well suited for porting compiler related stuff
  • Support for quickcheck! This means that our existing models can be used to test the encoder

As a POC that this approach is feasible I will port in my free time aebytecode/aeb_fate_encoding.erl at master · aeternity/aebytecode · GitHub to Purescript.

5 Likes
  • Support for quickcheck! This means that our existing models can be used to test the encoder

Erlang QuickCheck and PureScript QuickCheck are very different, so it will not be possible to use the existing models. Some of the models might be easy to port, while others I suspect would require a non-trivial effort.

3 Likes

Yes, that sounds like a very good POC.

2 Likes

(posting to subscribe to the topic in order to get back to it when back from honeymoon - glad someone brought this up after we discussed this, or even just a critical subset of this issue, the last time one year ago: En-/Decoding - a currently fatal design flaw in æternity? - #29 by emin.chain - generally I remember going on everyone’s nerves about this since 2018. Kill the hosted compiler, kill aeternity.)

5 Likes

Ok short update on the topic. I’ve taken a very close look at Purescript and tried to convert some existing aeternity code to it.
Converting code by hand shows that after creating a thin emulation layer for constructs used in erlang the rewritten code is exactly as long as the original - as I expected one can c&p existing code and change it slightly.

There are now 2 approaches feasible:

  • Rewrite the most important stuff in Purescript.
    This isn’t an option as it will take a lot of time - Aeophia has 12k LOC so at a moderate rate of 500 LOC per day it it will take a month just to rewrite the existing code :frowning: The worst part is that even after we manage to port aesophia there are still a lot of pieces which will be missing in the browser - Fate VM etc…
  • Create an automatic Erlang → Purescript compiler
    This is an idea which I like and want to pursue :slight_smile: My estimate is that creating this compiler will take 1-3 weeks of full time work but the main advantage is that this is a one time investment - suddenly after this is done an unbelievable amount of possibilities gets unlocked: AeBytecode, AeSophia, FATE VM etc… can be easily ported and brought to the browser which in fact means that both AeProject and AeStudio can stop being toys and start to be decent.
    Progress on this transpiler can be tracked here: GitHub - erlscripten/erlscripten: Erlang to PureScript transpiler. Run Erlang in the browser! Currently I’ve figured out how to do an automatic source conversion between Erlang and Purescript(compile matches to a clause and a list of guards, run everything in the Effect monad, abuse unsafePerformEffect if needed) - but the implementation is in early stages.

    Will post updates on erlscripten in this thread - right now this is WIP and I don’t have time for it as I’m consumed by the HC project. If @YaniUnchained agrees then erlscripten will become part of the HC project and I can work on it full-time or @radrow.chain can work on it.
8 Likes

This could become something very, very meaningful. Please approve @YaniUnchained

3 Likes