For some reason that took place beyond my time at aeternity, it was decided, that encoding and decoding call data should be done in the compiler. The reason back then was most likely a pragmatic one: Those who designed the language were the ones that built all the components for it, and it was all top-notch Erlang developers.
Now that we are coming close to a point of maturity, e.g. seeing a proper browser addon looming on the horizon, we face the following situation: Either
a) every user of a decentralized application running on aeternity will have to install a compiler and have it running all the time (not going to happen) or
b) we, aeternity, will have to host a compiler that will have to encode and decode all the contract call data. This is, in addition to also providing a public node infrastructure to actually execute smart contracts (both as transaction or, even more load-expensive, statically/locally), a hefty load one will have to handle.
For a generic user, this means that you not only broadcast transactions and read chain data through our public, centralized infrastructure (which is kind of okay for now, everyone can set up his own node and point the wællet browser addon to it soon), but even to do all that you will need to talk to a compiler which we will have to host for you. And this is going to cost and be another point of failure.
What do you think, shouldn’t encoding and decoding data be done in the SDK, without relying on another external component? Disclaimer: Yes this will be a crazy effort. But otherwise æternity might not retain a broad adoption, especially once the æternity foundation and hosted infrastructure is no more.
@mitch_lbw and I developed contract-support for our Java-SDK during the last weeks and observed exactly the situation you described above.
at the moment (if not explicitly implemented through the SDK itself) every SDK relies on the sophia compiler to encode contract-calls and to decode the call-results.
I personally think this is ok like it is. probably the implementation here won’t change often but if there are some changes this only needs to be adapted in the sophia-standalone-compiler. otherwise it would be necessary to adapt the changes in each SDK.
the sophia-standalone-compiler can easily be started locally. our SDK (if the new release is being published) will provide a SophiaCompilerService where the URL of the compiler-endpoint can be configured. by default this will point to the official compiler provided by æternity.
if a project seriously decides to build on top of æternity I expect it to host their own endpoint and their own node (including middleware).
for all others it should be ok to rely on the official compiler hosted by æternity. of course the team should track the load there and it might be needed to spread the load through a loadbalancer (which also shouldn’t be a big issue)
edit:
for client side implementations (e.g. the javascript sdk) it might be good if it would be able to handle encoding contract calls and decoding the results
That is the scenario I was talking about, it’s really important to point that out.
Sure, updating is done easier with a component as centralized as the compiler.
From my experience with building on Ethereum is that installing a browser extension is already almost more than you can put upon people. Everything else won’t work. This means that
we would have to host a compiler (costs !) , even every single non-statechanging function call will need to have its data encoded and decoded by our infrastructure,
absolutely everything would rely on this compiler infrastructure. If that would go down, people would not only need to find/enter a new node url, but also find some publicly usable compiler.
And using some public compiler from somewhere to encode things for you is risky as hell, anything could be injected.
For backend scenarios it’s surely a different thing. Just talking about the real actual browser daepps.
I’m personally all in for client side calldata encoder. It’s so obvious to me that it’s a mandatory feature of the clients.
A dApp already have hard dependancy on a node, even on a middleware (because of events?!), and a missing encoder adds one more dependancy and indeed on compiler service.
To not loose trust and be able to verify everything that’s happening anyone using the Blockchain should host their own node. If anyone relies on centrally hosted nodes they should also be fine to rely on a hosted compiler.
One could even think of a service that provides x requests free and charges after those and could also provide an SLA.
If one doesn’t want to rely on this hosted compiler it is as easy as running a node for them to self-host their compiler, it can easily be bundled as well using e.g. docker-compose.
I don’t feel it is a design flaw to have this distinction. Sure it may be easier to have the compiler as direct dependency in a webpage is bundle, but on the other hand building the compiler the way we have it did allow for a quicker release of the ecosystem as we could rely on components from the node and vm that were defined in Erlang previously.
For any dev-team it is only a matter of reimplementing existing code in another language, not reengineering the compiler, so I feel it is a good task for the one who depends on having it.
as easy as running a node for them to self-host their compiler
The case I’m more concerned about is the 9x% of users that will only causally interact with Aeternity and are barely able to send an email, let alone fiddle around with docker containers. Projects building on aeternity will expect to have a way to serve these users as easy as it’s possible with Ethereum and MetaMask right now, and requiring projects to ask people to use their compiler is a practice we should not foster, for security reasons mentioned above. But in general, I agree with the opportunities of a freemium service model for hosted infrastructure, but looking at the other protocols that’s not something we should go ahead now with: Serving a limited functionality and charging people in order to overcome the hurdles we put in place ourselves.
I totally agree with you. We should think of the 9x% of the users. But it they are fine and secure enough using a hosted node they should also be fine to use a hosted compiler, as it has similar security risks.
I don’t expect there to be any major issues re-implementing encoding/decoding of calldata. Porting the compiler should also be straightforward - there shouldn’t be anything that is coupled with the BEAM but certainly a tad bit of Erlang knowledge will help the translation
I think compiler is not an issue, because developers mostly will write, test and compile programs on the computers they control so there is no problem or bottleneck for that.
But decoupled encoding/decoding calldata from compiler is a different issue and there might be some benefits for that, especially if competent blockchains offered it from the beginning where you can use a JavaScript library to do it (web3, ether.js, etc I guess for Ethereum).
If you want to do encode/decode properly for an unknown contract I think you will need substantial parts of the compiler. But encode/decode for a known contract (and also unknown contracts with simple types) is not a huge effort.
I knew this. I saw this when I built my version of the middleware and realized there was information from contracts that I couldn’t read unless it was decoded too, which in turn meant making a connection to another server, calling another program or building an extensive decoder that I kept thinking should be part of the sdk but for some reason wasn’t there.
I think adding a decoder to the node is not a good idea because compilers are historically known for having vulnerabilities and exposing this port publicly could be a new attack vector for the network (this threat to me is more real than the flood of the network or the lack of miners, both of which never happened) . I guarantee that adding a port or endpoint that has access to a cpu intensive function will result in abuse and DDoS on the aeternity nodes.
I found this out quite recently and I feel like I am missing something.
For me a compiler is something you use before you start using your program. In this sense it makes 0 sense to me to use a compiler for anything during the workings of a program.
Imagine a golang compiled program to constantly ask the go compiler to do stuff for it to run. It is both slow and introduces another point of failure.
I do understand the reasons why this has been done, but my strong opinion is that everyone should be very clear that this is just a temporary workaround until it is implemented in the SDKs, and this should be at the very top of the SDKs backlog. Compilers are for developers.
I feel there is a misconception in communication here. You don’t need a compiler for anything else than compiling a smart contract. But currently the only implementation of a encoding/decoding helper is in the compiler due to the ease of doing it in Erlang. There is nothing that keeps a js developer away from implementing this in the sdk.
I’m curious though what we are more likely to see first: Someone with the skills in theoretical computer science, Erland and JS, or finally some actually working web assembly that is being hyped upon for years already, in which we can make the compiler run in a Rust implementation of the Erlang VM… sadly, I’m not really confident about any of these options, someone correct me, please
The problem is not the Erlang, but to understand what’s actually going on in there. Therefore you’d need to spend a lot of time with our Swedishes I guess…
