[PROPOSAL]: Javascript contract calldata encoding

Preface

Blockchain applications communicate with smart contracts via binary interfaces/protocols. This a protocol that translates contract calls e.g. order_beer(7) to stream of bytes that the environment of the smart contracts (VM) can understand and execute. Also the protocol defines how return data from the smart contract calls should be decoded.

Definition of Terms

  • HTTP Compiler - a thin erlang HTTP service wrapper for aesophia compiler
  • Calldata encoding - serializing function name and arguments of a contract call to its binary representation
  • Decentralized Apps (a.k.a daepps) - My opinion is a bit different than the popular one which consider only the smart contract itself as daepp. I assume all the layers and tools a user would need to use such an app. That includes the UI of the app, it also should be de-centralized e.g. deployed to decentralized system like IPFS or ad-hoc distributed to users as a bundle. That UI should be resilient to censorship (and any other SPOF) as well.

Status Quo

Back in the days a HTTP compiler was created to easy the adoption of the daepps and to safe some time from SDK developers to implement encoding/decoding themself. Also as 2019 was very rapid development year for the (core) protocol it would have been almost impossible for SDK developers to keep it up-to-date.

However, as the protocol is much stable now, I think it’s time to move forward.

Related public discussion: En-/Decoding - a currently fatal design flaw in æternity?

Value Proposition

Daepps written in javascript should be able to encode/decode contracts calldata/results without using external service a.k.a. HTTP compiler.

There are few main issues I see by using compiler service:

  • Security - this should be considered highly trusted service as the daepps rely on it to provide encoded/decoded contract calls data. A malicious service operator can temper it all and trick the user of the daepp to sign whatever contract call they want without any way by the daepp to prevent that. This eventually will be noticed on-chain, once your account is empty already.
  • High availability - if the service is down for whatever reason the daepp is not operational anymore. Remember how coronanews.org was stuck during a townhall demo ? Yep, the compiler service was overloaded.
  • UI Centralization - by having the dependancy on the above service, it should be run somewhere server side, this automatically makes the daepp UI subject to censorship
  • Performance - as this is HTTP service, each contract call implies 2 additional service calls to the compiler (encode calldata + decode call result). This is waste of user time, bad UX and waste of server resources

First 2 issues can be kinda worked around by hosting the service “yourself”, however this is waste of time and resources and need some technical skills, so users prefer to use “our public” service which imply a lot of trust and responsibility to us.

All of the above would be solved with this proposition.

Required Work

I started thinking and validating the concept at Sofia hackathon last year. I usually fall in love with binary protocols very easy, so I continued working on this mostly on my free time.

So I already have basic encoder that supports most of the types:

  ✔ Encoder › Encode implicit init
  ✔ Encoder › Encode empty arguments
  ✔ Encoder › Encode boolean arguments
  ✔ Encoder › Encode single int arguments
  ✔ Encoder › Encode multiple int arguments
  ✔ Encoder › Encode bytes arguments
  ✔ Encoder › Encode map arguments
  ✔ Encoder › Encode tuple arguments
  ✔ Encoder › Encode nested tuple arguments
  ✔ Encoder › Encode simple variant arguments
  ✔ Encoder › Encode variant arguments with non-zero arity
  ✔ Encoder › Encode type aliases
  ✔ Encoder › Encode records
  ✔ Serializer › Serialize booleans
  ✔ Serializer › Encode unsigned
  ✔ Serializer › RLP Encode Unsigned
  ✔ Serializer › Serialize integers
  ✔ Serializer › Serialize lists
  ✔ Serializer › Serialize maps
  ✔ Serializer › Serialize tuples
  ✔ Serializer › Serialize byte array
  ✔ Serializer › Serialize string
  ✔ Serializer › Serialize bits
  ✔ Serializer › Serialize variant
  ✔ Serializer › Serialize bytes
  ✔ Serializer › Serialize address
  ✔ Serializer › Serialize contract address
  ✔ Serializer › Serialize oracle
  ✔ Serializer › Serialize oracle query
  ✔ Serializer › Serialize channel

  30 tests passed

What’s left

General tasks

  • move to org repo, add licence
  • re-organize file structure to be useful as standalone library
  • setup CI
  • change current testing framework to something that runs in a browser as well (currently using AVA)
  • re-factor for better extensibility, maintainability and eventual work paralization
  • full test coverage

Complete encoding

  • maps serialization ordering
  • nested maps
  • nested lists
  • template variables (i.e. a’ b’) use in variant types
  • better error checking

Decoding

The same that has been and will be implemented for encoding, but in reverse (bytes → JS data)

Estimate

4 man-weeks (~160h)

I can’t commit for delivery date tho, as I’ve a lot of unplanned operational tasks that popup occasionally.

I’d also welcome some dedicated time for @hanssv.chain to help me with consulting and eventual general reviews. (He loves JS, but don’t tell anyone).

Known Limitations

  • Javascript is not very nice language to work with big numbers, data structures and binary protocols. However, there are workarounds and acceptable solutions for it.
  • Changes in the serizalition protocol has to be applied in this lib as well - a.k.a maintenance, however the protocol is quite stable so far.

Out of scope

This proposal is about standalone library and SDK integration is out of scope of the proposal.

Outlook

  • better “decentralized” daepps
  • more secure daepp frontends
  • actually the daepp development, deployment and maintenance should be easier if we get rid of the HTTP compiler runtime dependancy
6 Likes

Full-support for this proposal. It makes a lot of sense to have this as an aepp developer!

5 Likes

From early in the project I’ve seen this problem of encoding/decoding call-data being misunderstood. Everyone seemed to want a general solution (like the compiler provides), however for an aepp-developer this isn’t what they need. They need to be able to encode a limited set of function calls (and its arguments) and being able to decode the responses to those calls. They don’t need a generic framework, they need some library functions that help them handle exactly their cases…

So this sounds like the right thing™ - and I’m happy to help with some advice.

2 Likes

This would be benefitial for a potential future State Channel client as well as it would make it a little less dependent on a node.

I also want to support this work, I did already discuss and approve with @milenradkov.chain to work on this myself but I’d be happy if you can work on this with more available time than I have with P1.

1 Like

I’m more than happy to get more hands on this :ok_hand:t2:

Thanks for fast approval :pray:t2:

Definitely any additional help is very welcome

“template variables (i.e. a’ b’) use in variant types” I don’t think this is the case, because entrypoints can’t be polymorphic?

1 Like

For the sake of completeness the task is about this case:

datatype reallyType('a) = Nope | No | Yep('a) | Yes
entrypoint test_variants (a: reallyType(int)) = a

This is great proposal which solve a lot of issues and which we are waiting for a long time.
Is it a good idea to sync up our work as i made the very similar job in SDK but for JS types -> Compiler like args?
Here the SDK ACI integration

5 Likes

Intermediate report

The encoding is functional done with good test coverage. All closed issues can be found in github

Next step is working on decoding part, all open issues can be found in github as well.

About 73h has been spent so far.

2 Likes

Since the last report some work on the decoder side has been done, laying out the decoding architecture and implementing int and bool decoders. See closed issues for details.

To sumarize the budget progress:

  • the last intermediate report says 73h has been spent
  • after that 19h has been spent on decoder

160-73-19 = 68h budget left

Open issues (so far) can be found in Github

Project progress can be tracked in Github board

I would like to resume my work on this project, so @vacuumtraveler.chain could you please resume the proposal work ?

5 Likes

Intermediate report

The resumption of work was relatively smooth, following tasks has been worked out last week:

  • Add support for List of int and nested deserialization
  • 09aebf0 Use class to define global serializer
  • 499b528 Add bits deserializer
  • edf8b90 Add decoder support for Objects and Bytes
  • b99527c Add String decoder support
  • 3fda876 Add ByteArray deserialization support
  • 8ecc4cb Support stream deserialization of ints

Time spent: 20h
Budget left: 48h

2 Likes

Intermediate report

Completed tasks this week:

57dca0d Add support for Map deserialization/decode
8643e38 Fix small negative int serialization
a86eaf4 Tuple deserializer and decoder support
e8a684e Support stream deserialization for bool types
f00081d Improve tests

Time spent: 13h
Budget left: 35h

I’m on vacation next week so there will be no progress.

2 Likes

I’m back and resumed the work.

2 Likes

Intermediate report

Completed task list last week:

  • 6b5a683 Add Optional decoder tests
  • ab022d5 Add support for List and Map JS types
  • 693ac0c Add records decode support
  • 0041507 Add string stream deserialization support
  • ba613cf Add Variant decoder support

Time spent: 14h
Budget left: 21h

3 Likes

Intermediate report

Completed task list this week:

  • 3836629 Support ‘tape’ test runner in the test suite
  • 2885d10 Support all types in the TypeFactory
  • 69f33d0 Add stream deserialization support in all serializers
  • 8221832 Fix string Serializer tag check
  • 2d2a87c Keep type info during map deserialization
  • 5101de1 Keep type info during list deserialization
  • f1b6265 Cleanup decoder tests of unused imports
  • c99ae66 Decode addresses as base58c strings
  • 1eaa2b6 Keep varian names during deserialization
  • bab4dc5 Keep type info during record deserialization

Time spent: 21h
Budget left: 0h

1 Like

Next steps (budget request)

Unfortunately the the budget has been exhausted and IMO there are few more steps to have production ready reliable library.

1. Complete the “public beta” milestone

The library is in future complete state. There are 3 tasks that would close this milestone:

  • BigInt browser compatibility (some progress has been made to run tests in the browser)
  • Usage documentation
  • Publish the library to npm

Milestone list

This is estimated to 40h

2. SDK integration tests

Next step would be to do basic/hackish integration of the library with the JS SDK and run the SDK test suite to find out any missing features and preliminary integration complexity. Fix any inconsistencies and add missing functionality.

This is estimated to 40h with some additional help from SDK developers.

Milestone list

please note that this is NOT actual and proper SDK integration

3. Property based testing

To ensure full futures support and find some edge cases that one usually can’t think of during standard TDD mutation testing done by @hanssv.chain should be performed.

This is estimated to 40h split to me and Hans.

Milestone list

5 Likes