Transaction broadcasted and then not found

filip · March 10, 2021, 10:24pm

Hey everyone,

I called the smart contract on the mainnet using js-sdk. After receiving txHash I awaited for the confirmation using waitTxConfirm(hash) sdk function. Then after 1 confirmation , I get tx not found when querying for the hash.

contract call result:

result {
  hash: 'th_N7G5dWz37z9S7Y5BeFQLinbS8wHmCQz5vw4YUvhKv9T3jE3Bz',
  rawTx: 'tx_+M8LAfhCuECjFoQ0+53xZfWQQwGOFXJjM6EkR/03RcKIuuiDAEUCGQLR/mUaGQmEFf3Dr1dkwqdTs5h00HqV2NqCu8NqZqYMuIf4hSsBoQF7YX+i6NHksXm84S2CbftFuVDwlUQtUDM6obp+Mxm+vBGhBZB9g8MTrxWORZZaF9lmtDHRgJtsebWAZIyM8aGcvW+UA4amKiiZuAAAAIMYF/iEO5rKAKorEUzXC5YbnwCgFjlt1rf7HwH32MH58v+LNJCDW1+SWsqUGPAIDDxvE5CWG2Ts',
  result: {
    callerId: 'ak_wLbhjhs2TZ3vztaK9hb8rKRDBjip3N4xqDKLtAhvcwRMyCUmA',
    callerNonce: 17,
    contractId: 'ct_26dp1nJUjMrbpbGdvduwA4J8FsYVU5uEBXwq7xwoSmyHxaRgYL',
    gasPrice: 1000000000,
    gasUsed: 1887,
    height: 397965,
    log: [ [Object] ],
    returnType: 'ok',
    returnValue: 'cb_P4fvHVw='
  },
  txData: {
    blockHash: 'mh_2S3fEPFn1LN5pN79mU96VcCAC3Rxa4ngszaJ5Raii3gvBbiS7c',
    blockHeight: 397965,
    hash: 'th_N7G5dWz37z9S7Y5BeFQLinbS8wHmCQz5vw4YUvhKv9T3jE3Bz',
    signatures: [
      'sg_NLX9XkEbApNG2jj5Nnr5HgUpRG4KNTgCicAPUmgEPy9xBAW2THa48Vre4ckNCPtnyh32i6uwTydQFot9ddARwTHbQbLjk'
    ],
    tx: {
      abiVersion: 3,
      amount: 0,
      callData: 'cb_KxFM1wuWG58AoBY5bda3+x8B99jB+fL/izSQg1tfklrKlBjwCAw8bxOQ1BPABQ==',
      callerId: 'ak_wLbhjhs2TZ3vztaK9hb8rKRDBjip3N4xqDKLtAhvcwRMyCUmA',
      contractId: 'ct_26dp1nJUjMrbpbGdvduwA4J8FsYVU5uEBXwq7xwoSmyHxaRgYL',
      fee: 182700000000000,
      gas: 1579000,
      gasPrice: 1000000000,
      nonce: 17,
      type: 'ContractCallTx',
      version: 1
    },
    callerId: 'ak_wLbhjhs2TZ3vztaK9hb8rKRDBjip3N4xqDKLtAhvcwRMyCUmA',
    callerNonce: 17,
    contractId: 'ct_26dp1nJUjMrbpbGdvduwA4J8FsYVU5uEBXwq7xwoSmyHxaRgYL',
    gasPrice: 1000000000,
    gasUsed: 1887,
    height: 397965,
    log: [ [Object] ],
    returnType: 'ok',
    returnValue: 'cb_P4fvHVw=',
    rawTx: 'tx_+M8LAfhCuECjFoQ0+53xZfWQQwGOFXJjM6EkR/03RcKIuuiDAEUCGQLR/mUaGQmEFf3Dr1dkwqdTs5h00HqV2NqCu8NqZqYMuIf4hSsBoQF7YX+i6NHksXm84S2CbftFuVDwlUQtUDM6obp+Mxm+vBGhBZB9g8MTrxWORZZaF9lmtDHRgJtsebWAZIyM8aGcvW+UA4amKiiZuAAAAIMYF/iEO5rKAKorEUzXC5YbnwCgFjlt1rf7HwH32MH58v+LNJCDW1+SWsqUGPAIDDxvE5CWG2Ts'
  },
  decode: [Function: decode],
  decodedResult: [],
  decodedEvents: [
    {
      address: 'ct_26dp1nJUjMrbpbGdvduwA4J8FsYVU5uEBXwq7xwoSmyHxaRgYL',
      data: 'cb_Xfbg4g==',
      topics: [Array],
      name: 'WalletAdded',
      decoded: [Array]
    }
  ]
}

await confirmation

await client.waitForTxConfirm("th_N7G5dWz37z9S7Y5BeFQLinbS8wHmCQz5vw4YUvhKv9T3jE3Bz", { confirm: 1, interval: 20000, attempts: 40 })
height = await client.height()
console.log("fetched height", height)

The height output is: 397966.
But I queried for the hash and received the Tx Not Found status. I looked for the micro block and I can see it here: mh_2S3fEPFn1LN5pN79mU96VcCAC3Rxa4ngszaJ5Raii3gvBbiS7c - Micro block transactions of Aeternity - AEKnow
My transaction is there but can’t open the details, the details are: NULL.

What is the issue here? And what is the proper way for awaiting the transaction to be mined? Thanks!

Baixin.chain · March 11, 2021, 3:54am

@gorbak25
Right now the data on the chain often fails, the failure rate is very high, you can look at, for example, the loop calls a contract 100 times, after waiting for a block how many failed.

filip · March 11, 2021, 7:17am

I’ve noticed. Can someone from the team take a look? I’ll provide info or help in any way I can…

dimitar.chain · March 11, 2021, 1:51pm

This test in particular can be really misleading: basic accounts (non-generalised ones) have a monotonically incrementing nonce . The first transaction ever from this account has nonce=1, the second one - 2 and so on. The protocol forbids “skipping” nonces - if the account’s nonce is 42 the next transaction included must have a nonce 43 and any transaction with a bigger nonce will be waiting for it before it gets included. Any transaction with a nonce ≤ 42 will be outright invalid. The node has a setting that if it receives a nonce too big - it will be rejected. It is up to the node operator to use a meaningful value there, the default one is 5.

Back to the proposed test: if you create a burst of 100 transactions, chances are that those will reach miners in a non-deterministic order - even if you send those to your node sequentially, your node will gossip them to other nodes in a non-predictable order, they will gossip them to other nodes in the same way and etc. The chances are that they will not arrive on a particular node in the order they were published, meaning that some will have a difference with current nonce greater than the setting (again, by default - 5) and will be rejected by this receiving node. Then on every ping the nodes that still have the transactions will push them to other nodes via gossip and some will accept them and gossip them further. A transaction will be included once it is received by the current generation leader. @gorbak25 @radrow.chain @cytadela8 @uwiger - this setting seems quite counter intuitive for users, I suggest we remove it (it is not in the scope of the protocol).

I am tracking what had happened. I am still investigating but so far the story goes like this:

@filip posted a contract call and got his first confirmation. First confirmation is the transaction being included in a micro block. BitcoinNG has a lot of perks but one of its drawbacks is that micro blocks can get evicted from the blockchain, especially at the end of the generation. Basically those micro blocks had not yet reached the new leader before one mines the next key block. Those are expected and even have a name - microforks. If your transaction ended in a microfork, then it was kicked out of the blockchain and is expected to get included in a subsequent micro block, unless it gets invalidated. Posting a transaction from the same account and the same nonce basically creates two competing transactions, one invalidating the other. This is exactly what had happened when @filip had produced a competing transaction with the same nonce. It was included so the transaction above has no chance of being included. What is curious is that the second transaction with the same nonce (the one that eventually got included) was included only 3.5 hours after the rejection of the first one, so the first one had plenty of time to get re-included. I am investigating this now.

When a transaction is invalid or it had lingered in the mempool for too long (this is a node setting, 256 generations by default) - it gets GCed. You will not see GCed transactions in the HTTP endpoints:

$ curl -s http://13.48.43.98:3013/v2/transactions/th_N7G5dWz37z9S7Y5BeFQLinbS8wHmCQz5vw4YUvhKv9T3jE3Bz
{"reason":"Transaction not found"}%

If you have an access to a node, though, you can verify it is there:


(aeternity@localhost)7> TxBin = <<"th_N7G5dWz37z9S7Y5BeFQLinbS8wHmCQz5vw4YUvhKv9T3jE3Bz">>.
<<"th_N7G5dWz37z9S7Y5BeFQLinbS8wHmCQz5vw4YUvhKv9T3jE3Bz">>
(aeternity@localhost)8> {ok, TxHash} = aeser_api_encoder:safe_decode(tx_hash, TxBin).
{ok,<<47,237,154,140,180,81,7,41,46,132,70,95,226,103,
      112,97,38,52,63,199,26,208,197,118,100,34,230,...>>}
(aeternity@localhost)9> mnesia:dirty_read(aec_signed_tx, TxHash).
[{aec_signed_tx,<<47,237,154,140,180,81,7,41,46,132,70,95,
                  226,103,112,97,38,52,63,199,26,208,197,
                  118,100,34,...>>,
                {signed_tx,{aetx,contract_call_tx,aect_call_tx,135,
                                 {contract_call_tx,{id,account,
                                                       <<123,97,127,162,232,209,228,177,121,188,225,45,130,...>>},
                                                   17,
                                                   {id,contract,
                                                       <<144,125,131,195,19,175,21,142,69,150,90,...>>},
                                                   3,182700000000000,0,0,1579000,1000000000,
                                                   <<43,17,76,215,11,150,27,...>>,
                                                   [],
                                                   <<123,97,127,162,232,...>>}},
                           [<<163,22,132,52,251,157,241,101,245,144,67,1,142,21,
                              114,99,51,161,36,71,253,...>>]}}]
(aeternity@localhost)10>

Maybe it is worth discussing if we want to keep the garbage collector on for somewhat valid transactions (at least not obviously invalid ones), cc @gorbak25 @radrow.chain @cytadela8 @uwiger

LiuYang.chain · March 11, 2021, 2:08pm

With golang SDK, txs with long ttl seem to be mined at last, after several microblocks re-org.

Aeknow records the microblocks info when they were mined at the first time…so if the txs recoded in aeknow had missed, they might be dropped by the miner, since the keyblocks are normal.

LiuYang.chain · March 11, 2021, 2:17pm

The re-org of microblocks happened in the past, but not in such a high rate as recently reported, especially from users using js-sdk.

As @Baixin.chain mentioned, aebox’s user reported so many cases these days.

It barely works with golang SDK, such as aeknow.chain wallet, but the rolling back of microblocks can be watched from time to time.

filip · March 11, 2021, 3:50pm

@dimitar.chain thank you for such a detailed explanation and taking the time to investigate this!
Things are more clear now.

Knowing all of this now, I have a few questions:

After posting the transaction, what is the correct way then to await for transaction to actually get included in the chain? Should I just wait for every new key block and try to fetch the tx info (as you’ve said, this lasted for 3 hours in my case)? Is there a proper way to do this using js-sdk?
If I’m creating a contract , I’m only safe to interact with the contract once the creation transaction is provably included in the chain?
If I’d like to post more then 1 transaction from the same wallet (transactions are not related and order of execution does not matter) can I do this safely? Or I have to go one by one, one per keyblock?

Your help is much appreciated, thanks…

filip · March 12, 2021, 11:21am

Oh and one more thing , can you shed some light on the following?

I posted a successful contract create transaction (20 block confirmations as I’m typing). Contract deployed at ct_wiTGR8dXphVwUP9DMwDGuY7Hyi8jKyfDPQSuZD4KSc7oU5feX .
I can’t interact with the contract though as I get an error: contract not found, see here and here.

Am I missing something? @dimitar.chain can you please have a look, or someone from the team? I’d like to have a full understanding of the processes behind so I can improve my aepp ux

ae-omar · March 12, 2021, 12:10pm

Hi @filip your contract seems to exist, maybe you can share how you are interacting with it?

ae-omar · March 12, 2021, 12:11pm

@gorbak25 @uwiger this could be done because of unstable hashrate?

filip · March 12, 2021, 1:38pm

Node says it doesn’t
I’ve also checked by querying my own mainnet node to make sure there’s no middleware or something cached in front of the official one, again: contract not found. The weird thing is I can query contract creation transaction by hash and everything looks perfectly fine - 50 block confirmations and counting.

Anyways, I’m doing the basic interaction with the contract over the js-sdk , as always. I load up the contract instance at the deployed address and call functions. Here is a snippet with responses included:

let keypair = Crypto.generateKeyPair()
let node = await Node({
    url: "http://mainnet.aeternity.io",
    internalUrl: "http://mainnet.aeternity.io"
})

let client = await Ae({
    nodes: [
        { name: "node", instance: node } 
    ],
    compilerUrl: "https://latest.compiler.aepps.com",
    accounts: [
        MemoryAccount({ keypair })
    ],
    address: keypair.publicKey,
    networkId: "ae_mainnet"
})

let contractInstance = await client.getContractInstance(contracts.projSource, {
    contractAddress: "ct_wiTGR8dXphVwUP9DMwDGuY7Hyi8jKyfDPQSuZD4KSc7oU5feX"
})
let owner = await contractInstance.call('owner')
let ownerDecoded = await owner.decode()
console.log(`Owner: ${ownerDecoded}`)

And a very clear error message when running code above:

Error: Contract with address ct_wiTGR8dXphVwUP9DMwDGuY7Hyi8jKyfDPQSuZD4KSc7oU5feX not found on-chain or not active
      at Object.<anonymous> (node_modules/@aeternity/aepp-sdk/dist/aepp-sdk.js:1:197138)
      at s (node_modules/@aeternity/aepp-sdk/dist/aepp-sdk.js:1:18015)
      at Generator._invoke (node_modules/@aeternity/aepp-sdk/dist/aepp-sdk.js:1:17768)
      at Generator.forEach.t.<computed> [as next] (node_modules/@aeternity/aepp-sdk/dist/aepp-sdk.js:1:18372)
      at n (node_modules/@aeternity/aepp-sdk/dist/aepp-sdk.js:1:2458)
      at i (node_modules/@aeternity/aepp-sdk/dist/aepp-sdk.js:1:2669)
      at processTicksAndRejections (internal/process/task_queues.js:93:5)

ae-omar · March 12, 2021, 2:02pm

Thanks @filip. Maybe @philipp.chain or @nikitafuchs.chain can shed some light here ?

hanssv.chain · March 12, 2021, 4:01pm

Did the contract create transaction actually succeed (if the arguments to the init function are bad, etc.) the ContractCreataTx might exist but no contract is created…

Hint: no it did not

So yes there is a contract create tx but it was bad so no there is no contract

filip · March 12, 2021, 4:13pm

Awesome, I overlooked this one! Thanks
Do you have a suggestion on the proper way of handling the transactions state - few questions from my previous reply?

hanssv.chain · March 12, 2021, 4:29pm

Javascript? No, not my thing

filip · March 12, 2021, 4:30pm

You lucky dude!
Ok, someone will jump in I hope

nikitafuchs.chain · March 14, 2021, 3:29pm

What exactly do you want to know ? How to distinguish successful and failed contract deployments?

filip · March 14, 2021, 5:24pm

Hey Nikita! Here’s a c/p from one of my previous replies:

After posting the transaction, what is the correct way then to await for transaction to actually get included in the chain? Should I just wait for every new key block and try to fetch the tx info (as you’ve said, this lasted for 3 hours in my case)? Is there a proper way to do this using js-sdk?
If I’m creating a contract , I’m only safe to interact with the contract once the creation transaction is provably included in the chain?
If I’d like to post more then 1 transaction from the same wallet (transactions are not related and order of execution does not matter) can I do this safely? Or I have to go one by one, one per keyblock?

Don’t know if these are clear enough, or actually make any sense… Thanks for any help or info you can provide!

nikitafuchs.chain · March 15, 2021, 8:28am

easy

After posting the transaction, what is the correct way then to await for transaction to actually get included in the chain?

If you use the JS SDK’s native way of calling contracts, it returns a promise once the transaction has successfully been included in a microblock, no need to manually wait for keyblocks here except if you want to make sure you have a minimum block depth.

If I’m creating a contract , I’m only safe to interact with the contract once the creation transaction is provably included in the chain?

Yes

If I’d like to post more then 1 transaction from the same wallet…

…then you need to wait for the first one to get mined before you send the second one, because the nonce of each transaction is unique and must get mined in a consecutive order. So tx nonce 2 must come after 1, or else it will get rejected.

A hint for the first question: I’ve included a code generator in AEstudio, you can use its output as a working example:

Deploy a contract, 2. scroll to your function of choice and enter its parameters if necessary, 3. click:

Then you can chose to see the code for SDK setup, contract deployment and function calling , or deactivate tickboxes at the top to see only the desired part (e.g. only contract function calling)

filip · March 15, 2021, 1:18pm

Thanks mate. aestudio looks great, didn’t know about that one!