We have some more information for you.
TLDR; this should be solved now 
Longer version: we did a lot of digging. Yes, the issue we solved last week was indeed a bug but it was not causing the problem at hand. We dug deeper and deeper and found some optimisation points but no bug that could cause transactions to simply disappear. We were checking syncing, gossiping and even block candidate generation. Everything seemed solid.
Thanks to your input here, we were actually able to trace a transaction (@filip’s) that was accepted in the blockchain and then rolled back. Then accepted again and once more rolled back. @hanssv.chain wrote a small script and we got an a-ha moment:
TX found in micro block mh_GJtShFamkJKL74B8CkG7oDoMEJMAhKq1DRMkjNVcUTq8GgvPw at 401550
TX found in micro block mh_28DLeSxooxSV3ciwsZNY8uXtPCkHqBrwfabHuFrjyt4Qrmdn63 at 401554
TX found in micro block mh_2XwKWVGg4dtqmFY18DQX2q7WDxvuAtPkgDRtQudcytDVzDiYoN at 401556
TX found in micro block mh_2NZvRnFwpcUZ3ZRrv5Pk9bko8mHXQjEZxMyHiRH4yjaJxAAcrr at 401558
TX found in micro block mh_2FGqfQMRjYC5uy6iFMmuZE7JHzFvbjQ3WEJUgvF1NMExsyHUJ at 401559
TX found in micro block mh_EJrYGFSg88ht6ZjPDpRyu3gYQAmamg58q3aWPKww47Ma7rQGm at 401564
TX found in micro block mh_2ivuthR78Tc2VUDDy2NY963a8Km5zPf2EVqCxnt3W3kE1bYP3R at 401569
TX found in micro block mh_RvLDXCeKvP4JLVP54pjwzSJFwVVmDiuY3TULZrzn4ZgVQqNWH at 401571
TX found in micro block mh_2orCBh99J4w4QC2uyKYofxxgefvuh9HCA4AZX5iMR8MB5Eo98U at 401572
TX found in micro block mh_hCm9gvQBXMhVH22FNriQaq9k72CuZuTZEaombmNRNEBByTRqm at 401573
TX found in micro block mh_ZfUHGhexYoTTN9wEgQSoB9Qxef4UUgP4eQT32DwsRt5zX89ai at 401574
TX found in micro block mh_5xHaR8q2YGEZaaaMzwss5AFNymJR29sXerNdRftiD1yVfKUMZ at 401575
TX found in micro block mh_2ecaiJiUXcPrvXK8DbaSbtazV488fZfinD6YJUhHLfBTm4mTiY at 401579
TX found in micro block mh_2VWqNdeHcojpcZt6QMnTTCTJMhtToD57uuySCLmzwVzpxifBcD at 401587
TX found in micro block mh_qebWGA84TgpcNsKBGgGzAH9wXkhi5gpk8PheBzwFDCdHMamCU at 401590
TX found in micro block mh_Ma3vJ5BrQY1mKXmWQSNoeFLDrbH11Abjz2HjwxAbo2WCM6fNz at 401592
TX found in micro block mh_3iYrGtqmtXY37n9EnAA6oSDUAzyqpkWxC52zz79XezwzWh1Ny at 401593
TX found in micro block mh_3kwBJexZFLzWCmMFtFe6F1kZjhW8F1xr7BAECdabuAubsxZyG at 401594
TX found in micro block mh_mfuvMHTumMD1zMemLFsTsugsa1i3egPacmUMXuJJfhC2JTrj3 at 401596
TX found in micro block mh_2hmwwX4hoTP7wzaXUom9rApsrEL9S19jT3tC2ssK8rPnvPpEZs at 401597
The transaction was reaching the miner (so no issue with sync or gossip), it was included in a block but then it was rolled back. After that it was included again (so block candidate was solid) only be rolled back again.
In the context of BitcoinNG microforks are expected. Keyblocks are being produced roughly every 3 minutes but micro blocks are PoW-free and can be issued every 3 seconds. This results in a really elastic and dynamic system, where you can expect a first confirmation in a matter of seconds. The downside is those micro blocks still have to propagate in the network, restart the miner and this usually means that the last few micro blocks in a generation are discarded in a microfork. There is nothing we could to there, the limit is pretty much the speed of light. A general rule of the thumb is if your transaction was included in the end of the 3 minutes interval, it is likely to get rolled back but included win the next generation.
After some further digging, we found out that not only a handful of micro blocks were rolled back but whole generations full of micro blocks. The cause was a configuration on the miner’s end, nothing to fix in the node, really. The miners applied the new setting around generation height 404840 and you can see the result below:
We do still have microforks but they are completely normal and expected. What is more - they are all short in the end of the generation.
Bottom line is the blockchain is a distributed system with a lot of moving parts and different actors. It is really hard to debug but that is a small price to pay for its benefits.
