does anyone want to jump the queue?
In epoch 7 the CLIO1 pool was a leader to produce a block for slot 3161. Our monitoring normally shows a constantly rising red line for each block created by the testnet pools:
The brown horizontal line is the approached next scheduled slot. If everything goes correctly, the block is created at the exact time and the line jumps to the next scheduled slot in the monitoring graphic.
But at the mentioned slot 3161 in epoch 7 we observed something very unusual: The actually constantly growing block chain suddenly jumped to slot 3245 and this before the time for slot number 3161 has come.
So: How was this possible and what effects did it have?
As can be seen in the graph, there was no progress for a long period of time then due to the extraordinary event. The stake pools probably tried to find out which of the two blocks was the right one.
Since CLIO1 could only refer to the block 3123 previously produced with the lower slot number, there were suddenly two blocks and thus a fork. If everything goes right and honest there shouldn’t be any fork!
If the blocks had been produced in order according to their height, there would be no problem and the hash checksums of all blocks in a row would fit together.
But so the subsequent block creators had to make a selection, append their block to one and discard the other including the user transactions contained in it. In this case, the pool who had pushed forward had won the race and CLIO1 had unfortunately lost its block, even if created correctly on time.
For us pool operators it is an exciting time. Especially when the network still has strong fluctuations, you try to find out what the error is and what you can do on your own site to ensure stability and performance.
Misconception or deliberate strategy?
The obviously too early generated and distributed block came from BNTY1 Pool. The question now is whether this is due to an imprecise server clock or whether the pool operator deliberately tried to generate and send his block a bit earlier in order to beat a competitor for the same slot and then shine with better performance values at the end of the epoch. But all this at the expense of stability and with the side effect of rolling back user transactions that were discarded with the CLIO1 block.
After this incident, we observed the matter, recorded a lot of data, and consulted with other pool operators known to us. They checked their own recordings and confirmed the “extraordinary” event was also seen on their nodes. We also contacted the Jormungandr development team. They checked if there is a possible process error that can be fixed.
If, on the other hand, it were a deliberate strategy of a pool operator, a completely different situation and challenge would arise for the stable and proper operation of the Cardano blockchain.
We assume that in the past epochs some pool operators have fiddled around with time and slot-schedules, with the aim of gaining an unfair advantage. However, since this has led to twists in the block order and thus to forks, we also consider it likely that these tricks have made a significant contribution to the instabilities and problems in epoch 4-5-6.
The solution is v0.8.5
We are really pleased to report that release v0.8.5 includes a mechanism that allows honest pool operators to sideline these prematurely created and distributed blocks. It is a fork that only becomes significant when its time has come.
This means that this trick turns into a normal 51% attack Vector. Only if a majority of pool operators decide to move their pools’ system clocks into the future they will win. If there is no majority, they will create a block that may contain invalid transactions because they created it too early.
If a majority of pool operators would decide to always put their system clocks a little bit before the others, it could happen that we humans at the end of the test net actually have April, but the blockchain is already in May. And that would probably be pretty crazy, we think.
So please let us remain honest and serious!
The challenges are there, and with the research and development work made available to us all free of charge, there are many real opportunities.