In the past two days, we have been experiencing technical issues that required unexpected maintenance and caused system downtime. Our system architecture is something we are very proud of, moreover, we continuously strive to offer the best trading experience and minimize downtimes altogether. However, there are situations that are out of our control. Our clients have always put a lot of trust in us, therefore, we believe in full transparency and would like to explain what caused these issues.
Yesterday we experienced a bug in Erlang itself, which could be described as a DOS vulnerability. This was caused by a bot, that (unintentionally) kept sending a series of requests to our platform in such a way, that triggered this vulnerability and brought all our web nodes down. Importantly, the master (matching engine) node was not affected.
Before we managed to locate and hotfix the bug, we experienced this situation twice. Thus, causing two downtimes in a very short period of time. We did, however, manage to restart our web-nodes in less than 10 minutes each time.
Although the bug is within Erlang itself, we have implemented a workaround in our code and have reported the bug to the Erlang team, which will fix the issue.
You can see this bug report here.
In a separate incident today, our most important web node (which is also responsible for load balancing) experienced a Linux kernel panic when handling an interruption from one of its gigabit network cards.
Please note that all other web-nodes (Hermes and Mercury) and master (Matching Engine) node were not affected. This issue was not related to the Erlang vulnerability and was a technical issue. Due to this, we had to reboot the main web-node, again causing downtime.
We are planning to implement a different type of load balancing system, which will have no single point of failure, thus mitigating the risk of this happening ever again.
We take each of these incidents very seriously, and our development team does everything it can to prevent these issues from happening again. We are grateful for your understanding and hope you enjoy trading at Deribit.
John Jansen
Founder and CEO of Deribit
AUTHOR(S)
We are the best resource for Crypto Derivatives trading. Deribit is not available in the United States or other restricted countries.