9/22/19: A Note to the Exodus Community
September 22, 2019
A Note to the Exodus Community
Between September 14, 17:00 UTC and September 15, 06:00 UTC, if you opened Exodus, you would have experienced an issue causing the local currency value of assets in your Exodus wallet to show as 0. If you didn't open Exodus during this time, then you would not have noticed a difference. We’re thankful you let us know about this issue right away, and we’ve fixed it.
Although no funds were at risk, we deeply regret the frustration, anxiety and concern we caused as a result of this incident. We’re disappointed in ourselves that it happened, and also that we didn’t let you know quickly what was happening.
Here are the details:
- Our server which provides asset values in your local currency, as well as the market value displayed in your portfolio, became overloaded. As a result, Exodus displayed local currency asset values as zero when you opened your wallet. This is perhaps our most glaring shortcoming: We didn’t account for this possibility and allowed the wallet to default to zero value in your local currency, even though the actual asset balances were accurate.
- We didn’t react as quickly as we could have and failed to communicate well with you. We have alerts in place for our infrastructure which warn us when something goes wrong. In this particular instance, an alert was triggered but was missed by the team. Meaning; we discovered there was a problem the same way you did
- Once the team mobilized, our DevOps engineers worked feverishly for nine hours to pinpoint the issue; this delayed our response and resolution. After investigating multiple possibilities, the team determined the server, which provides pricing information displayed in the wallet, overloaded and failed. The root cause of the server's overload was increased traffic that we didn't anticipate.
What we’re changing to improve Exodus
We’ve learned from this incident, and we’re changing a few things to improve your experience.
- Firstly, we've upgraded our pricing information service:
- Since this event, we have already deployed half a dozen improvements to this service to improve logging, availability, reliability, and decrease load.
- We now have a caching layer on the pricing service to further decrease the load and increase overall reliability.
- We will continue to source multiple data feeds for your asset pricing to ensure accurate and timely data for your wallet.
- We've created stress tests to test how our servers handle increased loads and traffic.
- We will also be expanding our beta-testing program to help keep us further ahead of the curve.
- We're also looking to improve our strategy for incident response:
- We’re establishing procedures and investing in resources to allow our Support Team to quickly identify incidents when they occur. Things will go wrong and we can’t account for all the possibilities, but here's what we can ensure: Our team will be well-equipped to support you during an incident.
- Timely and transparent status updates will be available on our website and in your wallet, both during and after incidents:
- Status reports will be more visible on our webpage.
- We will send in-app updates during an incident, so you have accurate information available wherever you are.
- We’ll provide updates throughout the course of an incident as well as post-incident communication to explain what happened.
- Finally, we will be raising the alarm faster to minimize disruptions to your service:
- Although Exodus is already a 24/7 operation, we’re now going one step further with the implementation of rotating "on-call" responsibilities for our technical teams.
- Looking back, this was one of the most important lessons for us; When something goes wrong, we’ll be notified quickly and have already identified specific team members who are empowered to take the next steps.
We'd love to hear from you!
Our relationship is built on trust and transparency, and we let you down last weekend.
There’s nothing more important to us than your trust. We’re working hard to earn it back by ensuring the implementation of the fixes listed above and being transparent with you throughout the process. We are committed to all of these steps, although some will take longer than others.
We’ll be happy to join a phone or video call with you to discuss any feedback or concerns you might have. Please send us a note at [email protected] and our Support team will arrange a meeting.
The Exodus Team