Status : Resolved 2019-07-19 12:55 UTC
List of impacted services :
As part of our resolution efforts, some services appeared up at 11:47 UTC, but were still impacted by the root cause. At 12:55 UTC all of our services are up.
We are closely monitoring the state of our services and we'll provide more information about the incident after we've completed our investigation.
Since it has been a full week since the incident, I wanted to publish a report of the investigation as it stands right now. Sadly, we are still working with our hosting provider to ascertain if the issue was a fluke, or if we can pinpoint the root cause and work on mitigating the risk of the same anomaly occurring.
The investigation is complicated by the fact that the piece of the infrastructure we are focusing on is offered by a third party. The last response from our provider indicated that sadly they were not experts on that specific technology. Be assured that we are still pursuing a resolution to our ticket.
We use multiple such clusters throughout our service offering, and the one used by our login layer is the only one that has suffered such an issue. We are in the process of creating a standby cluster to help in case of future events.
We will update this report as information comes in. Do not hesitate to open a ticket using https://devolutions.atlassian.net/servicedesk/customer/portal/1 for more information.