Our online services are experiencing issues.
Being on Azure, we have received the following
Microsoft Azure portal - Multi-Region - Advisory [Multi-Region]
SUMMARY OF IMPACT: From 21:28 UTC on 13 Apr, 2016 to 17:39 UTC on 14 Apr, 2016, Azure customers attempting to authenticate to the Azure Management Portal (portal.azure.com) and Classic Portal (manage.windowsazure.com) using a Microsoft Service Account (MSA) may have experienced login failures. Additionally, some Visual Studio Team Services customers leveraging MSA also experienced failures when accessing their services. Additional information specific to VSTS impact has been provided on the VSTS team blog at https://blogs.msdn.microsoft.com/vsoservice. PRELIMINARY ROOT CAUSE: As part of our release processes we “flight” changes to a small subset of our infrastructure. A recent MSA configuration update was flighted to approximately 1% of the MSA service and introduced an unexpected failure during the authentication process for customers with SSO enabled. Retry attempts failed due to a race condition that has been identified and being further investigated. MITIGATION: Engineers engaged and diagnosed the issue as being related to a change in MSA configuration. Engineers performed a rollback of this change, resetting the configuration values back to the known good state they were in previously. This mitigated the issue immediately for impacted customers. NEXT STEPS: The MSA team will continue to investigate the root cause of the issue and the race condition that extended the duration of impact. Additionally, we will review our change / test processes specifically for this issue to avoid a repeat of the issue.