Scheduler issue
Incident Report for Cyan Mobility
Resolved
This incident has been resolved.
Posted Sep 04, 2018 - 19:51 PDT
Monitoring
Here is the latest status on this issue from Microsoft.

CUSTOMER IMPACT: Starting at 09:29 UTC on 04 Sep 2018, customers in South Central US may experience difficulties connecting to resources hosted in this region. Additionally, this issue impacts Azure Active Directory authentication, which may impact services in other regions.

PRELIMINARY ROOT CAUSE: A severe weather event, including lightning strikes, occurred near one of the South Central US datacenters. This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process.

ENGINEERING STATUS: Engineers have successfully restored power to the datacenter. Additionally, engineers have recovered a majority of the impacted network devices. While some services are starting to see signs of recovery, mitigation efforts are still ongoing.
Posted Sep 04, 2018 - 13:01 PDT
Update
Microsoft is still working to fully recover Azure services. We are manually running Run Integration messages to avoid application issues and reduce any delay users may be experencing.
Posted Sep 04, 2018 - 11:33 PDT
Identified
We are currently investigating an issue with scheduled jobs in Cyan which uses Microsoft Azure Scheduler. This issue will affect Run Integration messages, for POD and P&D applications, users may see delays in getting orders to devices. Other apps may see delays if they are using Run Integration messages as well. Azure support is aware of the issue and working to resolve.
Posted Sep 04, 2018 - 05:39 PDT
This incident affected: Cyan Live Workers.