At approximately 12:40 UTC 25 May the performance of our dashboard, workflows, and DNS interactions was disrupted by an administrative workload that caused an unexpectedly high volume of reads to a database those systems all rely on. The exact source of the workload was identified and halted at 20:04 UTC 25 May. We are implementing processes to better coordinate our efforts to avoid similar disruption and scheduling work to make our systems more resilient.
Posted Jun 03, 2022 - 09:07 PDT
Resolved
This incident has been resolved.
Posted May 25, 2022 - 14:54 PDT
Update
We are continuing to monitor for any further issues.
Posted May 25, 2022 - 14:13 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 25, 2022 - 13:26 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted May 25, 2022 - 13:15 PDT
Update
We are continuing to investigate this issue.
Posted May 25, 2022 - 13:02 PDT
Update
We are continuing to investigate this issue.
Posted May 25, 2022 - 12:30 PDT
Update
We are continuing to investigate this issue.
Posted May 25, 2022 - 12:00 PDT
Update
We are continuing to investigate this issue.
Posted May 25, 2022 - 11:34 PDT
Update
We are continuing to investigate this issue.
Posted May 25, 2022 - 11:00 PDT
Update
We are continuing to investigate this issue.
Posted May 25, 2022 - 10:31 PDT
Update
Our monitoring has detected elevated error rates for the Dashboard and Terminus commands. These may manifest as slow page loads, failed logins, failed SFTP connection, failed Git connections, or failures with Terminus commands.
We are continuing to investigate this issue.
Posted May 25, 2022 - 10:03 PDT
Update
We are continuing to investigate this issue.
Posted May 25, 2022 - 09:34 PDT
Update
We are continuing to investigate this issue.
Posted May 25, 2022 - 09:06 PDT
Update
We are continuing to investigate this issue.
Posted May 25, 2022 - 08:34 PDT
Update
We are continuing to investigate this issue.
Posted May 25, 2022 - 08:02 PDT
Update
We are continuing to investigate this issue.
Posted May 25, 2022 - 07:31 PDT
Investigating
Our monitoring has detected elevated error rates for the Dashboard and Terminus commands. These may manifest as slow page loads, failed logins, SFTP connection, Git connections, or failures with Terminus commands.