At 19:40 UTC on June 22nd, we applied a networking configuration change to one of our internal services. This change unexpectedly caused an error within another internal service which caused some customer sites to become unresponsive temporarily. We quickly noticed the impact of the configuration change and reverted the configuration, allowing functionality to be automatically restored. By 20:15 UTC all affected sites were operational once again. We identified several improvements that should prevent such an incident in the future.
Posted Jun 24, 2022 - 21:14 PDT
Resolved
This incident has been resolved.
Posted Jun 22, 2022 - 13:49 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 22, 2022 - 13:20 PDT
Update
We are continuing to investigate this issue.
Posted Jun 22, 2022 - 13:18 PDT
Investigating
We are addressing an infrastructure failure that is affecting a small portion of customer sites.