Infrastructure Issue Affecting Customer Sites
Incident Report for Pantheon Operations
Postmortem

At 19:40 UTC on June 22nd, we applied a networking configuration change to one of our internal services. This change unexpectedly caused an error within another internal service which caused some customer sites to become unresponsive temporarily. We quickly noticed the impact of the configuration change and reverted the configuration, allowing functionality to be automatically restored. By 20:15 UTC all affected sites were operational once again. We identified several improvements that should prevent such an incident in the future.

Posted Jun 24, 2022 - 21:14 PDT

Resolved
This incident has been resolved.
Posted Jun 22, 2022 - 13:49 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 22, 2022 - 13:20 PDT
Update
We are continuing to investigate this issue.
Posted Jun 22, 2022 - 13:18 PDT
Investigating
We are addressing an infrastructure failure that is affecting a small portion of customer sites.
Posted Jun 22, 2022 - 13:17 PDT
This incident affected: Customer Sites.