Infrastructure Issue Affecting Customer Sites
Incident Report for Pantheon Operations
Postmortem

At 00:00 UTC on 3 August 2022, a certificate at our CDN vendor continued to be served after its expiration, causing AGCDN sites to become inaccessible. After diagnosing the issue, we engaged with our vendor to update the affected services to use the correct certificate. This change was made at 02:25 UTC, after which the affected sites became available again. This downtime did not have any effect on site data or backups.

We are working with our vendor to improve monitoring that will detect this issue in the future, and we have updated our internal process to include additional proactive checks to our certificate update process that will prevent an issue like this from occurring again.

Posted Aug 09, 2022 - 07:49 PDT

Resolved
This incident has been resolved.
Posted Aug 02, 2022 - 21:12 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Aug 02, 2022 - 19:47 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted Aug 02, 2022 - 19:36 PDT
Update
We are continuing to investigate this issue.
Posted Aug 02, 2022 - 18:59 PDT
Update
We are continuing to investigate this issue.
Posted Aug 02, 2022 - 18:21 PDT
Investigating
We are currently investigating this issue.
Posted Aug 02, 2022 - 17:51 PDT
This incident affected: Customer Sites and Global CDN.