Infrastructure Issue Affecting Customer Sites

Incident Report for Pantheon Operations

Postmortem

On November 12, 2023, at 13:33 UTC, we encountered an infrastructure issue affecting routing, which led to temporary service disruptions on multiple Pantheon-hosted sites. Our team quickly identified and addressed the root cause, restoring all services by 14:42 UTC. A brief recurrence of the issue occurred on November 13 at 02:06 UTC, which was rapidly resolved by our engineering team.

We apologize for the impact caused to our customers and assure you that we are actively taking steps to enhance our systems to prevent similar incidents in the future. These improvements include strategic upgrades to our traffic management protocols, set to be implemented in November 2023, and refinement of our response strategies to ensure swift resolution of potential issues. At Pantheon, we remain committed to providing reliable and secure hosting services.

Posted Nov 16, 2023 - 10:46 PST

Resolved

The issue has been successfully resolved, and we anticipate no further disruptions. If you encounter any unexpected issues, please feel free to reach out to our support team for immediate assistance.

We sincerely appreciate your patience and continued support during this incident.

Posted Nov 12, 2023 - 08:35 PST

Monitoring

A fix has been implemented and we are actively monitoring the situation as we observe a complete recovery in progress.

Posted Nov 12, 2023 - 07:59 PST

Identified

The issue has been identified and a fix is being implemented.

Posted Nov 12, 2023 - 07:29 PST

Investigating

We are currently investigating an infrastructure failure impacting customer sites. The identified symptoms include encountering 503 errors, resulting in site unavailability.

For urgent issues, please contact helpdesk@pantheon.io

Posted Nov 12, 2023 - 06:54 PST

This incident affected: Customer Sites.