Database endpoint down
Incident Report for Pantheon Operations

At 15:36 UTC on 19th May 2022, a configuration change was made during a database server upgrade. The configuration changes caused one database server endpoint to become overloaded and unresponsive. In response to this we fixed the configuration and restarted the database server endpoint. During the restart process several websites were down until their configuration was reloaded. By 17:10 UTC on the 19th May 2022 all sites were back up. 

We have identified several improvements to our configuration and update process to help us detect and prevent these issues in the future.

Posted May 26, 2022 - 15:13 PDT

This incident has been resolved.
Posted May 19, 2022 - 10:41 PDT
A fix has been implemented and we are monitoring the results.
Posted May 19, 2022 - 10:11 PDT
We are investigating a failed database endpoint.
Posted May 19, 2022 - 09:19 PDT
This incident affected: Customer Sites.