Pantheon Operations Status

Mar 23, 2023

No incidents reported today.

Mar 22, 2023

Resolved - This incident has been resolved. The implemented fix has restored functionality in the impacted workflows. This has been confirmed through multiple successful workflows.
Mar 22, 23:34 PDT

Identified - The issue has been identified and a fix is being implemented.
Mar 22, 23:22 PDT

Investigating - Our monitoring has detected elevated error rates for the Pantheon Workflows. We are currently investigating the issue.
Mar 22, 23:14 PDT

Mar 21, 2023

Degraded AGCDN Performance

Resolved - The recent incident has been resolved and the errors you may have experienced should no longer appear.

A change in rules for a limited set of WAF users resulted in mistaken 403 errors, especially during CMS login and content editing. If you still encounter unexpected 403 errors, please try clearing the cache and testing in an incognito window to confirm it is not a local caching issue.

However, if you're still having any trouble accessing your site, our support team is available to help. Please don't hesitate to contact us for further assistance.
Mar 21, 10:34 PDT

Update - We are continuing to monitor for any further issues.
Mar 21, 09:45 PDT

Update - We are continuing to monitor for any further issues.
Mar 21, 09:09 PDT

Update - We are continuing to monitor for any further issues.
Mar 21, 08:38 PDT

Update - We are continuing to monitor for any further issues.
Mar 21, 08:08 PDT

Update - We are continuing to monitor for any further issues.
Mar 21, 07:18 PDT

Update - We are continuing to monitor for any further issues.
Mar 21, 06:37 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Mar 21, 06:05 PDT

Investigating - We are currently investigating an issue affecting a small number of sites using AGCDN and WAF. This is causing issues logging into the CMS admin for those sites only, sites not using this configuration are not affected.
Mar 21, 05:54 PDT

Mar 20, 2023

Degraded Dashboard and Terminus Performance

Resolved - This incident has been resolved. Thank you so much for your patience and understanding!
Mar 20, 12:05 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Mar 20, 11:35 PDT

Investigating - Our monitoring has detected elevated error rates for the Dashboard and Terminus commands. These may manifest as slow page loads, failed logins, or failures with Terminus commands.

For urgent issues please contact support via helpdesk@pantheon.io.
Mar 20, 11:02 PDT

Degraded Workflows Performance

Resolved - This incident has been resolved. The implemented fix has restored functionality in the impacted workflows. This has been confirmed through multiple successful workflows.
Mar 20, 07:41 PDT

Monitoring - Our monitoring has detected elevated error rates for the Pantheon dashboard workflows. These may manifest as failed deployments, failed multidev creation, or failed sync code workflows.
Mar 20, 07:00 PDT

Degraded Sites and Dashboard Performance

Resolved - This incident has been resolved.
Mar 20, 05:40 PDT

Update - We are continuing to monitor for any further issues.
Mar 20, 05:20 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Mar 20, 04:50 PDT

Update - We are continuing to investigate this issue.
Mar 20, 04:41 PDT

Update - We are continuing to investigate this issue.
Mar 20, 04:29 PDT

Investigating - Our monitoring has detected elevated error rates for the Pantheon Sites and Dashboard. These may manifest as slow page loads, failed logins, or failing to load the Site and Dashboard.
Mar 20, 04:28 PDT

Mar 19, 2023

No incidents reported.

Mar 18, 2023

No incidents reported.

Mar 17, 2023

Degraded Workflows

Resolved - This incident has been resolved. As evidenced by the successful results, the implemented fix has proven to be effective in resolving the issue.
Mar 17, 15:57 PDT

Monitoring - Our engineering team has deployed a fix for the issue, and we are seeing successful results. We are currently monitoring the platform to ensure that everything is running smoothly.

If you're experiencing any ongoing issues, please don't hesitate to reach out to our support team for assistance.
Mar 17, 14:56 PDT

Update - After further investigation, we have determined that a specific subset of Drupal sites that had a code sync or deployment between 13:00-16:00 UTC on 2023-03-17 have been impacted by a bug. This bug resulted in an issue with some environments accessing files within their code directory. This issue caused cascading failures on our platform when deploying new code to environments.

We would like to assure you that this problem has not affected the entire platform, and the files are not lost but are currently inaccessible to the affected sites. If you suspect that your site has been affected, we encourage you to contact us via chat or by submitting a ticket, so that we can investigate the issue further.

Thank you for your understanding and patience as we work diligently to resolve this issue.
Mar 17, 13:53 PDT

Investigating - We are currently conducting a more thorough investigation of the issue since the initial solution seems to have been ineffective.
Mar 17, 10:52 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Mar 17, 09:02 PDT

Identified - The issue has been identified and a fix is being implemented.
Mar 17, 08:18 PDT

Investigating - Our monitoring has detected elevated errors for some workflows, which may result in failed code deployment, backup restore, and creation of Multidev environments.
Mar 17, 07:56 PDT

Mar 16, 2023

Degraded Dashboard and Terminus Performance

Resolved - This incident has been resolved.
Mar 16, 09:39 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Mar 16, 09:08 PDT

Identified - The issue has been identified and a fix is being implemented.
Mar 16, 08:57 PDT

Update - We are continuing to investigate the issue and we appreciated your patience!
Mar 16, 08:55 PDT

Investigating - Our monitoring has detected elevated error rates for the Dashboard and Terminus commands. These may manifest as slow page loads, failed logins, or failures with Terminus commands.

For urgent issues please contact support via helpdesk@pantheon.io.
Mar 16, 08:20 PDT

Mar 15, 2023

Support Ticket Issues

Resolved - This incident has been resolved.
Mar 15, 17:14 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Mar 15, 17:10 PDT

Identified - The issue has been identified and a fix is being implemented.
Mar 15, 17:01 PDT

Investigating - We are currently investigating issues with our ticketing system.

If you have urgent issues, please open a support chat.
Mar 15, 16:26 PDT

Mar 14, 2023

Degraded Dashboard and Terminus Performance

Resolved - This incident has been resolved.
Mar 14, 08:53 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Mar 14, 08:22 PDT

Investigating - Our monitoring has detected elevated error rates for the Dashboard and Terminus commands. These may manifest as slow page loads, failed logins, or failures with Workflows and Terminus commands.

For urgent issues please contact support via helpdesk@pantheon.io.
Mar 14, 07:46 PDT

Mar 13, 2023

Degraded Dashboard Performance

Resolved - This incident has been resolved.
Mar 13, 23:55 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Mar 13, 22:49 PDT

Investigating - We're experiencing issues related to dashboard login, we are working on the issue as quickly as possible.

If you need immediate assistance, please email helpdesk@pantheon.io. Thank you for your patience.
Mar 13, 22:42 PDT

Dashboard Login

Resolved - This incident has been resolved.
Mar 13, 10:06 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Mar 13, 09:51 PDT

Identified - We're experiencing issues related to dashboard login which are associated with one of our authentication providers. We are working with the provider to resolve the issue as quickly as possible.

If you need immediate assistance, please email helpdesk@pantheon.io. Thank you for your patience.
Mar 13, 09:28 PDT

Investigating - We're experiencing issues related to dashboard login which are associated with one of our authentication providers. We are working with the provider to resolve the issue as quickly as possible.

If you need immediate assistance, please email helpdesk@pantheon.io. Thank you for your patience.
Mar 13, 09:24 PDT

Mar 12, 2023

Redis endpoint down

Resolved - This incident has been resolved.
Mar 12, 21:24 PDT

Identified - The issue has been identified and a fix is being implemented.
Mar 12, 20:26 PDT

Update - We are continuing to investigate this issue.
Mar 12, 19:39 PDT

Investigating - We are currently conducting ongoing investigations into the matter with the aim of identifying a sustainable solution.
Mar 12, 18:42 PDT

Update - We are actively working on a solution to address this issue.
Mar 12, 17:40 PDT

Update - We are continuing to work on a fix for this issue.
Mar 12, 16:42 PDT

Identified - The issue has been identified and a fix is being implemented.
Mar 12, 15:52 PDT

Investigating - We are currently investigating a Redis Out Of Memory issue being experienced on a subset of sites.
Mar 12, 15:17 PDT

Mar 11, 2023

No incidents reported.

Mar 10, 2023

No incidents reported.

Mar 9, 2023

No incidents reported.