Pantheon Operations Status

Sep 21, 2023

No incidents reported today.

Sep 20, 2023

Resolved - Drupal core security releases 10.1.4, 10.0.11 and 9.5.11 address a cache poisoning vulnerability. See https://drupal.org/sa-core-2023-006 for details. Pantheon engineering is investigating whether our CDN and WAF layers do or can provide additional mitigation for this particular vulnerability. Our investigations so far have discovered that this Drupal vulnerability can be reproduced for some sites running on Pantheon that have not applied the security update, including sites running Drupal 8.7.0 and later. Please upgrade your Drupal sites to secure versions as soon as possible.
Sep 20, 12:44 PDT

Sep 19, 2023

Degraded Dashboard and Terminus Performance

Resolved - This incident has been resolved.
Sep 19, 19:50 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Sep 19, 18:44 PDT

Identified - The issue has been identified and a fix is being implemented.
Sep 19, 18:00 PDT

Investigating - We have detected elevated error rates for the Dashboard and Terminus commands. These may manifest as slow page loads, failed logins, or failures with Terminus commands.

For urgent issues, please contact support via helpdesk@pantheon.io
Sep 19, 17:21 PDT

Sep 18, 2023

Degraded Dashboard and Terminus Performance

Resolved - We are pleased to announce that the service interruption some of you may have experienced with our Dashboard and Terminus functionality has been resolved. Our engineering team has taken corrective actions to ensure that the platform is now stable and fully operational.

We appreciate your patience and understanding during this time. Should you experience any further issues, please don't hesitate to reach out to our support team.
Sep 18, 16:24 PDT

Identified - The issue has been identified and a fix has been implemented.
Sep 18, 15:53 PDT

Update - We are continuing to investigate the issue.
Sep 18, 14:25 PDT

Update - We are continuing to investigate the issue.
Sep 18, 13:19 PDT

Update - We are continuing to investigate the issue.
Sep 18, 12:39 PDT

Investigating - We have detected elevated error rates for the Dashboard and Terminus commands. These may manifest as slow page loads, failed logins, or failures with Terminus commands. Our engineering team is actively investigating the root cause of these issues.

For urgent issues, please contact support via helpdesk@pantheon.io or by opening a support chat.
Sep 18, 12:09 PDT

Sep 17, 2023

No incidents reported.

Sep 16, 2023

No incidents reported.

Sep 15, 2023

Degraded Dashboard and Terminus Performance

Resolved - This incident has been resolved.
Sep 15, 17:49 PDT

Update - We are seeing improvements in the situation and can confirm with our customers. We'll continue to monitor the issue and provide the next update within an hour.

For urgent issues, please contact support via helpdesk@pantheon.io or by opening a support chat.
Sep 15, 17:36 PDT

Monitoring - As the issue shows improvement, our team is monitoring as the situation has remained stable. We are committed to closely monitoring it and will provide updates as needed. We genuinely appreciate your patience throughout this process.

For urgent issues, please get in touch with support via helpdesk@pantheon.io or by opening a support chat.
Sep 15, 16:14 PDT

Update - While the issue shows improvement, our team remains actively engaged in the investigation and internal tuning to reduce errors. We are committed to quickly resolving this matter and will continue to provide updates as we identify and address contributing factors. We really appreciate your patience throughout this process.

For urgent issues, please get in touch with support via helpdesk@pantheon.io or by opening a support chat.
Sep 15, 15:10 PDT

Update - Our team is actively investigating and is doing internal tuning to reduce some errors. We understand the impact it may have on your operations. We apologize for the any inconvenience caused and want to ensure that resolving this issue is our utmost priority and we are working diligently to rectify it as quickly as possible.

For urgent issues, please contact support via helpdesk@pantheon.io or by opening a support chat.
Sep 15, 14:15 PDT

Update - Our top priority remains the investigation of this issue, given its significant impact on Dashboard and Terminus availability. We are actively engaged in resolving this matter as swiftly as possible and will keep you informed with updates on this page as soon as we have more information. Your patience is greatly appreciated.

For urgent issues, you may reach support via helpdesk@pantheon.io
Sep 15, 13:41 PDT

Investigating - We have detected elevated error rates for the Dashboard and Terminus commands. These may manifest as slow page loads, failed logins, or failures with Terminus commands.

For urgent issues, please contact support via helpdesk@pantheon.io or by opening a support chat.
Sep 15, 13:05 PDT

Degraded Email Deliverability

Resolved - This incident has been resolved.
Sep 15, 09:26 PDT

Monitoring - We have identified a bottleneck in our outgoing mail system and performed some tuning to alleviate the email deliverability. We are now monitoring the results.
Sep 15, 08:53 PDT

Update - We are continuing to investigate this issue.
Sep 15, 08:23 PDT

Update - Our team is actively investigating the disruption in email sending and we understand the impact it may have on your operations. We apologize for any inconvenience caused and want to assure you that resolving this issue is our utmost priority and we are working diligently to rectify it as quickly as possible.

For urgent issues, please contact support via helpdesk@pantheon.io or by opening a support chat.
Sep 15, 07:42 PDT

Update - We are continuing to investigate this issue.
Sep 15, 07:08 PDT

Investigating - We are currently investigating reports of issues with sending mail from customer sites.

For urgent issues, please contact support via helpdesk@pantheon.io or by opening a support chat.
Sep 15, 06:38 PDT

Sep 14, 2023

Infrastructure Issue Affecting Customer Sites

Resolved - The issue has been successfully resolved, and there should be no further disruptions. If you encounter any unexpected issues, please don't hesitate to contact our support team for immediate assistance. We greatly appreciate your patience and support throughout this incident.
Sep 14, 17:30 PDT

Update - We are making a small adjustment to increase stability and will continue to monitor for another 30 minutes, all sites are still holding stable.
Sep 14, 16:37 PDT

Update - All sites are reporting steady uptime and dashboard performance is back to normal. We will continue to monitor for another 30 minutes.
Sep 14, 16:05 PDT

Update - Ongoing monitoring is showing a steady improvement in dashboard and site responses.
Sep 14, 15:25 PDT

Update - We are continuing to monitor and seeing steady progress in dashboard and site responses. Multidev creation is now working as well, though it may take a minute for the multidev to be accessible.
Sep 14, 14:57 PDT

Update - Our team is actively addressing these issues and taking preventive measures. Thank you for your patience and support.
Sep 14, 14:19 PDT

Update - The original issue has been successfully resolved. While sites continue to recover, you may experience intermittent disruptions in both dashboard access and site functionality as we distribute the load evenly. We are also seeing some reports of new multidev instances not creating successfully.

Our team is actively addressing these side effects and implementing measures to prevent future incidents.
Sep 14, 13:39 PDT

Update - The original issue has been resolved, and while sites are recovering, some requests to the dashboard and sites may intermittently fail as load continues to be distributed. We are working to reduce the impact of this side effect.
Sep 14, 13:01 PDT

Update - Site availability is continuing to improve. We are actively monitoring and continue to see a full recovery in progress.
Sep 14, 12:12 PDT

Monitoring - A fix has been implemented and we are monitoring the results. Sites are beginning to respond but the higher initial load is leading to a delay in full site availability.
Sep 14, 11:44 PDT

Update - A fix is being deployed and is showing positive results. We expect this to be completed platform-wide very shortly.
Sep 14, 11:25 PDT

Update - We have identified the issue and are still actively working on a remediation. We apologize for the continued impact and are treating this with the highest priority.
Sep 14, 11:13 PDT

Identified - A certificate issue at the edge routing layer has been identified and a fix is being worked on immediately.
Sep 14, 10:41 PDT

Update - We are continuing to investigate this issue at the highest priority and understand this issue is significantly impacting site availability. We are working to resolve this as soon as possible. Initial investigation points to a networking failure affecting global infrastructure.

We will update this page as soon as we have more information.

For urgent issues, please contact helpdesk@pantheon.io
Sep 14, 10:35 PDT

Investigating - We are investigating an infrastructure failure that is affecting customer sites. Current symptoms identified are 503 errors leading to site unavailability.

For urgent issues, please contact helpdesk@pantheon.io
Sep 14, 10:00 PDT

Sep 13, 2023

No incidents reported.

Sep 12, 2023

No incidents reported.

Sep 11, 2023

No incidents reported.

Sep 10, 2023

No incidents reported.

Sep 9, 2023

No incidents reported.

Sep 8, 2023

No incidents reported.

Sep 7, 2023

No incidents reported.