Degraded Workflows
Incident Report for Pantheon Operations

On November 14, 2023, starting at 20:19 UTC, a surge of workflow failures alerted Pantheon engineering to an issue pulling container images from an upstream vendor registry. Confirmation of the upstream issue was quickly confirmed via the vendor status page site detailing the external service outage impacting container image push and pull operations. Upon upstream remediations, the workflow failure rate returned below threshold on November 14, 2023 at 23:25 UTC.

We have identified areas to improve our infrastructure to prevent issues like this from occurring in the future.

Posted Nov 28, 2023 - 11:15 PST

This incident has been resolved.
Posted Nov 15, 2023 - 06:48 PST
We're pleased to report that all systems are currently functioning correctly. Our team is actively monitoring performance to ensure optimal service. If you have any questions or remaining issues with workflows, please open a support chat with us, or email

We'll provide another status update within the next 6 hours or sooner if the situation is resolved. Thank you for your patience and understanding.
Posted Nov 15, 2023 - 03:36 PST
All systems are functioning correctly. However, we are currently in the process of closely monitoring to ensure optimal performance. Please feel free to reach out if you have any further questions. We will provide another update within 6 hours or sooner if the problem is resolved.
Posted Nov 14, 2023 - 21:11 PST
We are still in the monitoring phase to confirm the solution's effectiveness. The fix to address the recent workflow issues on Pantheon has been implemented for more than 2 hours now, and we haven't received any additional reports. Further updates will be provided as the situation progresses. Your patience and understanding are appreciated. If you have any questions or encounter workflow issues, please don't hesitate to contact support.
Posted Nov 14, 2023 - 18:36 PST
Our upstream provider has implemented a fix that resolves the recent workflow issues on Pantheon. We are now in the monitoring phase to ensure that the solution is effective. We will provide further updates as the situation progresses. Thank you for your patience and understanding, and please reach out to support if you have any questions or remaining issues with workflows.
Posted Nov 14, 2023 - 16:09 PST
The issue continues to affect various workflows on Pantheon. Our team is diligently working on addressing the current challenge, exploring all possible solutions while coordinating with our service provider. We will share the next update in 3 hours, or sooner if more information becomes available. Your patience and understanding during this time are greatly appreciated.
Posted Nov 14, 2023 - 15:12 PST
The issue is still impacting workflows on Pantheon. Our team is actively exploring all available options to alleviate the current issue. We will provide an update in 1 hour or as more information becomes available. Thank you for your ongoing patience.
Posted Nov 14, 2023 - 14:03 PST
We are currently experiencing an infrastructure issue that is affecting several workflows on Pantheon, including code synchronization, cache clearing, enabling Solr, and creating new sites or environments.

The current investigation points to an issue with one of our Docker image service providers. We are working with the provider to resolve the issue as quickly as possible.

We apologize for any inconvenience caused and appreciate your patience as we work towards a fix.
Posted Nov 14, 2023 - 13:14 PST
This incident affected: Workflow Operations.