Degraded Dashboard and Terminus Performance

Incident Report for Pantheon Operations

Postmortem

On September 18, 2023 at 18:38 UTC, Pantheon detected an unusually high level of activity on our system that led to temporary issues with Dashboard and Terminus performance. We identified this to be a DoS attack and swiftly blocked it by the Web Application Firewall (WAF) but resulted in performance problems. Users faced difficulties logging in, encountered errors, and noticed sluggish Dashboard performance during this time. These issues were resolved by 16:24 PDT.

‌To enhance security, Pantheon is improving its WAF configuration, optimizing platform performance, and strengthening monitoring capabilities. We apologize for the interruption and are committed to preventing future incidents.

Posted Oct 13, 2023 - 11:52 PDT

Resolved

We are pleased to announce that the service interruption some of you may have experienced with our Dashboard and Terminus functionality has been resolved. Our engineering team has taken corrective actions to ensure that the platform is now stable and fully operational.

We appreciate your patience and understanding during this time. Should you experience any further issues, please don't hesitate to reach out to our support team.

Posted Sep 18, 2023 - 16:24 PDT

Identified

The issue has been identified and a fix has been implemented.

Posted Sep 18, 2023 - 15:53 PDT

Update

We are continuing to investigate the issue.

Posted Sep 18, 2023 - 14:25 PDT

Update

We are continuing to investigate the issue.

Posted Sep 18, 2023 - 13:19 PDT

Update

We are continuing to investigate the issue.

Posted Sep 18, 2023 - 12:39 PDT

Investigating

We have detected elevated error rates for the Dashboard and Terminus commands. These may manifest as slow page loads, failed logins, or failures with Terminus commands. Our engineering team is actively investigating the root cause of these issues.

For urgent issues, please contact support via helpdesk@pantheon.io or by opening a support chat.

Posted Sep 18, 2023 - 12:09 PDT

This incident affected: Dashboard and Terminus Operations.