Degraded Performance
Incident Report for Pantheon Operations
Postmortem

At 1:04 UTC on 17 May, a sudden large influx of SMTP traffic caused servers to be overloaded and fail. Our on-call engineers were alerted to reports of customers’ SMTP connections being denied and unable to send emails. Our engineers immediately started to work on identifying the cause of the blocked connections and resolving them. SMTP email service was restored at 2:23 UTC on 19 May. We have identified improvements that will help us detect and prevent similar issues in the future.

Posted May 29, 2023 - 22:33 PDT

Resolved
We are no longer experiencing issues with email delivery. This incident has been resolved.

Thank you for your patience as we worked to resolve this issue.

If you are still experiencing issues, please contact support via helpdesk@pantheon.io or by opening a support chat.
Posted May 19, 2023 - 13:48 PDT
Update
Our Engineering Team is continuing to monitor the issue as email delivery remains stable.

We will provide you with regular updates on the progress of our investigation. The next update will be posted within 4 hours or sooner, as we continue to gather more information.

If you have any urgent concerns or require immediate assistance, please do not hesitate to reach out to our dedicated support team via helpdesk@pantheon.io or by initiating a support chat.
Posted May 19, 2023 - 08:56 PDT
Monitoring
Our Engineering Team has deployed a fix for the issue, and we are seeing successful results. We are currently monitoring the platform to ensure that everything is running smoothly.

If you're experiencing any ongoing issues, please don't hesitate to reach out to our support team for assistance.
Posted May 19, 2023 - 03:06 PDT
Update
Our Engineering Team is continuing the implemented measures to further investigate and identify the root cause of the issue.

The next update will be shared within the next 4 hours or sooner, as we continue to gather more information.

If you have any urgent concerns or require immediate assistance, please do not hesitate to reach out to our dedicated support team via helpdesk@pantheon.io or by initiating a support chat.
Posted May 18, 2023 - 23:03 PDT
Update
Our Engineering Team has implemented additional measures to further investigate and pinpoint the cause of the issue.

We will provide you with regular updates on the progress of our investigation. The next update will be posted within 4 hours or sooner, as we continue to gather more information.

If you have any urgent concerns or require immediate assistance, please do not hesitate to reach out to our dedicated support team via helpdesk@pantheon.io or by initiating a support chat.
Posted May 18, 2023 - 18:30 PDT
Update
Our team is actively investigating the disruption in email sending and we understand the impact it may have on your operations. We apologize for any inconvenience caused and want to assure you that resolving this issue is our utmost priority and we are working diligently to rectify it as quickly as possible.

We will continue to provide timely updates on the progress of our investigation. The next update will be posted in 2 hours or sooner as we gather more information.

If you have any urgent concerns or require immediate assistance, please reach out to our dedicated support team via helpdesk@pantheon.io or by initiating a support chat.
Posted May 18, 2023 - 16:17 PDT
Update
Our Engineering Team continues to investigate the cause of the disruption in email sending. At this moment, we do not have any significant updates or specific information to report. Please rest assured that resolving this issue is our top priority, and we are working diligently to rectify it as quickly as possible.

For any urgent matters, please don't hesitate to contact our support team via helpdesk@pantheon.io or by initiating a support chat.

We appreciate your understanding and patience during this time.
Posted May 18, 2023 - 11:28 PDT
Update
As of now, there is no significant update or specific information to report. The current status remains unchanged, and there are no notable developments or events to share at this time.

We will provide another update in 2 hours or sooner as more information becomes available.

For urgent issues, please contact support via helpdesk@pantheon.io or by opening a support chat.
Posted May 18, 2023 - 08:41 PDT
Update
Pantheon Engineers are actively working on addressing the disruption in sending email. We know how important email functionality is for our customers, and we appreciate your patience while we continue working towards a fix.

We will continue to provide updates as more information becomes available.

For urgent issues, please contact support via helpdesk@pantheon.io or by opening a support chat.
Posted May 18, 2023 - 06:04 PDT
Investigating
We are currently investigating reports of issues with sending mail from customer sites.

For urgent issues, please contact support via helpdesk@pantheon.io or by opening a support chat.
Posted May 17, 2023 - 21:37 PDT
This incident affected: Customer Sites.