Email Disruption

Incident Report for Pantheon Operations

Postmortem

A loophole was found in our invitation system that allowed sending unsolicited mail from our transactional email system by adding users to a strategically named organization. To alleviate this issue, outgoing messages were temporarily suspended. We have mitigated the problem by disabling our invite to team email functionality. All other transactional emails have been restored.

Posted Nov 30, 2021 - 11:09 PST

Resolved

This incident has been resolved.
Posted Nov 22, 2021 - 07:03 PST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Nov 22, 2021 - 05:34 PST

Update

No new information at this time. We will post an update in 60 minutes.
Posted Nov 22, 2021 - 05:19 PST

Update

No new information at this time. We will post an update in 60 minutes.
Posted Nov 22, 2021 - 04:20 PST

Update

No new information at this time. We will post an update in 60 minutes.
Posted Nov 22, 2021 - 03:19 PST

Update

No new information at this time. We will post an update in 60 minutes.
Posted Nov 22, 2021 - 02:19 PST

Update

No new information at this time. We will post an update in 60 minutes.
Posted Nov 22, 2021 - 01:19 PST

Update

No new information at this time. We will post an update in 60 minutes.
Posted Nov 22, 2021 - 00:19 PST

Update

No new information at this time. We will post an update in 60 minutes.
Posted Nov 21, 2021 - 23:19 PST

Update

No new information at this time. We will post an update in 60 minutes.
Posted Nov 21, 2021 - 22:19 PST

Update

No new information at this time. We will post an update in 60 minutes.
Posted Nov 21, 2021 - 21:19 PST

Update

No new information at this time. We will post an update in 60 minutes.
Posted Nov 21, 2021 - 20:19 PST

Update

No new information at this time. We will post an update in 60 minutes.
Posted Nov 21, 2021 - 19:19 PST

Update

We are continuing to work on a fix for this issue.
Posted Nov 21, 2021 - 18:18 PST

Identified

The issue has been identified and a fix is being implemented.
Posted Nov 21, 2021 - 17:17 PST

Update

We are continuing to investigate this issue.
Posted Nov 21, 2021 - 16:09 PST

Investigating

We are aware of an issue that is preventing emails from being sent when performing certain actions on the Pantheon dashboard. This is impacting the deliverability of password reset emails and notification emails when adding new team members to accounts.

This service disruption does not affect any site-specific email functionality.

Our Engineering team is actively working towards a resolution.
Posted Nov 21, 2021 - 15:09 PST