Apache Solr Service Degredation
Incident Report for Pantheon Operations
Postmortem

On 01/09/2017 and 01/10/2017, we experienced a degradation in performance on one of our Apache Solr servers. We were able to remediate the issue for most customers impacted within 30 minutes. During this time, the impacted customers who rely on Apache Solr to provide robust search capability found that capability diminished.

Apache Solr is a powerful service that Pantheon has been able to reliably offer its customers with the most content rich sites. We are working to increase the speed with which we can remediate this particular failure in the future. Additionally, we will look into ways to further reduce the occurrence of these sorts of incidents.

We apologize for any inconvenience caused by this incident.

Posted Jan 11, 2017 - 14:19 PST

Resolved
Incident is resolved: performance has remained steady and no further issues have been reported on solr. Thanks again for your patience.
Posted Jan 10, 2017 - 17:18 PST
Update
Performance continues to hold steady, but we will continue to monitor until issue has been resolved. Thanks again for your patience.
Posted Jan 10, 2017 - 16:39 PST
Monitoring
We have remediated the solr service incident and we will be monitoring for performance. Thank you for your patience.
Posted Jan 10, 2017 - 14:22 PST
Investigating
We are investigating degraded Apache Solr performance affecting a small number of customer sites.
Posted Jan 10, 2017 - 13:06 PST