Apache Solr service degradation

Incident Report for Pantheon Operations

Postmortem

At roughly 12:15pm EST May 23rd, 2023, a configuration change was made to prepare for upcoming planned infrastructure upgrades for Pantheon’s CMS Search offering. This change unexpectedly resulted in CMS Search being unavailable while the configuration was applied. This change did not have any impact on the machines running the Search backends or their data, and service was restored around 1:30pm EST.

Maintaining availability of these services is a high priority issue for Pantheon. As part of our internal review process, we have discovered areas to help us uncover and prevent similar issues in the future.

Posted May 30, 2023 - 14:17 PDT

Resolved

Issue with Solr search has been resolved for all affected users. We apologize for any inconvenience caused and appreciate your patience.

Posted May 23, 2023 - 11:18 PDT

Monitoring

The upgrade process has completed and Solr services are now restored.

We apologize for the unexpected interruption to services today, thank you for your patience with us as we worked to resolve this issue.

Please reach out to support by opening a ticket or chat if there are any remaining issues.

Posted May 23, 2023 - 10:47 PDT

Identified

We are investigating a partial outage for Apache Solr, affecting US region sites using Solr search. An upgrade is currently in progress and we expect to see improvement within 30 minutes and will update at that time.

For urgent issues, please contact support via helpdesk@pantheon.io or by opening a support chat.

Posted May 23, 2023 - 10:25 PDT

This incident affected: Customer Sites.