Summary

On Tuesday October 27, between approximately 19:44 UTC and 21:40 UTC we experienced an intermittent issue with our web application UI which affected the ability for specific pages in the application to load correctly. The mobile app, event ingestion, and the API were not impacted by this.

What Happened

Our front-end architecture uses a model that embeds different components into pages.  This issue was caused by a race condition when two versions of a frontend library were inadvertently loaded into the same page by different embedded components. This caused certain pages in the web app UI to render incorrectly.

The issue was resolved by reverting to a previous version of the component.

What We Are Doing About This

Our teams are investigating the events that led to two versions of the library being loaded and implementing changes to prevent recurrence.

• Improved Testing and Detection → We are working to improve our testing and deployment systems in order to detect and fix problems like these before they reach production
• Increasing Visibility →  We have combined all of our internal deployments to be able to better assess and analyze faster than before
• Cross Team Transparency → We have taken steps to make team ownership of certain responsibilities and repos more visible
• Improved Rollback Capabilities → Reducing the time needed to identify and roll back recent changes in the unlikely event that problematic code does reach production