On Mar 24th, 2021, from 12:14 to 13:09 UTC, a misconfiguration between internal systems responsible for status updates resulted in a backlog of status update notifications. All status updates during this time span would have seen notifications delayed until approximately 13:09 UTC when the configuration was corrected and the backlog was cleared.
On March 24th, 2021 a minor infrastructure configuration change was applied to one of the services responsible for status update notifications. This change coupled with other unrelated changes limited the service’s access to infrastructure necessary for status update notification delivery while also complicating and extending the normal process for investigation and resolution.
Service was restored following identification of the problem and applying the necessary fix, and the status update backlog cleared soon after.
We will be taking additional steps to ensure that unrelated infrastructure changes are run in isolation from one another, along with investigating prevention measures for this particular type of issue going forward. For any questions, comments, or concerns, please reach out to email@example.com.