Status Updates not being sent
Incident Report for PagerDuty
Postmortem

Summary

On Mar 24th, 2021, from 12:14 to 13:09 UTC, a misconfiguration between internal systems responsible for status updates resulted in a backlog of status update notifications. All status updates during this time span would have seen notifications delayed until approximately 13:09 UTC when the configuration was corrected and the backlog was cleared.

What Happened

On March 24th, 2021 a minor infrastructure configuration change was applied to one of the services responsible for status update notifications. This change coupled with other unrelated changes limited the service’s access to infrastructure necessary for status update notification delivery while also complicating and extending the normal process for investigation and resolution.

Service was restored following identification of the problem and applying the necessary fix, and the status update backlog cleared soon after.

What Are We Doing About This

We will be taking additional steps to ensure that unrelated infrastructure changes are run in isolation from one another, along with investigating prevention measures for this particular type of issue going forward. For any questions, comments, or concerns, please reach out to support@pagerduty.com.

Posted Apr 01, 2021 - 19:18 UTC

Resolved
We have fully recovered from this incident
Posted Mar 24, 2021 - 13:20 UTC
Monitoring
We have identified the cause and are currently monitoring the fix.
Posted Mar 24, 2021 - 13:13 UTC
Investigating
We are experiencing an issue where Status Updates are not sent to customers. We are actively investigating this issue.
Posted Mar 24, 2021 - 13:06 UTC
This incident affected: Notification Delivery.