Delayed Notifications
Incident Report for PagerDuty
Postmortem

Summary

On June 27th, from 5:52 pm to 6:31 pm UTC, we suffered a delay in sending notifications to customers. During this period we were able to accept all incoming events, and no events were dropped.

What Happened?

A deploy caused unforeseen errors, which blocked the processing of events. We quickly reverted to change to solve the issue.

What Are We Doing About This?

Our engineers are currently working on improving our alerting and testing infrastructure to prevent this issue from occurring in the future.

We would like to again apologize for any inconvenience this issue caused. If you have any questions, do not hesitate to contact us at support@pagerduty.com.

Posted about 1 year ago. Jul 07, 2017 - 23:48 UTC

Resolved
This incident has been resolved and notifications are processing normally. We are fully recovered at this time.
Posted about 1 year ago. Jun 27, 2017 - 18:48 UTC
Identified
We have identified the issue and notifications are recovering. We are monitoring for any further delays and other services remain functional at this time.
Posted about 1 year ago. Jun 27, 2017 - 18:44 UTC
Investigating
We are accepting events although notification processing/incident creation is delayed at this time. All other services are functional at this time.
Posted about 1 year ago. Jun 27, 2017 - 18:21 UTC