On May 3 2017 at 20:01 UTC, PagerDuty suffered a service degradation affecting our Events API; this incident lasted for three hours. Customers would have experienced difficulties sending events to the PagerDuty Events API. We apologize to any customers who were affected by the outage.
At 19:12 UTC, PagerDuty began maintenance on one of the Cassandra-based services responsible for processing events from the Events API. During this maintenance, the Cassandra cluster became unstable while engineers increased the capacity of the overall system. PagerDuty engineers were immediately alerted to the issue and worked to bring the cluster back into a stable state. At 22:10 UTC, the cluster was stable and the API was able to process events.
To avoid future issues like this one, we have put additional checks into place around how we scale our Cassandra cluster. We sincerely apologize if this degradation negatively impacted your team's usage of PagerDuty. If you have questions or concerns please contact us at email@example.com