Live Status\Incident History\Unplanned surge in email alerts

Unplanned surge in email alerts

Degraded Performance

Opened: Jan 11, 2024, 3:07 PM UTC

Duration: 1h 30m 25s

  • Investigating
    Jan 11, 2024, 3:07 PM UTC

    Beginning on Jan 10, 2024, we've experienced an issue where a significant number of undesired email alerts are being sent out, many of them containing no events. We apologize for the inconvenience - our engineering team is treating this as an emergency and is actively working on resolving the issue. Email alerts have been put on pause for a short period of time to avoid sending undesired emails. Any valid alert that is held back while we resolve the issue will be sent out later today.

  • Identified
    Jan 11, 2024, 4:10 PM UTC

    The issue has been identified, and relates to a missed acknowledgement that an alert has been sent, resulting in continuous recurring emails for certain users. We are working on a solution.

  • Monitoring
    Jan 11, 2024, 4:28 PM UTC

    A fix has been deployed and alerts have been reactivated.

  • Resolved
    Jan 11, 2024, 4:38 PM UTC

    Problematic emails are not being sent anymore