Help / Alerts
PagerDuty
A PagerDuty channel opens an incident in PagerDuty when a monitor goes down and resolves it when the monitor comes back. Who gets paged, and how, is then up to your on-call and escalation settings in PagerDuty.
What you need first
- A PagerDuty account, with permission to add integrations to a service.
- A PagerDuty service for the alerts, or permission to make one.
Get an integration key in PagerDuty
If these steps do not match what you see, follow PagerDuty's own guide, Services and Integrations.
- In PagerDuty, open the service that should get the alerts.
- Open its Integrations tab and add an integration.
- Choose Events API V2.
- Copy the Integration Key. It is 32 letters and numbers.
Add it in HonkWatch
- Open Alerts and go to Add a channel.
- Under Where should alerts go?, choose PagerDuty.
- Paste the integration key into Send to.
- If you like, type the PagerDuty service name in Label (optional).
- Tick the events. Keep both Down and Back up ticked, so incidents are resolved automatically.
- Press Add channel. The list shows the last four characters of the key.
What happens in PagerDuty
| Event | In PagerDuty |
|---|---|
| Down | Opens an incident with severity critical. |
| Back up | Resolves the incident that the down alert opened. |
| Certificate expiring | Opens a separate incident with severity warning. |
| Slow | Opens a separate incident with severity warning. |
Each monitor has one down incident at a time. If you set the monitor to repeat its down alert, the repeats go to the same incident rather than opening new ones.
Certificate and slow incidents are not resolved automatically. Resolve them in PagerDuty once you have dealt with them.
The incident's summary is the alert's subject line: the service name in square brackets, then for example "Shop is DOWN". The event also carries the monitor's name, type and address, the reason, the monitor's status and a link back to the monitor's page.
Changed DNS answers and maintenance window notices are not sent to PagerDuty. Use another channel, such as email, for those.
Send a test
- In the channel list on Alerts, press Send test on the PagerDuty row.
- A new incident about "Test monitor" should open in PagerDuty, with severity
info. - Resolve the test incident yourself in PagerDuty. It is not resolved automatically.
Depending on how the PagerDuty service is set up, a test can page whoever is on call. Tell them before you press it.
If it does not work
- You see "A PagerDuty integration key is 32 letters and numbers". You copied something else, such as a service ID or an API key. Copy the Integration Key from the Events API V2 integration.
- Recently sent shows failed. PagerDuty refused the event. The integration may have been deleted or the service disabled. Make a new Events API V2 integration, add its key as a new channel here, test it, then remove the old channel.
- Recently sent shows sent, but no incident opened. Check that you used the key of the right service, and look at that service's event rules or orchestration in PagerDuty, which can drop or change events.
- Incidents stay open after the monitor recovers. Check that Back up is ticked on the channel, and that the monitor's setting Send a "back online" message when it recovers is on. Without both, no resolve is sent.
See also Setting up alerts and Incidents.
Still stuck? Email [email protected] and say which guide you were following.