Skip to content

Alert when a run fails

Say the nightly sync fails now and then, and nobody notices until the morning. Here’s how to post an alert to Slack the moment it fails, using a trigger. The alert includes the error and a link to the run.

A workspace owner or admin saves a Slack bot token with permission to post messages (chat:write), and lets Reminix send it only to slack.com:

Terminal window
printf %s "$SLACK_BOT_TOKEN" | reminix secrets set SLACK_TOKEN --hosts slack.com
{
"slug": "alert-on-failure",
"name": "Alert on failure",
"command": "sh alert.sh",
"hosts": ["slack.com"],
"secrets": {
"SLACK_TOKEN": {
"host": "slack.com",
"header": "Authorization",
"format": "Bearer {secret}"
}
},
"inputSchema": {
"type": "object",
"required": ["runId"],
"properties": {
"runId": { "type": "string", "title": "Failed run" },
"script": { "type": "string", "title": "Script" },
"error": { "type": "string", "title": "Error" }
}
}
}

A short shell script does it. It reads its inputs from the JSON file Reminix names in $REMINIX_INPUTS_FILE:

alert.sh
set -e
text=$(jq -r '(.script // "A script") + " failed.\nError: " + (.error // "none given") +
"\nhttps://app.reminix.com/runs/" + .runId' "$REMINIX_INPUTS_FILE")
answer=$(jq -n --arg text "$text" '{channel: "#alerts", text: $text}' |
curl -s https://slack.com/api/chat.postMessage -H "Content-Type: application/json" -d @-)
if [ "$(echo "$answer" | jq .ok)" != true ]; then
echo "Slack: $(echo "$answer" | jq -r .error)" >&2
exit 1
fi
echo '{"posted": true}'

Publish it:

Terminal window
reminix publish ./alert-on-failure

On the alert script’s page, choose Triggers → Add trigger. Pick the event run.failed, run it only when script.slug equals nightly-sync, and take the inputs from the event. Through the API:

Terminal window
curl -X POST https://api.reminix.com/v1/scripts/alert-on-failure/triggers \
-H "Authorization: Bearer $REMINIX_TOKEN" -H "X-Workspace: acme" \
-H "Content-Type: application/json" \
-d '{
"type": "event",
"name": "Nightly sync failed",
"event": "run.failed",
"filter": { "script.slug": "nightly-sync" },
"inputs": {
"runId": "{{event.data.runId}}",
"script": "{{event.data.script.slug}}",
"error": "{{event.data.error}}"
}
}'

The alert runs as you, within about a minute of the failure. Reminix records the run on the alert script’s page, marked Trigger.

  • Reminix stops a chain of triggered runs after 3, so an alert that fails can’t set off an endless loop.
  • On the sync’s page, the Used by card lists the alert’s trigger under Automations. Check there before changing or retiring the sync.

Related: Schedules and triggers, Webhooks.