Notifications and reporting

Email, Slack, Microsoft Teams and generic webhook alerts, remote log shipping to your object storage, and run reporting to the DLH.io platform.

An agent that runs unattended on a schedule needs to tell someone when it stops working. The agent has three independent visibility channels; configure at least one alert channel and keep run reporting on.

Channel	Carries	Configured in
Notifications (email, Slack, Teams, webhook)	Failure and warning alerts with the failed tables	`dlh_notifications`
Remote log shipping	The full run log (secrets removed) and a JSON run summary in your object storage	`dlh_log_store` or platform credentials
Platform run reporting	Run started / completed / warning / failed events to the DLH.io platform	`dlh_notifications.platform_run_report`

Notifications

dlh_notifications:
  enabled: true
  # Email via SendGrid
  sendgrid_api_key: '<YOUR_SENDGRID_API_KEY>'   # or env var DLH_SENDGRID_API_KEY
  from_email: noreply@yourcompany.com
  to_emails:
    - data-ops@yourcompany.com
  error_severity: error
  # Webhooks
  webhook_timeout_seconds: 10
  webhooks:
    - url: 'https://hooks.slack.com/services/T000/B000/XXXXXXXX'
      format: slack
    - url: 'https://prod-00.westus.logic.azure.com/workflows/...'
      format: teams
      error_severity: critical
    - url_env_var: DLH_OPS_WEBHOOK_URL
      format: generic
  platform_run_report: true

Key	Default	Description
`enabled`	`false`	Master switch for email and webhooks.
`sendgrid_api_key`	none	SendGrid API key for email. May be supplied via the `DLH_SENDGRID_API_KEY` environment variable instead.
`from_email`, `to_emails`	none	Sender and recipient list for email alerts.
`error_severity`	`error`	Threshold: `error` sends every alert (per-table failures, warnings, fatal errors); `critical` sends only fatal run failures.
`webhook_timeout_seconds`	`10`	HTTP timeout per webhook POST (1 to 120).
`webhooks[].url` / `url_env_var`	none	Webhook destination, inline or read from an environment variable so the URL never sits in the YAML.
`webhooks[].format`	required	`slack` (incoming webhook or workflow), `teams` (Workflows / Power Automate webhook) or `generic` (JSON payload).
`webhooks[].error_severity`	inherits	Per-webhook override of the threshold, for example a critical-only pager channel next to an everything channel.

What triggers an alert

Event	Severity	When
`Agent Run <id> completed with N errors`	error	One or more tables failed but the run finished. Includes the per-table summary with the failed tables and their error text.
`Agent Run <id> FATAL ERROR`	critical	The run could not complete (connection lost, storage unreachable, unhandled error).
`Agent Pre-flight Diagnostics Failed`	critical	A required preflight check (ODBC, SQL Server, storage, state database) failed before extraction.
`DLH API Bootstrap Validation Failed`	critical	The platform rejected the API key and the agent could not continue.
`DLH Trial Expiry Warning`	warning	TRIAL account within `trial_expiry_warning_days` of expiry (once per run).

A run with zero errors sends nothing to email or webhooks; success visibility comes from run reporting and the shipped logs.

Create an incoming webhook (Slack app, Incoming Webhooks, Add New Webhook to Workspace) or a Workflow Builder webhook trigger and paste its URL. Messages arrive as a formatted block with the subject, severity, details and the list of failed tables.

Use a Workflows (Power Automate) "When a Teams webhook request is received" flow that posts to a channel; Office 365 connector webhooks are being retired by Microsoft. The agent posts an Adaptive Card.

The agent POSTs a JSON document you can route into PagerDuty, Opsgenie, Datadog or your own API:

{
  "source": "dlh-agent-sql-server",
  "severity": "error",
  "subject": "Agent Run 1788202802 completed with 1 errors",
  "details": "Error extracting SalesDb.dbo.Location: ...",
  "failed_tables": [
    {
      "database": "SalesDb",
      "table": "dbo.Location",
      "message": "Schema drift detected for ..."
    }
  ],
  "failed_table_count": 1
}

Keep webhook URLs out of the YAML

Webhook URLs are secrets. Set them as machine-level environment variables on the Windows host and reference them with url_env_var. The configuration file can then be copied between machines without leaking the URL.

Remote log shipping

At the end of every run (including fatal failures) the agent uploads the run log and a JSON summary to the log store, so an operator can see what happened without logging into the Windows host:

<org>/<prj>/logs/2026-08-31/run_1788202802/dlh_agent_run_1788202802.log
<org>/<prj>/logs/2026-08-31/run_1788202802/run_1788202802_summary.json

Logs are scrubbed before upload. Passwords, keys, tokens, SAS signatures, connection-string secrets and authorization headers are replaced with ***REDACTED***.
The summary JSON contains the run id, start and end time, agent version, host, per-table outcomes (rows, sync mode, status, error text) and the fatal error if any.
The destination is dlh_log_store when configured, otherwise the platform-issued credentials. In Community mode nothing is shipped.
Local log files stay in the agent's logs\ folder and are pruned by agent_log_retention_days (default 30).

Use the shipped logs as the source of truth for support requests: the run id in a Slack alert maps directly to the folder in storage.

Platform run reporting

dlh_notifications:
  platform_run_report: true

With a validated API key the agent posts four kinds of events to the DLH.io platform:

Event	When	Purpose
`run_started`	After the bootstrap succeeds, before extraction	Heartbeat. Lets the platform detect an agent that has stopped running on schedule.
`run_completed`	End of a successful run	Per-table results (capped at 50 tables in the payload), row counts, duration and the storage location of the shipped log.
`run_warning`	End of a run with table-level errors	Same payload with the failed tables marked.
`run_failed`	Fatal error	Error text and the log location.

Reporting is best effort: a slow or unreachable platform never delays or fails a run. Set platform_run_report: false if your security policy forbids outbound metadata; extraction and storage uploads are unaffected. Run reports contain table names, counts and error messages, never row data.

Recommended baseline

platform_run_report: true (default).
One Slack or Teams webhook at error_severity: error for the data team.
Optionally a critical only webhook or email for the on-call rotation.
Confirm log shipping in the first run's log (Log shipping: uploaded 2 file(s) under logs/...) and bookmark the logs/ prefix.
Add --diagnose to your RMM health checks so a host that cannot reach SQL Server or storage is flagged before the next scheduled run.

Notifications

What triggers an alert

Remote log shipping

Platform run reporting

Recommended baseline

Page Contents