Monitoring and logs

Where to look

The run id (a Unix timestamp, for example 1788202802) ties all of these together: it appears in the log file name, the shipped log folder, the run summary, alerts and run reports.

Anatomy of a run log

A successful run prints these blocks in order. Read them top to bottom when something looks wrong.

1. Boot banner and version

Version           : 1.9.4
Run ID            : 1788202802
Started (UTC)     : 2026-08-31 15:00:02
Host              : SQLAGENT01 (win32)
Python            : 3.12.4
Working directory : C:\Program Files\DLH\dlh-agent-sql-server
Config file       : dlh_agent_config.yaml
Log file          : ./logs/dlh_agent_20260831_1788202802.log
Output format     : delta
DLH org_guid      : ...
DLH prj_guid      : ...
DLH conn_guid     : ...
================================================================

Confirms which version ran, from which directory and against which DLH.io connection. If the version or working directory is not the one you deployed, the scheduled task is pointing at an old install.

2. Steps

Every phase is bracketed by [STEP START] <name> @ <time> and [STEP FINISH] <name> @ <time> (duration: Ns) (or [STEP FAILED] with the error). Step names in order: Load Configuration, Initialize State Store (DuckDB), Initialize Notifiers, Credential Decryption, DLH Platform Bootstrap Validation, Database Connection Check, Setup Output Directory, Fetch Data (Full / CT / CDC), Schema Metadata Collection, Compress Output Files, Upload to Storage, Upload Logs + Platform Run Report, Comprehensive Run Summary, Post-Historical-Load Cleanup, Post-Completion Lifecycle Actions.

Comparing step durations across runs is the quickest way to spot a regression (a fetch step that grew from 40 seconds to 9 minutes points at a missing CT index or a table that fell back to full).

3. Tier and license banner

================================================================
  DLH.io AGENT TIER / LICENSE STATUS
================================================================
  Tier              : CUSTOMER
  Connection        : Sales Azure Raw
  Storage provider  : AZURE_BLOB
  Bucket/Container  : sales-raw
  Credential source : Live DLH bootstrap
  Row cap           : Unlimited
  Cloud uploads     : Enabled
================================================================

Things to check: the tier you expect, Credential source (live bootstrap versus cached, or a Community fallback reason), a TRIAL countdown line if applicable. A LOCAL COMMUNITY MODE banner on a paid connection means the API key was rejected or unreachable; nothing will be uploaded. See Storage and bootstrap.

4. State store telemetry

================================================================
  DLH.io AGENT STATE STORE (DuckDB)
================================================================
  Path              : C:\Program Files\DLH\dlh-agent-sql-server\dlh_agent_state.duckdb
  Status            : Existing
  File size         : 48.2 MB (+ WAL 1.1 MB)
  Created           : 2026-05-02 09:14:31
  Last modified     : 2026-08-31 15:02:10
  DuckDB            : v1.1.3, threads=8, memory_limit=12.4 GiB, memory_usage=0 bytes
  Tables tracked    : 312 (ct=298, full=14)
  Run history       : 4,120 run(s), 1,285,440 table-run row(s)
  Tables last FAILED: 1 (most recent recorded outcome was an error)
  Last success      : 2026-08-31 15:02:10 (run 1788202502)
  Pending drift     : 1 table(s) awaiting schema drift remediation
  Disk (state vol)  : 212.4 GB free of 476.3 GB (44.6% free)
  Agent process RSS : 186.2 MB
================================================================

This block answers the questions support asks first: is the state file the one you think it is, is it growing, when did the last good run happen, are tables stuck failing, is anything waiting for drift remediation, is the disk filling up. Status: CREATED on this run (no prior state) on a host that has been running for weeks means the state file was deleted or the task is running from a different directory; every table will be reloaded in full.

5. Per-table extraction

Each table logs its sync mode, the CT/CDC version range or full extract, row counts per batch, sanitization notes, the write result (for Delta: Delta merge complete ... version N, total rows: M) and any schema drift message. Tables with nothing to do log No CT/CDC changes - skipped write.

6. Upload and log shipping

Uploading files to storage service ...
Log shipping: uploaded 2 file(s) under logs/2026-08-31/run_1788202802/

With direct_cloud_write: true Delta and Iceberg tables report Data saved to abfss://... (or s3://...) at write time and the upload step only handles CSV, Parquet and SQL query outputs.

7. Run summary

================================================================================
  DLH.io AGENT RUN SUMMARY
================================================================================
  Run ID          : 1788202802
  Run Type        : Incremental (CT/CDC/full)
  Output Format   : delta
  Upload Strategy : end_of_run
  Fetch Batch Size: 100,000
  Memory Threshold: 85%
--------------------------------------------------------------------------------
  Tables/Views Processed : 312
  Succeeded              : 311
  Failed                 : 1
  Total Rows Extracted   : 48,913
--------------------------------------------------------------------------------
  Database  Table                     Type    Mode      Rows          Status    Notes
  --------  ------------------------  ------  --------  ------------  --------  --------------------
  SalesDb   dbo.Customer              table   ct                 122  OK
  SalesDb   dbo.Location              table   ct                   0  FAILED    Schema drift detected for ...
  SalesDb   dbo.Note                  table   ct                  31  OK
  ...
--------------------------------------------------------------------------------

  ERRORS (1):
    1. Error extracting SalesDb.dbo.Location: Schema drift detected for .../delta/SalesDb.dbo.Location:
       [GeocodeValidated: source=string target=bool (incompatible)] - on_schema_drift_action_handling=fail_once: ...

The summary is the same content that alerts and run reports carry. A run with Failed: N still exits 0; the failures are surfaced through this block, the notification channels and the run_warning platform event.

What healthy looks like

For a five-minute CT schedule on a stable estate:

• Every run: tier banner shows the expected tier, Failed: 0, most tables No CT/CDC changes - skipped write, a handful with small row counts, log shipping uploaded 2 files.
• Day over day: Delta versions of active tables increase, Last success in the telemetry block is within the last interval, Tables last FAILED: 0, Pending drift: 0, state file size growing slowly.
• Platform: run_started and run_completed heartbeat pairs at the expected cadence.

Fleet monitoring

For more than a couple of agents, do not read logs; watch signals:

1. Platform heartbeats. With platform_run_report: true every run posts a run_started event. The DLH.io platform surfaces agents whose last heartbeat is older than expected, which catches a disabled task, a powered-off VM or blocked outbound HTTPS without any log parsing.
2. Failure and warning alerts. One Slack or Teams channel for error severity across the fleet; a pager integration at critical only.
3. Silence detection on your side. If you cannot rely on the platform view, alert when the newest object under <org>/<prj>/logs/ is older than three intervals, or run --diagnose --json from your RMM and alert on exit code and last_successful_run_at.
4. Downstream freshness. Monitor the consuming warehouse (for example the latest Delta commit timestamp or Iceberg snapshot time, or MAX(ModifiedDate) on a busy table) so an agent that runs but writes nothing new is noticed. Most tables reporting zero changes is normal; every table reporting zero changes for a day on a busy system is not.

Log verbosity and retention

• Logs are written at INFO level with one line per step, per table and per batch. There is no debug switch in the customer build; the audit trail (produce_audit_trail: true) is the detailed view when you need queries and samples.
• agent_log_retention_days (default 30) prunes local logs; shipped logs follow your object storage lifecycle rules.
• Task Scheduler's own history (Last Run Result 0x0) only tells you that the process exited; use the run summary or alerts for the outcome.