SQL Server Agent
Overview of the DLH.io SQL Server Agent: a Windows-native agent that extracts SQL Server data with Change Tracking or CDC and writes CSV, Parquet, Delta Lake or Apache Iceberg to your own cloud storage.
The DLH.io SQL Server Agent is a self-contained Windows executable that runs on a server inside your network, reads from one or more Microsoft SQL Server databases, and writes the data to Amazon S3 or Azure Blob Storage / ADLS Gen2 in the format you choose. It keeps its own local state so that every run after the first only moves the rows that changed.
Product page: datalakehouse.io/tools/sql-server-agent
How it works
SQL Server (on-prem / private cloud)
|
| ODBC (Change Tracking, CDC or full extract)
v
+------------------------------+
| DLH.io SQL Server Agent | local state: dlh_agent_state.duckdb
| Windows EXE, Task Scheduler | config: dlh_agent_config.yaml
+------------------------------+ credentials: encrypted at rest
|
| CSV / Parquet / Delta Lake / Apache Iceberg
v
Amazon S3 or Azure Blob / ADLS Gen2
|
v
Snowflake, Databricks, Trino, DuckDB, DLH.io Sync Bridge ...Each run follows the same sequence:
- Boot and license check. The agent prints a startup banner, validates its API key against the DLH.io platform and resolves your tier (Customer, Trial or Community). Storage credentials are issued by the platform for the run, or read from static configuration if you provide them.
- State store. The agent opens its local DuckDB state database and prints a telemetry block (size, tables tracked, last successful run, pending schema drift, free disk).
- Preflight. ODBC driver, SQL Server connectivity, Change Tracking / CDC status, storage access and notification channels are checked before any data moves.
- Extraction. For each configured table or view the agent chooses full extract, Change Tracking (CT) or Change Data Capture (CDC), streams rows in batches with memory throttling, applies optional column masks, and writes the batch in the configured format and write strategy.
- Upload. Data is written directly to cloud storage (recommended for Delta and Iceberg) or staged locally and uploaded at the end of the run, per table, or in batches.
- Summary and reporting. A per-table run summary is printed and logged, logs are shipped to your log storage, the run report is sent to the DLH.io platform, notifications go out for any failures, and optional lifecycle actions (Sync Bridges, CI/CD dispatches, DBDeux jobs) are triggered.
Key capabilities
| Capability | Details |
|---|---|
| Sync modes | full, ct (Change Tracking), cdc (Change Data Capture) or auto per database and per table |
| Output formats | CSV (gzip), Parquet, Delta Lake, Apache Iceberg |
| Write strategies | merge (default), append, replace, truncate+insert, delete+insert, scd2 (SCD Type 2 history) |
| Cloud targets | Amazon S3, Azure Blob Storage / ADLS Gen2 (SAS token or account key) |
| Schema drift | Detects incompatible column type changes and can fail, coerce or migrate the target table |
| Declared types | Decimal, numeric and money columns are typed from the SQL Server declared precision and scale |
| Tables without a primary key | Configurable fallback (full, truncate_reload, drop_recreate) |
| Wildcards | dbo.* style table and view selection with per-table overrides |
| Column masking | hash, partial, redact, round |
| Custom SQL | Run .sql files per database and store results as CSV or Parquet |
| Alerts | Email (SendGrid), Slack, Microsoft Teams, generic JSON webhooks |
| Platform reporting | Run start, completion, warning and failure reports to DLH.io for run history and missed-run detection |
| Log shipping | Secret-scrubbed logs and a JSON run summary uploaded to your log storage per run |
| Lifecycle actions | Trigger DLH.io Sync Bridges, GitHub / GitLab / Bitbucket pipelines and DBDeux scheduled jobs |
| Diagnostics | --diagnose read-only health check with JSON output for support |
| Deployment | Windows installer, scheduled task with a dedicated service account, unattended and RMM friendly |
Plans and tiers
The tier is resolved from dlh_ref.api_key on every run and shown in the startup banner.
| Tier | Extraction | Cloud uploads | Notes |
|---|---|---|---|
| Customer | Uncapped | Enabled | Storage credentials issued by the platform or configured statically |
| Trial | Uncapped | Enabled | Time limited. The banner shows a countdown and the agent sends a warning notification as expiry approaches |
| Community | Capped at 10,000 rows per table or view | Disabled | Runs locally with no API key or when the key is missing, invalid or rejected. Output stays in the local output_path |
Upgrade from Community
Community mode is intended for evaluation on a workstation. To sync full tables and write to your cloud storage, contact us for a Trial or Customer plan and we will provision the API key and connection for your agent.
Documentation map
- Getting started: requirements, installation, first configuration and first run.
- Configuration: every section of
dlh_agent_config.yaml. - CLI reference: all command line flags and exit codes.
- Operations: unattended scheduling, monitoring, the state store and troubleshooting.
- Scenarios: copy-ready configurations for common deployments (CT to Delta on Azure, CDC to Iceberg on S3, SCD2 history, multi-database, Community mode, SQL queries, CI/CD triggers).
- Security: credential handling, encryption, network egress.
- FAQ
- Changelog
Versioning
The agent and its configuration template share one version number, agent_version, which is printed by --version and written at the top of the configuration template. The changelog is organized by that version. The current version is 1.9.4.