Getting Started
Install the DLH.io SQL Server Agent on Windows, prepare SQL Server, create your first configuration and complete a first successful run.
This guide takes a new installation from an empty Windows server to a first successful sync in about an hour. If you already have the agent running and want to change how it behaves, jump to the Configuration reference.
Before you begin
Host requirements
| Requirement | Recommendation |
|---|---|
| Operating system | Windows Server 2016 or later, or Windows 10 / 11 (64-bit). The agent is delivered as a Windows installer and runs as a scheduled task. |
| CPU and memory | 2 vCPU and 8 GB RAM minimum. 16 GB is recommended for databases with hundreds of large tables. The agent streams rows in batches and pauses when host memory exceeds max_memory_percent (default 85%). |
| Disk | 20 GB free on the install volume for logs, audit files and the local state database. If you stage data locally before upload (direct_cloud_write: false) allow for the size of the largest full extract. |
| Network egress (HTTPS, port 443) | api.datalakehouse.io (license and run reports), your Amazon S3 or Azure Storage endpoints, and any webhook endpoints you enable (Slack, Microsoft Teams, SendGrid). No inbound ports are required. |
| SQL Server connectivity | TCP access from the agent host to the SQL Server instance (default port 1433). |
| ODBC driver | Microsoft ODBC Driver 18 for SQL Server (Driver 17 is also supported). The agent can install it for you, see step 2 below. |
| Permissions on the host | Local administrator for installation. The scheduled task runs under a dedicated local service account that the installer creates. |
SQL Server requirements
- A SQL Server instance that supports Change Tracking or Change Data Capture if you want incremental sync (full extracts work on any edition). CDC also requires the SQL Server Agent service to be running on the SQL Server instance because the capture and cleanup jobs run there.
- A SQL login for the agent with, at minimum,
SELECTon every table and view you plan to sync. For Change Tracking also grantVIEW CHANGE TRACKINGon those tables. For CDC the login must be able to read thecdcschema (membership in the gating role used when CDC was enabled, ordb_owner). - Tables synced with
ctorcdcmust have a primary key. Tables without one fall back to the strategy inno_pk_strategy(full extract by default).
-- Once per database. Retention should comfortably exceed your sync interval
-- and any expected downtime; 2 days is the SQL Server default.
ALTER DATABASE [YourDb]
SET CHANGE_TRACKING = ON (CHANGE_RETENTION = 2 DAYS, AUTO_CLEANUP = ON);
-- Once per table. Repeat for every table you want to sync incrementally.
ALTER TABLE [dbo].[Customer] ENABLE CHANGE_TRACKING WITH (TRACK_COLUMNS_UPDATED = OFF);
-- Agent login
CREATE LOGIN dlh_agent WITH PASSWORD = '<strong password>';
USE [YourDb];
CREATE USER dlh_agent FOR LOGIN dlh_agent;
GRANT SELECT ON SCHEMA::dbo TO dlh_agent;
GRANT VIEW CHANGE TRACKING ON SCHEMA::dbo TO dlh_agent;USE [YourDb];
EXEC sys.sp_cdc_enable_db;
EXEC sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name = N'Customer',
@role_name = N'cdc_reader',
@supports_net_changes = 1;
CREATE LOGIN dlh_agent WITH PASSWORD = '<strong password>';
CREATE USER dlh_agent FOR LOGIN dlh_agent;
GRANT SELECT ON SCHEMA::dbo TO dlh_agent;
ALTER ROLE cdc_reader ADD MEMBER dlh_agent;Change Tracking retention
If the agent cannot complete a successful sync of a table for longer than CHANGE_RETENTION, SQL Server purges the change history and the agent must perform a full reload of that table. Keep retention at least several times longer than your sync interval and monitor failed tables (see Operations).
From your DLH.io account
Create a source connection of type SQL Server Agent in the DLH.io portal (or ask your account manager to do it). You will need the following values for the dlh_ref section of the configuration:
- Organization GUID (
org_guid) - Project GUID (
prj_guid) - Connection GUID of the target connection the agent extracts for (
connection_guid) - Agent API key (
api_key) - A short unique prefix for this agent, for example
Site003(target_schema_prefix)
Do not have an account yet? Contact us to start a trial. Without an API key the agent still runs in Community mode (local output only, 10,000 rows per table) so you can evaluate it on a workstation.
Installation
Download and run the installer
The installer is provided by your DLH.io account manager or from your DLH.io portal. It is named dlh-agent-for-sql-server-installer-v<YYYYMMDD>.exe.
Run it as an administrator and accept the defaults. The installer:
- Installs to
C:\Program Files\DLH.io Agent for SQL Server\ - Places
dlh_agent_sql_server.exe, the_internal\runtime folder,dlh_agent_config.template.yamland a starterdlh_sql_server_credentials.yaml - Creates a local service account (
svc_dlh_agent_runnerby default) with a generated password stored in Windows Credential Manager - Registers a Windows Task Scheduler task named
DLH.io SQL Server Agent Xthat runs the agent daily at 22:01 UTC (you will change the schedule later)
Upgrades reuse the same folder, replace the EXE and _internal\ folder, and never overwrite dlh_agent_config.yaml, your credentials, the encryption key or the state database.
SmartScreen and antivirus
If Windows SmartScreen or endpoint protection prompts on first run, choose More info and Run anyway, or have your security team allow the installer hash. Add the install directory to real-time scanning exclusions to avoid file locks on the state database during a run.
Install the ODBC driver
Open an elevated terminal (Command Prompt or PowerShell) in the install directory and run:
cd "C:\Program Files\DLH.io Agent for SQL Server"
.\dlh_agent_sql_server.exe --install-driver --auto--install-driver prints guided instructions; adding --auto downloads and installs Microsoft ODBC Driver 18 for SQL Server silently. Skip this step if Driver 17 or 18 is already installed.
Create the configuration file
Copy the template and open it in a text editor:
copy dlh_agent_config.template.yaml dlh_agent_config.yaml
notepad dlh_agent_config.yamlFor a first run keep the file small. The example below syncs every table in the dbo schema of one database to Delta Lake on Azure with credentials issued by the DLH.io platform:
agent_version: 1.9.4
state_db_path: dlh_agent_state.duckdb
connection_information:
connection_type: sql_server
server_name_or_ip: 'sql01.corp.local'
server_port: 1433
database_names: [SalesDb]
data_retrieval:
historical_load: false
output_format: delta
direct_cloud_write: true
storage_base_path: /data/delta_iceberg_tables
output_path: ./output
databases:
- name: SalesDb
sync_mode: auto
tables:
- name: dbo.*
all_columns: true
sync_mode: auto
no_pk_strategy: full
dlh_ref:
org_guid: '<YOUR_ORG_GUID>'
prj_guid: '<YOUR_PROJECT_GUID>'
connection_guid: '<YOUR_CONNECTION_GUID>'
target_schema_prefix: 'Site003'
api_key: '<YOUR_DLH_API_KEY>'
dlh_notifications:
enabled: true
platform_run_report: true
webhooks:
- url_env_var: DLH_OPS_WEBHOOK_URL
format: slackNotes:
- Use forward slashes in every path, including on Windows (
C:/dlh/output). dlh_storageanddlh_log_storeare omitted on purpose: with an API key the platform issues the storage credentials at run time. Add them only if you want to use your own static credentials (see Storage and bootstrap).- The plaintext API key is encrypted in place on the first run.
Enter the SQL Server credentials
Open dlh_sql_server_credentials.yaml and set the login you created earlier:
credentials:
encryption: no
credentials_file: dlh_sql_server_credentials.sec
username: dlh_agent
password: '<strong password>'On the first run the agent generates an encryption key (dlh_agent_secret.key), writes the encrypted password to dlh_sql_server_credentials.sec, replaces the plaintext password in the YAML with <encrypted> and sets encryption: yes. The plaintext password never needs to be on disk again. See Security for key management options including the DLH_AGENT_SECRET_KEY environment variable.
Check the host before extracting anything
.\dlh_agent_sql_server.exe --config-check
.\dlh_agent_sql_server.exe --diagnose--config-check compares your file with the template and lists missing or unknown keys. --diagnose is a read-only health check: it verifies CPU, memory and disk, the ODBC driver, the configuration and credential files, the DLH.io license, SQL Server connectivity, Change Tracking / CDC status and storage access, and prints PASS, WARN or FAIL per check. It never extracts data or moves sync pointers. Fix every FAIL before continuing; WARN lines are informational.
Run the agent once from a terminal
Always run the first sync from a terminal rather than by double-clicking the EXE so that you can see the output:
.\dlh_agent_sql_server.exeWhat a healthy first run looks like:
- The DLH.io banner prints, followed by a tier status line such as
DLH credentials OK [CUSTOMER]orTRIAL: 29 day(s) remaining - expires 2026-10-08. - The
DLH.io AGENT STATE STORE (DuckDB)block reportsStatus: CREATED on this run (no prior state)because there is no state yet. - Preflight checks all report
PASS. - Because there is no prior state, the run is treated as a historical (full) load of every table, regardless of
sync_mode. - The run summary lists every table with a row count and
OK, and theERRORSblock is absent. - Data appears in your storage account under
<storage_base_path>/delta/<Database>.<schema>.<Table>/(Delta) or<storage_base_path>/iceberg/...(Iceberg). CSV and Parquet output uses aYYYY/MM/DD/<epoch>/folder per run.
Run the agent a second time. Tables with Change Tracking enabled now report ct in the mode column and only changed rows are written; tables with no changes print No CT/CDC changes - skipped write.
Schedule it
The installer registered a daily task. Most customers change it to run every 5 to 15 minutes for Change Tracking sources. Open Task Scheduler, edit DLH.io SQL Server Agent X, and on the Triggers tab set Repeat task every: 5 minutes for a duration of Indefinitely. Keep Do not start a new instance on the Settings tab so that overlapping runs never happen. See Operations for RMM deployment and monitoring.
Next steps
- Tune what is synced and how: Data retrieval and Write strategies.
- Choose between Delta Lake and Iceberg and connect Snowflake or Databricks: Output formats.
- Get alerted when something fails: Notifications and reporting.
- Understand each log section: Monitoring and logs.