DLH.io Documentation logoDLH.io Documentation
AgentsSQL Server Agent

Getting Started

Install the DLH.io SQL Server Agent on Windows, prepare SQL Server, create your first configuration and complete a first successful run.

This guide takes a new installation from an empty Windows server to a first successful sync in about an hour. If you already have the agent running and want to change how it behaves, jump to the Configuration reference.

Before you begin

Host requirements

RequirementRecommendation
Operating systemWindows Server 2016 or later, or Windows 10 / 11 (64-bit). The agent is delivered as a Windows installer and runs as a scheduled task.
CPU and memory2 vCPU and 8 GB RAM minimum. 16 GB is recommended for databases with hundreds of large tables. The agent streams rows in batches and pauses when host memory exceeds max_memory_percent (default 85%).
Disk20 GB free on the install volume for logs, audit files and the local state database. If you stage data locally before upload (direct_cloud_write: false) allow for the size of the largest full extract.
Network egress (HTTPS, port 443)api.datalakehouse.io (license and run reports), your Amazon S3 or Azure Storage endpoints, and any webhook endpoints you enable (Slack, Microsoft Teams, SendGrid). No inbound ports are required.
SQL Server connectivityTCP access from the agent host to the SQL Server instance (default port 1433).
ODBC driverMicrosoft ODBC Driver 18 for SQL Server (Driver 17 is also supported). The agent can install it for you, see step 2 below.
Permissions on the hostLocal administrator for installation. The scheduled task runs under a dedicated local service account that the installer creates.

SQL Server requirements

  • A SQL Server instance that supports Change Tracking or Change Data Capture if you want incremental sync (full extracts work on any edition). CDC also requires the SQL Server Agent service to be running on the SQL Server instance because the capture and cleanup jobs run there.
  • A SQL login for the agent with, at minimum, SELECT on every table and view you plan to sync. For Change Tracking also grant VIEW CHANGE TRACKING on those tables. For CDC the login must be able to read the cdc schema (membership in the gating role used when CDC was enabled, or db_owner).
  • Tables synced with ct or cdc must have a primary key. Tables without one fall back to the strategy in no_pk_strategy (full extract by default).
-- Once per database. Retention should comfortably exceed your sync interval
-- and any expected downtime; 2 days is the SQL Server default.
ALTER DATABASE [YourDb]
SET CHANGE_TRACKING = ON (CHANGE_RETENTION = 2 DAYS, AUTO_CLEANUP = ON);

-- Once per table. Repeat for every table you want to sync incrementally.
ALTER TABLE [dbo].[Customer] ENABLE CHANGE_TRACKING WITH (TRACK_COLUMNS_UPDATED = OFF);

-- Agent login
CREATE LOGIN dlh_agent WITH PASSWORD = '<strong password>';
USE [YourDb];
CREATE USER dlh_agent FOR LOGIN dlh_agent;
GRANT SELECT ON SCHEMA::dbo TO dlh_agent;
GRANT VIEW CHANGE TRACKING ON SCHEMA::dbo TO dlh_agent;
USE [YourDb];
EXEC sys.sp_cdc_enable_db;

EXEC sys.sp_cdc_enable_table
  @source_schema = N'dbo',
  @source_name   = N'Customer',
  @role_name     = N'cdc_reader',
  @supports_net_changes = 1;

CREATE LOGIN dlh_agent WITH PASSWORD = '<strong password>';
CREATE USER dlh_agent FOR LOGIN dlh_agent;
GRANT SELECT ON SCHEMA::dbo TO dlh_agent;
ALTER ROLE cdc_reader ADD MEMBER dlh_agent;

Change Tracking retention

If the agent cannot complete a successful sync of a table for longer than CHANGE_RETENTION, SQL Server purges the change history and the agent must perform a full reload of that table. Keep retention at least several times longer than your sync interval and monitor failed tables (see Operations).

From your DLH.io account

Create a source connection of type SQL Server Agent in the DLH.io portal (or ask your account manager to do it). You will need the following values for the dlh_ref section of the configuration:

  • Organization GUID (org_guid)
  • Project GUID (prj_guid)
  • Connection GUID of the target connection the agent extracts for (connection_guid)
  • Agent API key (api_key)
  • A short unique prefix for this agent, for example Site003 (target_schema_prefix)

Do not have an account yet? Contact us to start a trial. Without an API key the agent still runs in Community mode (local output only, 10,000 rows per table) so you can evaluate it on a workstation.

Installation

Download and run the installer

The installer is provided by your DLH.io account manager or from your DLH.io portal. It is named dlh-agent-for-sql-server-installer-v<YYYYMMDD>.exe.

Run it as an administrator and accept the defaults. The installer:

  • Installs to C:\Program Files\DLH.io Agent for SQL Server\
  • Places dlh_agent_sql_server.exe, the _internal\ runtime folder, dlh_agent_config.template.yaml and a starter dlh_sql_server_credentials.yaml
  • Creates a local service account (svc_dlh_agent_runner by default) with a generated password stored in Windows Credential Manager
  • Registers a Windows Task Scheduler task named DLH.io SQL Server Agent X that runs the agent daily at 22:01 UTC (you will change the schedule later)

Upgrades reuse the same folder, replace the EXE and _internal\ folder, and never overwrite dlh_agent_config.yaml, your credentials, the encryption key or the state database.

SmartScreen and antivirus

If Windows SmartScreen or endpoint protection prompts on first run, choose More info and Run anyway, or have your security team allow the installer hash. Add the install directory to real-time scanning exclusions to avoid file locks on the state database during a run.

Install the ODBC driver

Open an elevated terminal (Command Prompt or PowerShell) in the install directory and run:

cd "C:\Program Files\DLH.io Agent for SQL Server"
.\dlh_agent_sql_server.exe --install-driver --auto

--install-driver prints guided instructions; adding --auto downloads and installs Microsoft ODBC Driver 18 for SQL Server silently. Skip this step if Driver 17 or 18 is already installed.

Create the configuration file

Copy the template and open it in a text editor:

copy dlh_agent_config.template.yaml dlh_agent_config.yaml
notepad dlh_agent_config.yaml

For a first run keep the file small. The example below syncs every table in the dbo schema of one database to Delta Lake on Azure with credentials issued by the DLH.io platform:

agent_version: 1.9.4
state_db_path: dlh_agent_state.duckdb

connection_information:
  connection_type: sql_server
  server_name_or_ip: 'sql01.corp.local'
  server_port: 1433
  database_names: [SalesDb]

data_retrieval:
  historical_load: false
  output_format: delta
  direct_cloud_write: true
  storage_base_path: /data/delta_iceberg_tables
  output_path: ./output
  databases:
    - name: SalesDb
      sync_mode: auto
      tables:
        - name: dbo.*
          all_columns: true
          sync_mode: auto
          no_pk_strategy: full

dlh_ref:
  org_guid: '<YOUR_ORG_GUID>'
  prj_guid: '<YOUR_PROJECT_GUID>'
  connection_guid: '<YOUR_CONNECTION_GUID>'
  target_schema_prefix: 'Site003'
  api_key: '<YOUR_DLH_API_KEY>'

dlh_notifications:
  enabled: true
  platform_run_report: true
  webhooks:
    - url_env_var: DLH_OPS_WEBHOOK_URL
      format: slack

Notes:

  • Use forward slashes in every path, including on Windows (C:/dlh/output).
  • dlh_storage and dlh_log_store are omitted on purpose: with an API key the platform issues the storage credentials at run time. Add them only if you want to use your own static credentials (see Storage and bootstrap).
  • The plaintext API key is encrypted in place on the first run.

Enter the SQL Server credentials

Open dlh_sql_server_credentials.yaml and set the login you created earlier:

credentials:
  encryption: no
  credentials_file: dlh_sql_server_credentials.sec
  username: dlh_agent
  password: '<strong password>'

On the first run the agent generates an encryption key (dlh_agent_secret.key), writes the encrypted password to dlh_sql_server_credentials.sec, replaces the plaintext password in the YAML with <encrypted> and sets encryption: yes. The plaintext password never needs to be on disk again. See Security for key management options including the DLH_AGENT_SECRET_KEY environment variable.

Check the host before extracting anything

.\dlh_agent_sql_server.exe --config-check
.\dlh_agent_sql_server.exe --diagnose

--config-check compares your file with the template and lists missing or unknown keys. --diagnose is a read-only health check: it verifies CPU, memory and disk, the ODBC driver, the configuration and credential files, the DLH.io license, SQL Server connectivity, Change Tracking / CDC status and storage access, and prints PASS, WARN or FAIL per check. It never extracts data or moves sync pointers. Fix every FAIL before continuing; WARN lines are informational.

Run the agent once from a terminal

Always run the first sync from a terminal rather than by double-clicking the EXE so that you can see the output:

.\dlh_agent_sql_server.exe

What a healthy first run looks like:

  1. The DLH.io banner prints, followed by a tier status line such as DLH credentials OK [CUSTOMER] or TRIAL: 29 day(s) remaining - expires 2026-10-08.
  2. The DLH.io AGENT STATE STORE (DuckDB) block reports Status: CREATED on this run (no prior state) because there is no state yet.
  3. Preflight checks all report PASS.
  4. Because there is no prior state, the run is treated as a historical (full) load of every table, regardless of sync_mode.
  5. The run summary lists every table with a row count and OK, and the ERRORS block is absent.
  6. Data appears in your storage account under <storage_base_path>/delta/<Database>.<schema>.<Table>/ (Delta) or <storage_base_path>/iceberg/... (Iceberg). CSV and Parquet output uses a YYYY/MM/DD/<epoch>/ folder per run.

Run the agent a second time. Tables with Change Tracking enabled now report ct in the mode column and only changed rows are written; tables with no changes print No CT/CDC changes - skipped write.

Schedule it

The installer registered a daily task. Most customers change it to run every 5 to 15 minutes for Change Tracking sources. Open Task Scheduler, edit DLH.io SQL Server Agent X, and on the Triggers tab set Repeat task every: 5 minutes for a duration of Indefinitely. Keep Do not start a new instance on the Settings tab so that overlapping runs never happen. See Operations for RMM deployment and monitoring.

Next steps