Configuration | DLH.io Documentation

Structure of dlh_agent_config.yaml, the DLH.io SQL Server Agent configuration file, with links to the reference page for each section.

The agent is configured by a single YAML file, dlh_agent_config.yaml, that lives next to the executable. Start from the dlh_agent_config.template.yaml shipped with every release; it documents every key inline and ends with a version-by-version changelog of configuration changes. SQL Server credentials live in a separate file, dlh_sql_server_credentials.yaml.

File layout

agent_version: 1.9.4            # Template version this file was written against
state_db_path: dlh_agent_state.duckdb

connection_information: ...     # SQL Server host, port, databases, ODBC driver
data_retrieval: ...             # What to extract, formats, write strategies, drift, retention,
                                # memory, upload strategy and the per-database table lists
dlh_ref: ...                    # Your DLH.io org / project / connection GUIDs and API key
pre_start_actions: ...          # Optional lifecycle actions before the run
post_completion_actions: ...    # Optional lifecycle actions after the run
dlh_credential__<name>: ...     # Optional named credentials (storage, scm, api)
dlh_storage: ...                # Optional static data storage credentials
dlh_log_store: ...              # Optional static log storage credentials
dlh_notifications: ...          # Email, webhooks and platform run reporting

Reference pages

Section	Page
`connection_information`, `state_db_path`, `agent_version`	Connection
`data_retrieval.databases`, tables, views, wildcards, sync modes, columns, masks, SQL queries	Data retrieval
`output_format`, Delta and Iceberg options, naming, paths, compression	Output formats
`write_strategy` (database and table level), delete modes, SCD2, no primary key fallback	Write strategies
`on_schema_drift_action`, `on_schema_drift_action_handling`, declared decimal typing	Schema drift
`dlh_ref`, tiers, `dlh_storage`, `dlh_log_store`, named credentials	Storage and bootstrap
`dlh_notifications`: email, Slack, Teams, generic webhooks, `platform_run_report`, log shipping	Notifications and reporting
`pre_start_actions`, `post_completion_actions`: Sync Bridges, CI/CD dispatches, DBDeux jobs	Lifecycle actions
`fetch_batch_size`, `max_memory_percent`, upload strategy, retention and cleanup settings	Performance and retention

Conventions

Paths use forward slashes, including on Windows: C:/dlh/output, never C:\dlh\output.
Relative paths are resolved from the agent's working directory. The installer's scheduled task starts in the install directory, so ./output and dlh_agent_state.duckdb land there. If you run the agent from another directory, or from your own task, use absolute paths for state_db_path and output_path.
Precedence for settings that exist at several levels is always the table entry first, then the database entry (sync_mode, schema_refresh_mode), then data_retrieval (write_strategy, schema drift settings). A specific table entry also overrides a wildcard entry that matches the same table.
Placeholders such as <YOUR_ORG_GUID> are detected: the agent refuses to run with an unresolved placeholder and tells you which key to fill in.
Secrets (API key, storage secrets, SCM and API tokens, the SQL Server password) are encrypted in place on the first run. Enter the plaintext value once; see Security.
Comments in your file are preserved when the agent rewrites it (for example to reset historical_load or mark a value as encrypted).

Checking your file

dlh_agent_sql_server.exe --config-check compares your configuration against the shipped template and lists keys that are missing (new since your agent_version), unknown, or still placeholders. Run it after every upgrade. See CLI reference.

Minimal working examples

Community mode on a workstation (no account)

agent_version: 1.9.4
state_db_path: dlh_agent_state.duckdb
connection_information:
  connection_type: sql_server
  server_name_or_ip: localhost
  server_port: 1433
  database_names: [AdventureWorks2019]
data_retrieval:
  historical_load: false
  output_format: parquet
  output_path: ./output
  databases:
    - name: AdventureWorks2019
      sync_mode: full
      tables:
        - name: Sales.*
          all_columns: true

Customer account, Change Tracking to Delta Lake on Azure

agent_version: 1.9.4
state_db_path: C:/dlh/state/dlh_agent_state.duckdb
connection_information:
  connection_type: sql_server
  server_name_or_ip: sql01.corp.local
  server_port: 1433
  database_names: [SalesDb, HrDb]
data_retrieval:
  historical_load: false
  output_format: delta
  direct_cloud_write: true
  storage_base_path: /data/delta_iceberg_tables
  output_path: C:/dlh/output
  parquet_compression: zstd
  on_schema_drift_action: migrate_target
  on_schema_drift_action_handling: fail_once
  databases:
    - name: SalesDb
      sync_mode: ct
      schema_refresh_mode: 1D
      tables:
        - name: dbo.*
          all_columns: true
          no_pk_strategy: full
        - name: dbo.AuditLog
          all_columns: true
          write_strategy: append
    - name: HrDb
      sync_mode: ct
      tables:
        - name: dbo.Employee
          all_columns: true
          write_strategy: scd2
          masks:
            - column: Salary
              algorithm: round
              param: 1000
dlh_ref:
  org_guid: '<YOUR_ORG_GUID>'
  prj_guid: '<YOUR_PROJECT_GUID>'
  connection_guid: '<YOUR_CONNECTION_GUID>'
  target_schema_prefix: Site003
  api_key: '<YOUR_DLH_API_KEY>'
dlh_notifications:
  enabled: true
  platform_run_report: true
  webhooks:
    - url_env_var: DLH_OPS_WEBHOOK_URL
      format: teams

Complete examples

The Scenarios section has full, copy-ready configurations with the expected first-run and steady-state behaviour for each: Change Tracking to Delta Lake on Azure read by Snowflake, CDC to Apache Iceberg on S3, SCD Type 2 history tables, several databases in one agent, Community mode, SQL queries and custom extracts, and CI/CD or DBDeux triggers after each run.

Page Contents