Storage and platform bootstrap

The dlh_ref section, platform-issued storage credentials, static S3 and Azure credentials, named credentials, upload strategies and Community mode.

The agent needs two things to deliver data: an identity on the DLH.io platform (dlh_ref) and somewhere to write (object storage). Since template version 1.9.3 the platform can issue the storage credentials for you, so most installations only need dlh_ref.

dlh_ref

dlh_ref:
  org_guid: '<YOUR_ORG_GUID>'
  prj_guid: '<YOUR_PROJECT_GUID>'
  connection_guid: '<YOUR_CONNECTION_GUID>'
  target_schema_prefix: 'Store560'
  api_key: '<YOUR_DLH_API_KEY>'
  # is_encrypted_api_key: false       # managed by the agent
  # cache_ttl_days: 7                 # bootstrap cache override, max 30
  # trial_expiry_warning_days: 7      # TRIAL accounts only

Key	Required	Description
`org_guid`	yes	Organization GUID from your DLH.io account.
`prj_guid`	yes	Project GUID.
`connection_guid`	yes	GUID of the target connection this agent extracts for. Determines which storage credentials the platform issues.
`target_schema_prefix`	yes	Unique qualifier for this agent installation, used as a folder under `<org>/<prj>/` in storage. Use something stable such as a site or store code.
`api_key`	recommended	API key from your DLH.io account. Encrypted in place on the first run. Without it the agent runs in Community mode.
`cache_ttl_days`	no	How long a successful bootstrap response may be reused when the platform is unreachable. Default 7 days, maximum 30. Short-lived storage tokens are always refreshed on their own, shorter schedule.
`trial_expiry_warning_days`	no	Days before a TRIAL account expires at which the banner shows a WARNING and one notification per run is sent. Default 7.

Where to find the GUIDs and API key: the organization, project and connection pages of the DLH.io platform show their GUIDs; API keys are created under your account settings. If you do not yet have an organization, contact us.

What happens at startup

On every run the agent calls the platform bootstrap endpoint with your API key and connection GUID. The response confirms the account tier and, for Customer and Trial accounts, includes short-lived storage credentials for the connection's bucket or container.

================================================================
  DLH.io AGENT TIER / LICENSE STATUS
================================================================
  Tier              : CUSTOMER
  Connection        : Sales Azure Raw
  Storage provider  : AZURE_BLOB
  Bucket/Container  : sales-raw
  Credential source : Live DLH bootstrap
  Row cap           : Unlimited
  Cloud uploads     : Enabled
================================================================

Situation	Behaviour
Bootstrap succeeds	Tier and storage credentials from the platform. Response cached locally (encrypted) for `cache_ttl_days`.
Platform temporarily unreachable, cache valid	Run continues with the cached tier and credentials. Banner shows `Credential source : Local cache` or `STALE cache (DLH API unreachable)`.
Platform unreachable, cache expired or missing	Run stops with an actionable error. Nothing is extracted.
API key missing, invalid, rejected or expired	Local Community mode (see below). Static storage credentials are ignored.
Trial expired	Treated like a rejected key: Community mode with an EXPIRED notice in the banner.

The cache lives under %PROGRAMDATA%\DLH\dlh-agent-sql-server and can be wiped with --clear-credentials-cache (see CLI reference).

Storage credentials

Storage credentials are resolved in this order:

Complete static credentials in dlh_storage (and dlh_log_store for logs), when present and the API key validated.
Platform-issued credentials from the bootstrap response.
If neither is available, the run continues in Community mode with cloud uploads disabled.

Omit dlh_storage and dlh_log_store entirely. The platform issues credentials scoped to the connection's bucket or container on every run; they are never written into your YAML file. Rotation and revocation happen in the DLH.io platform.

dlh_ref:
  org_guid: '...'
  prj_guid: '...'
  connection_guid: '...'
  target_schema_prefix: 'Store560'
  api_key: '...'
# no dlh_storage / dlh_log_store sections

dlh_storage:
  storage_type: s3
  access_key: '<YOUR_S3_ACCESS_KEY>'
  secret_key: '<YOUR_S3_SECRET_KEY>'
  bucket_name: '<YOUR_S3_BUCKET>'
  region: us-east-1
  is_encrypted_storage_password: false   # flipped to true after first run

secret_key is encrypted in place on the first run. Grant the IAM user or role s3:PutObject, s3:GetObject, s3:DeleteObject and s3:ListBucket on the bucket and prefix; Delta and Iceberg maintenance (vacuum, snapshot expiry) needs delete.

dlh_storage:
  storage_type: azure
  azure_account: '<YOUR_AZURE_ACCOUNT>'
  azure_sas_token: '<YOUR_AZURE_SAS_TOKEN>'
  azure_container_name: '<YOUR_AZURE_CONTAINER>'
  azure_storage_uri_scheme: az      # az, abfss, wasbs or azure
  is_encrypted_storage_password: false

Use a container-scoped SAS token with Read, Add, Create, Write, Delete and List permissions and an expiry you track; the agent reports storage failures when the token lapses. azure_storage_uri_scheme controls the URI style written into Delta metadata (abfss for Databricks and Synapse, azure for Snowflake external volumes). Iceberg metadata always uses the Snowflake compatible azure:// form.

Define the credential once and reference it from both storage sections. Section names must start with dlh_credential__.

dlh_credential__blobstorage:
  credential_type: storage
  storage_type: azure
  azure_sas_token: '<YOUR_AZURE_SAS_TOKEN>'
  azure_account: '<YOUR_AZURE_ACCOUNT>'
  azure_container_name: '<YOUR_AZURE_CONTAINER>'
  is_encrypted_storage_password: false

dlh_storage:
  storage_type: azure
  reference_credential: dlh_credential__blobstorage

dlh_log_store:
  storage_type: azure
  reference_credential: dlh_credential__blobstorage

Named credentials also hold SCM tokens (credential_type: scm) and API tokens (credential_type: api) used by lifecycle actions.

dlh_log_store follows the same rules as dlh_storage and receives the shipped run logs. It is usually the same account and container; see Notifications and reporting for what gets uploaded.

Mixing static and platform credentials

Static credentials win when they are complete, which lets you keep an existing bucket while the platform provides the tier check. To move to platform-issued credentials later, delete the dlh_storage section; the next run picks up the platform credentials without any other change. The startup banner always states which source is in use.

Upload behaviour

data_retrieval:
  direct_cloud_write: true
  cloud_upload_strategy: end_of_run
  cloud_upload_batch_size: 5
  list_cloud_objects_after_upload: false

Key	Default	Description
`direct_cloud_write`	`false`	Delta and Iceberg tables are written straight to storage. When true the upload steps below only apply to CSV/Parquet, SQL query results, audit files and logs.
`cloud_upload_strategy`	`end_of_run`	`end_of_run` uploads everything after all tables finish; `per_table` uploads each table's files as soon as it completes so a crash mid-run still leaves earlier tables delivered; `batch` uploads every `cloud_upload_batch_size` tables.
`cloud_upload_batch_size`	`5`	Tables per upload batch when the strategy is `batch`.
`list_cloud_objects_after_upload`	`false`	Log the objects under the run prefix after uploading. Useful for a first run, noisy afterwards.

Community mode

Without a valid API key (missing, invalid, rejected or an expired trial) the agent runs in local Community mode:

Extraction is capped at 10,000 rows per table or view.
Cloud uploads and direct cloud writes are disabled; static storage credentials are ignored. Output stays in output_path on the local machine.
Logs are not shipped and no run reports are sent to the platform.
Everything else works: CT/CDC, Delta and Iceberg output to local disk, schema drift handling, notifications via webhooks and email.

The startup banner prints Tier : LOCAL COMMUNITY MODE with the reason, Row cap : 10,000 rows per table/view and Cloud uploads : DISABLED, and the per-table summary flags capped tables. Server-validated Community accounts are capped the same way.

Ready for production volumes?

Customer and Trial accounts remove the row cap and enable platform-issued storage credentials, log shipping and run reporting. Contact us to start a trial or upgrade an existing organization.

dlh_ref

What happens at startup

Storage credentials

Upload behaviour

Community mode

Page Contents