CI/CD and DBDeux triggers

Run dbt transformations as soon as a sync lands: a dbdeux scheduled job on success, a second job on failure, and a GitHub Actions dispatch guarded by a schedule window and concurrency group.

Once the agent writes fresh data, something downstream usually has to run: a dbt project, a warehouse refresh, a notification to the analytics team. Lifecycle actions let the agent start that step itself, so transformations follow each extraction instead of running on a clock that is either too early (stale data) or too late (wasted latency).

When to use it

A dbt or SQL transformation project should run after every successful sync, not on its own schedule.
Different things should happen on success and on failure (build the marts, or page the on-call channel and skip the build).
Transformations live in a Git repository with a CI pipeline (GitHub Actions here; GitLab and Bitbucket work the same way) and you want the agent to dispatch it.

Design used here

dbdeux scheduled job Transform SalesDb runs on success. dbdeux is the DLH.io transformation and orchestration service; starting a job there is idempotent and cheap, so it is safe on a five-minute schedule.
dbdeux scheduled job Sync failure triage runs on error, passing the agent run id so the job can look up the run report.
GitHub Actions dispatch runs a heavier full dbt build in the analytics repository, but only on weekdays between 06:00 and 20:00 Eastern, and the workflow uses a concurrency group so overlapping dispatches collapse into one build.

Configuration

agent_version: 1.9.4
state_db_path: C:/dlh/state/dlh_agent_state.duckdb

connection_information:
  connection_type: sql_server
  server_name_or_ip: sql01.corp.local
  server_port: 1433
  database_names: [SalesDb]

data_retrieval:
  historical_load: false
  output_format: delta
  direct_cloud_write: true
  storage_base_path: /data/delta_iceberg_tables
  output_path: C:/dlh/output
  parquet_compression: zstd
  on_schema_drift_action: migrate_target
  on_schema_drift_action_handling: fail_once
  databases:
    - name: SalesDb
      sync_mode: ct
      schema_refresh_mode: 1D
      tables:
        - name: dbo.*
          all_columns: true
          no_pk_strategy: full

dlh_ref:
  org_guid: '<YOUR_ORG_GUID>'
  prj_guid: '<YOUR_PROJECT_GUID>'
  connection_guid: '<YOUR_CONNECTION_GUID>'
  target_schema_prefix: Site003
  api_key: '<YOUR_DLH_API_KEY>'

dlh_notifications:
  enabled: true
  platform_run_report: true
  error_severity: error
  webhooks:
    - url_env_var: DLH_OPS_WEBHOOK_URL
      format: teams

post_completion_actions:
  enabled: true
  dbdeux_jobs:
    - name: 'Transform SalesDb'
      job_guid: '<DBDEUX_TRANSFORM_JOB_GUID>'
      credential_ref: 'dbdeux_svc'
      run_condition: on_success
      variables:
        source_prefix: 'Site003'
    - name: 'Sync failure triage'
      job_guid: '<DBDEUX_TRIAGE_JOB_GUID>'
      credential_ref: 'dbdeux_svc'
      run_condition: on_error
  cicd_dispatches:
    - name: 'Full dbt build via GitHub Actions'
      provider: github
      owner: 'acme'
      repo: 'acme-analytics-dbt'
      event_type: 'dlh-agent-post-completed'
      branch: 'main'
      dbt_profile_target: 'prod'
      variables:
        source_db: 'SalesDb'
        dbt_selector: 'source:sql_server_site003+'
      credential_ref: 'github_dispatch'
      wait_for_response: true
      timeout_seconds: 30
      schedule_window:
        timezone: 'America/New_York'
        days: [monday, tuesday, wednesday, thursday, friday]
        start_time: '06:00'
        end_time: '20:00'

dlh_credential__dbdeux_svc:
  credential_type: api
  token: '<YOUR_DBDEUX_SERVICE_ACCOUNT_TOKEN>'
  is_encrypted_credential: false

dlh_credential__github_dispatch:
  credential_type: scm
  provider: github
  token: '<YOUR_GITHUB_FINE_GRAINED_TOKEN>'
  is_encrypted_credential: false

Block	Why
Two `dbdeux_jobs` with different `run_condition`	`on_success` fires only when every table succeeded; `on_error` fires when any table failed or the run aborted. Routing them to different jobs means a broken table never triggers a marts build on partial data.
`variables.source_prefix`	Merged into the job run request together with the values the agent always adds: `triggered_by`, `agent_run_id` and `agent_run_status`. The triage job can use `agent_run_id` to fetch the run report.
`cicd_dispatches` with `schedule_window`	The GitHub build is skipped (not deferred) when a run finishes outside 06:00 to 20:00 Eastern on a weekday. dbdeux jobs have no window here, so they still follow every run.
`wait_for_response: true`, `timeout_seconds: 30`	The agent waits for GitHub's HTTP response and logs the status, so a rejected token or a missing workflow shows up in the agent log rather than silently doing nothing.
`dlh_credential__*` sections	Tokens are encrypted in place on the first run. `credential_ref` values are the suffix after `dlh_credential__`.

Lifecycle actions never fail the run: an action that errors is logged and shown in the run summary as a non-blocking action failure, and the data that was written stays written.

GitHub workflow

.github/workflows/dlh-agent-dispatch.yml in the acme-analytics-dbt repository:

name: dbt build on DLH agent completion

on:
  repository_dispatch:
    types: [dlh-agent-post-completed]

concurrency:
  group: dlh-agent-${{ github.event.client_payload.source_db }}
  cancel-in-progress: false

jobs:
  dbt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.client_payload.branch }}
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install -r requirements.txt
      - run: dbt deps
      - run: dbt build --select "${{ github.event.client_payload.dbt_selector }}" --target "${{ github.event.client_payload.dbt_profile_target }}"
        env:
          SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}

The variables, branch and dbt_profile_target from the agent arrive in github.event.client_payload. The concurrency block is what makes a five-minute agent safe with CI: a dispatch that arrives while a build is running queues behind it, and GitHub keeps only the latest queued run per group.

The token in dlh_credential__github_dispatch needs permission to create repository_dispatch events: for a fine-grained token, Contents: Read and write on that repository; for a classic token, the repo scope.

Run behaviour

After the run summary the agent prints the post-completion phase:

[post_completion_actions] Triggering dbdeux scheduled job: Transform SalesDb (<DBDEUX_TRANSFORM_JOB_GUID>)
[post_completion_actions] dbdeux job 'Transform SalesDb' triggered successfully (412ms).
[post_completion_actions] Skipping dbdeux job 'Sync failure triage' - run_condition 'on_error' not met (run status: success).
[post_completion_actions] Dispatching CI/CD workflow: Full dbt build via GitHub Actions (github acme/acme-analytics-dbt)
[post_completion_actions] CI/CD dispatch 'Full dbt build via GitHub Actions' successful: OK

When the run finishes at 22:15 Eastern the dispatch lines are replaced by Skipping 'Full dbt build via GitHub Actions' - outside schedule window. When a table fails, the transform job is skipped and the triage job is triggered with agent_run_status=error. In dbdeux, each triggered job run shows triggered_by and agent_run_id in its variables so you can trace it back to the agent log.

Why not just schedule dbt every 15 minutes?

A fixed dbt schedule either runs on stale data or wastes warehouse credits rebuilding unchanged models. Chaining from the agent means transformations run exactly once per delivery, and not at all when a delivery failed.

Variations

GitLab or Bitbucket instead of GitHub: change provider, use workflow_ref for the branch to run on, and provide the appropriate token type. The variables arrive as pipeline variables together with DLH_RUN_ID, DLH_EVENT_TYPE, DLH_BRANCH and DLH_DBT_PROFILE_TARGET. See Lifecycle actions.
Refresh Snowflake without CI: add a sync_bridges entry with the GUID of a DLH.io sync bridge that refreshes the Iceberg or Delta tables the agent writes. Sync bridges use the dlh_ref.api_key and need no extra credential.
Pre-start actions: pre_start_actions accepts the same action types and runs after preflight checks pass, for example to pause a downstream refresh while a large historical load runs.
Always run, then decide in the job: use run_condition: always on a single dbdeux job and branch on agent_run_status inside the job instead of configuring two entries.