DLH.io Documentation logoDLH.io Documentation
Databases

Amazon DynamoDB

DLH.io documentation for Amazon DynamoDB

Amazon DynamoDB is a fully managed, serverless NoSQL key-value and document database from Amazon Web Services (AWS). The DLH.io Amazon DynamoDB connector replicates the tables in one AWS region to your destination. Each table lands with its partition key and sort key as columns and the full item as JSON in the ITEM_DATA column. When a table has DynamoDB Streams turned on, DLH.io syncs only the changes since the previous run and flags deleted items.

The connection is read only: DLH.io never writes to, creates or changes your DynamoDB tables.

Setup Instructions

Follow the setup steps guide for configuring your Amazon DynamoDB connection to enable data to flow into your destination via DLH.

Supported Features

Here are key features supported for this connector.

Sync FeatureSupportedDetails
Custom Data and Development✓Ability to enhance connector upon request
Historical Re-Load/Load✓
Incremental/Delta Load✓On tables with DynamoDB Streams turned on. Other tables are read in full on every run
Column Selection
Column Hashing
Re-Sync Table/Entity✓Select at the table level to reload data
history (on next Sync Bridge run)
Custom Queries
Custom Data
Captures Deleted Rows✓On tables with DynamoDB Streams turned on
API Sync Bridge Initiation✓
Priority Scheduling✓
Private VPC/Link
DLH Data Model Available-

If you have any questions about these supported features please reach out to our Customer Support team.

Details on Sync Processing

First Sync

The first synchronization reads each table with a parallel scan that uses at most half of a provisioned table's read capacity (or about 2,000 read units per second on an on-demand table), so your applications keep their throughput.

Ongoing Syncs

  • Tables with a stream are synchronized incrementally from the DynamoDB stream: inserts, updates and deletes since the previous run. Deleted items stay in the destination with __DLH_IS_DELETED set, and the DELETE_SOURCE column tells a delete by a user or application (USER) from an expiry by Time to Live (TTL).
  • Tables without a stream are read in full on every run. Items deleted from such a table are not flagged in the destination; turn on a stream to capture deletes.
  • DynamoDB keeps stream records for 24 hours. If a table is not synchronized for longer than that, or its stream is turned off and on again, DLH.io reads that table in full again on the next run. Schedule the Sync Bridge to run at least every few hours. A table so large that one full read takes longer than a day keeps being read in full; contact Customer Support for such tables.

Sync Format: Packed or Unpacked

  • Packed (default): each table lands with its key attributes as columns and the whole item as JSON in ITEM_DATA. Nothing is read at schema load, and new attributes never need a change.
  • Unpacked: in addition, each top-level attribute found in a sample of the table (one read of up to 1 MB per table at schema load) becomes its own column, up to 500 per table. Values are kept as text, and maps, lists and sets as JSON, so an attribute that changes type never breaks a load; cast them in your warehouse. ITEM_DATA is kept, so an attribute first seen later is never lost: reload the schema to add its column. Attribute names longer than 63 characters, or that clash with a key or a DLH.io column, stay inside ITEM_DATA only.

After changing the Sync Format, reload the schema and run a historical re-sync so the destination tables match.

Data Types

  • Partition and sort keys land as text columns.
  • Numbers keep their full precision.
  • Binary values and binary sets are Base64 text.
  • String and number sets are sorted JSON arrays.
  • On Azure Synapse and Microsoft Fabric destinations ITEM_DATA is limited to 4,000 characters, so larger items do not fit there. Other destinations store items up to the DynamoDB maximum of 400 KB.