Skip to main content

DynamoDB

Overview​

Dynamodb is a data platform used to store and query analytical or operational data. Learn more in the official Dynamodb documentation.

The DataHub integration for Dynamodb covers core metadata entities such as datasets/tables/views, schema fields, and containers. It also captures tags and stateful deletion detection.

Concept Mapping​

Source ConceptDataHub ConceptNotes
"dynamodb"Data Platform
DynamoDB TableDataset

Module dynamodb​

GA

Important Capabilities​

CapabilityStatusNotes
Column-level Lineage✅Optionally enabled via include_s3_export_lineage with include_s3_export_column_lineage.
Detect Deleted Entities✅Enabled by default via stateful ingestion.
Extract Tags✅Optionally enabled via extract_table_tags to extract dynamoDB table tags as DataHub tags.
Platform Instance✅By default, platform_instance will use the AWS account id.
Table-Level Lineage✅Optionally enabled via include_s3_export_lineage for DynamoDB Export to S3 destinations.

Overview​

The dynamodb module ingests metadata from Dynamodb into DataHub. It is intended for production ingestion workflows and module-specific capabilities are documented below.

Prerequisites​

Breaking Change

Starting v0.13.3, aws_region is required. The connector no longer loops through all AWS regions and only uses the region specified in your recipe.

Attach the AmazonDynamoDBReadOnlyAccess policy to a user in your AWS account, then create an API access key and secret. The following privileges are required:

dynamodb:ListTables
dynamodb:DescribeTable
dynamodb:Scan
dynamodb:ListTagsOfResource

dynamodb:Scan is required because DynamoDB does not return schema information in dynamodb:DescribeTable. The connector samples a few values to infer the schema.

dynamodb:ListTagsOfResource is required only when extract_table_tags is enabled to extract DynamoDB table tags.

When include_s3_export_lineage is enabled, also grant:

dynamodb:ListExports
dynamodb:DescribeExport

These permissions only read existing Export to S3 job metadata. The connector does not start or manage exports.

Install the Plugin​

pip install 'acryl-datahub[dynamodb]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: dynamodb
  config:
    aws_access_key_id: "${AWS_ACCESS_KEY_ID}"
    aws_secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
    aws_region: "${AWS_REGION}"

    # Number of items to sample from each table for schema inference (default: 100)
    # schema_sampling_size: 100

    # Discover existing DynamoDB Export to S3 jobs and emit COPY lineage to S3.
    # Does not start exports — only reads ListExports/DescribeExport metadata.
    # include_s3_export_lineage: true

    # If there are items that have most representative fields of the table, users could use the
    # `include_table_item` option to provide a list of primary keys of the table in dynamodb format.
    # For each `region.table`, the list of primary keys can be at most 100.
    # We include these items in addition to the first 100 items in the table when we scan it.
    #
    # include_table_item:
    #   region.table_name:
    #     [
    #       {
    #         "partition_key_name": { "attribute_type": "attribute_value" },
    #         "sort_key_name": { "attribute_type": "attribute_value" },
    #       },
    #     ]

sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
aws_access_key_id
One of string, null
AWS access key ID. Can be auto-detected, see the AWS boto3 docs for details.
Default: None
aws_advanced_config
object
Advanced AWS configuration options. These are passed directly to botocore.config.Config.
aws_endpoint_url
One of string, null
The AWS service endpoint. This is normally constructed automatically, but can be overridden here.
Default: None
aws_profile
One of string, null
The named profile to use from AWS credentials. Falls back to default profile if not specified and no access keys provided. Profiles are configured in ~/.aws/credentials or ~/.aws/config.
Default: None
aws_proxy
One of string, null
A set of proxy configs to use with AWS. See the botocore.config docs for details.
Default: None
aws_region
One of string, null
AWS region code.
Default: None
aws_retry_mode
Enum
One of: "legacy", "standard", "adaptive"
Default: standard
aws_retry_num
integer
Number of times to retry failed AWS requests. See the botocore.retry docs for details.
Default: 5
aws_secret_access_key
One of string(password), null
AWS secret access key. Can be auto-detected, see the AWS boto3 docs for details.
Default: None
aws_session_token
One of string(password), null
AWS session token. Can be auto-detected, see the AWS boto3 docs for details.
Default: None
extract_table_tags
boolean
When enabled, tags associated with DynamoDB tables in AWS will be emitted as DataHub tags. Tags are applied using OVERWRITE mode, meaning any existing tags including tags manually added or modified from the UI will be replaced. Use with caution.
Default: False
include_s3_export_column_lineage
boolean
When include_s3_export_lineage is enabled, also emit column-level COPY lineage inferred from the DynamoDB table's schema. With a DataHub connection and the S3 export dataset already in DataHub, only fields that match its schema (case-insensitive) are linked. Without a connection, or when the dataset has no schema in DataHub, the DynamoDB field paths are used as-is. If the schema cannot be read, only table-level lineage is emitted. Off by default: native Export to S3 writes DynamoDB JSON/Ion, whose field paths often differ from the connector's flattened schema.
Default: False
include_s3_export_lineage
boolean
When enabled, discovers existing DynamoDB Export to S3 jobs via ListExports/DescribeExport and emits COPY lineage from each DynamoDB table to its S3 destination. Does not start or manage exports — only reads export metadata, which AWS keeps for about 90 days. Requires dynamodb:ListExports and dynamodb:DescribeExport.
Default: False
max_exports_per_table
integer
When include_s3_export_lineage is enabled, the maximum number of completed exports to describe per table. A table exported frequently accumulates many export ARNs over the ~90-day retention window that nearly all resolve to the same destination; this bounds the DescribeExport calls. Exports are described in the order ListExports returns them; AWS does not document that ordering, so which exports are kept when a table exceeds this limit is not guaranteed.
Default: 50
max_schema_size
integer
Maximum number of fields to include in the schema.
Default: 300
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
read_timeout
number
The timeout for reading from the connection (in seconds).
Default: 60
s3_export_platform_instance
One of string, null
When include_s3_export_lineage is enabled, the platform_instance to stamp on the S3 export dataset URNs. Set this to match the platform_instance used by the S3 source that ingests the export bucket, otherwise the lineage edges point at an S3 URN that does not exist.
Default: None
schema_sampling_size
integer
Number of items to sample from each table for schema inference. This controls how many items are scanned to determine the table's schema structure.
Default: 100
env
string
The environment that all assets produced by this connector belong to
Default: PROD
aws_role
One of string, array, null
AWS roles to assume. If using the string format, the role ARN can be specified directly. If using the object format, the role can be specified in the RoleArn field and additional available arguments are the same as boto3's STS.Client.assume_role.
Default: None
aws_role.union
One of string, AwsAssumeRoleConfig
aws_role.union.RoleArn ❓
string
ARN of the role to assume.
aws_role.union.ExternalId
One of string, null
External ID to use when assuming the role.
Default: None
database_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
database_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
domain
map(str,AllowDenyPattern)
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
domain.key.allow
array
List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.
Default: ['.*']
domain.key.allow.string
string
domain.key.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
domain.key.deny
array
List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.
Default: []
domain.key.deny.string
string
include_table_item
One of array, null
[Advanced] The primary keys of items of a table in dynamodb format the user would like to include in schema. Refer "Advanced Configurations" section for more details
Default: None
include_table_item.key.object
object
table_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
table_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
classification
ClassificationConfig
classification.enabled
boolean
Whether classification should be used to auto-detect glossary terms
Default: False
classification.info_type_to_term
map(str,string)
classification.max_workers
integer
Number of worker processes to use for classification. Set to 1 to disable.
Default: 4
classification.sample_size
integer
Number of sample values used for classification.
Default: 100
classification.classifiers
array
Classifiers to use to auto-detect glossary terms. If more than one classifier, infotype predictions from the classifier defined later in sequence take precedance.
Default: [{'type': 'datahub', 'config': None}]
classification.classifiers.DynamicTypedClassifierConfig
DynamicTypedClassifierConfig
classification.classifiers.DynamicTypedClassifierConfig.type ❓
string
The type of the classifier to use. The built-in datahub classifier has been removed; register a custom classifier and reference its type here.
classification.classifiers.DynamicTypedClassifierConfig.config
One of object, null
The configuration required for initializing the classifier. If not specified, uses defaults for classifer type.
Default: None
classification.column_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
classification.column_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
classification.table_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
classification.table_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Capabilities​

Use the Important Capabilities table above as the source of truth for supported features and whether additional configuration is required.

Using schema_sampling_size config​

By default, the connector samples 100 items from each table to infer the schema. You can adjust this using the schema_sampling_size configuration option if you need more comprehensive schema coverage:

# Sample 500 items instead of default 100
schema_sampling_size: 500

Using include_table_item config​

If there are items that have most representative fields of the table, users could use the include_table_item option to provide a list of primary keys of the table in dynamodb format. We include these items in addition to the items sampled based on schema_sampling_size (default 100) when we scan the table.

Take AWS DynamoDB Developer Guide Example tables and data as an example, if a account has a table Reply in the us-west-2 region with composite primary key Id and ReplyDateTime, users can use include_table_item to include 2 items as following:

Example:

# The table name should be in the format of region.table_name
# The primary keys should be in the DynamoDB format
include_table_item:
  us-west-2.Reply:
    [
      {
        "ReplyDateTime": { "S": "2015-09-22T19:58:22.947Z" },
        "Id": { "S": "Amazon DynamoDB#DynamoDB Thread 1" },
      },
      {
        "ReplyDateTime": { "S": "2015-10-05T19:58:22.947Z" },
        "Id": { "S": "Amazon DynamoDB#DynamoDB Thread 2" },
      },
    ]

DynamoDB Export to S3 lineage​

Set include_s3_export_lineage: true to discover existing DynamoDB Export to S3 jobs and emit COPY lineage from each DynamoDB table to its S3 destination (s3://bucket/prefix).

By default this emits table-level COPY edges only. Set include_s3_export_column_lineage: true to also emit column-level lineage inferred from the DynamoDB table's schema. When the recipe has a DataHub connection and the S3 export dataset already exists in DataHub, only fields that match its schema (case-insensitive) are linked. Without a connection, or when the dataset is not in DataHub, the inferred DynamoDB field paths are used as-is. If the schema cannot be read, only table-level lineage is emitted. Leave it off unless you know the S3 export layout matches those field paths — native Export to S3 writes DynamoDB JSON or Amazon Ion, which often does not. For a robust column-level story, ingest the downstream Glue/Iceberg tables (via the Glue source), which resolves column lineage against real schemas.

The connector only calls ListExports and DescribeExport. It does not create exports. AWS retains export task metadata for about 90 days, so exports older than that stop being returned. max_exports_per_table (default 50) bounds how many completed exports are described per table, since a frequently-exported table accumulates many export records that nearly all resolve to the same destination. AWS does not document the order ListExports returns, so when a table exceeds this limit, which exports are kept is not guaranteed; the connector reports truncation, as a warning when the described exports already reach more than one destination.

include_s3_export_lineage: true
# include_s3_export_column_lineage: true
# max_exports_per_table: 50
# s3_export_platform_instance: my-s3-instance

Lineage is written as a patch, so upstreams that other sources write to the same S3 dataset are kept. The connector only adds edges and never removes them: an export destination that ages out, column edges written before include_s3_export_column_lineage was turned off, and inferred column edges from a run without a DataHub connection all stay until you remove them.

Matching the S3 URNs. Two things must line up for these edges to join the S3 datasets you ingest separately:

  • Destination path. The emitted S3 URN is the export's configured destination (s3://bucket/prefix). DynamoDB actually writes the data one level deeper, under s3://bucket/prefix/AWSDynamoDB/{exportId}/data/…. If your S3 source ingests with a path_spec pointed at that deeper data path, point it at the s3://bucket/prefix level (or add a matching path_spec) so the URNs join.
  • Platform instance. The S3 URN carries no platform_instance by default. If your S3 source ingests the export bucket with a platform_instance, set s3_export_platform_instance to the same value, otherwise these edges point at an S3 URN that does not exist.

When the recipe has a DataHub connection, the ingestion report lists export destinations that no other source has ingested (the dataset has no datasetProperties) under s3_export_destinations_not_in_datahub. Those edges land on an S3 entity that only carries lineage, which usually means one of the two settings above does not match your S3 source. The same list appears as an info entry in the ingestion run's structured logs. It also includes destinations the S3 source has not ingested yet, for example when the S3 recipe runs after this one. A soft-deleted S3 dataset still has datasetProperties, so it is not listed.

Exports started by Glue jobs. The Glue DynamoDB export connector (dynamodb.export: "ddb") requests a regular DynamoDB export on every job run, so those exports appear in ListExports and get COPY edges like any other. If the job sets neither dynamodb.s3.bucket nor dynamodb.s3.prefix, the export goes to the job's temporary directory (--TempDir), so the lineage points at that temporary path, and a scheduled job's exports also count against max_exports_per_table. Set dynamodb.s3.bucket and dynamodb.s3.prefix on those jobs to give the exports a stable destination.

Limitations​

Module behavior is constrained by source APIs, permissions, and metadata exposed by the platform. Refer to capability notes for unsupported or conditional features.

S3 export lineage is limited to DynamoDB's native Export to S3 feature (DynamoDB JSON or Amazon Ion — not Parquet or Iceberg). Glue/Spark jobs that convert exports to Parquet or Iceberg are discovered by the Glue source when job scripts use connection_type: dynamodb.

Troubleshooting​

If ingestion fails, validate credentials, permissions, connectivity, and scope filters first. Then review ingestion logs for source-specific errors and adjust configuration accordingly.

If S3 export lineage is missing, confirm include_s3_export_lineage is enabled, the IAM principal can call dynamodb:ListExports and dynamodb:DescribeExport, and a COMPLETED export exists for the table within the last 90 days.

Code Coordinates​

  • Class Name: datahub.ingestion.source.dynamodb.dynamodb.DynamoDBSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for DynamoDB, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.