Skip to main content

Salesforce

Overview​

Salesforce is a DataHub utility or metadata-focused integration. Learn more in the official Salesforce documentation.

The DataHub integration for Salesforce covers metadata entities and operational objects relevant to this connector. Depending on module capabilities, it can also capture features such as lineage, usage, profiling, ownership, tags, and stateful deletion detection.

Concept Mapping​

Source ConceptDataHub ConceptNotes
SalesforceData Platform
Standard ObjectDatasetsubtype "Standard Object"
Custom ObjectDatasetsubtype "Custom Object"

Module salesforce​

Certified

Important Capabilities​

CapabilityStatusNotes
Data Profiling✅Only table level profiling is supported via profiling.enabled config field. Supported for types - Table.
Detect Deleted Entities✅Enabled by default via stateful ingestion.
Domains✅Supported via the domain config field.
Extract Tags✅Enabled by default.
Platform Instance✅Can be equivalent to Salesforce organization.
Schema Metadata✅Enabled by default.
Table-Level Lineage✅Extract table-level lineage for Salesforce objects. Supported for types - Custom Object, Object.

Overview​

The salesforce module ingests metadata from Salesforce into DataHub. It is intended for production ingestion workflows and module-specific capabilities are documented below.

This plugin extracts Salesforce Standard and Custom Objects and their details (fields, record count, etc) from a Salesforce instance. Python library simple-salesforce is used for authenticating and calling Salesforce REST API to retrieve details from Salesforce instance.

REST API Resources used in this integration​

Prerequisites​

Before running ingestion, ensure network connectivity to the source, valid authentication credentials, and read permissions for metadata APIs required by this module.

Authentication options​

To ingest metadata from Salesforce, you need one of:

  • Salesforce username, password, security token
  • Salesforce username, consumer key and private key for JSON web token access
  • Salesforce instance url and access token/session id (suitable for one-shot ingestion only, as access token typically expires after 2 hours of inactivity)

The account used to access Salesforce requires the following permissions for this integration to work:

  • View Setup and Configuration
  • View All Data

Install the Plugin​

pip install 'acryl-datahub[salesforce]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

pipeline_name: my_salesforce_pipeline
source:
  type: "salesforce"
  config:
    instance_url: "https://mydomain.my.salesforce.com/"
    username: user@company
    password: password_for_user
    security_token: security_token_for_user
    platform_instance: mydomain-dev-ed
    domain:
      sales:
        allow:
          - "Opportunity$"
          - "Lead$"

    object_pattern:
      allow:
        - "Account$"
        - "Opportunity$"
        - "Lead$"

sink:
  type: "datahub-rest"
  config:
    server: "http://localhost:8080"

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
access_token
One of string(password), null
Access token for instance url
Default: None
api_version
One of string, null
If specified, overrides default version used by the Salesforce package. Example value: '59.0'
Default: None
auth
Enum
One of: "USERNAME_PASSWORD", "DIRECT_ACCESS_TOKEN", "JSON_WEB_TOKEN"
consumer_key
One of string(password), null
Consumer key for Salesforce JSON web token access
Default: None
ingest_tags
boolean
Ingest Tags from source. This will override Tags entered from UI
Default: False
instance_url
One of string, null
Salesforce instance url. e.g. https://MyDomainName.my.salesforce.com
Default: None
is_sandbox
boolean
Connect to Sandbox instance of your Salesforce
Default: False
password
One of string(password), null
Password for Salesforce user
Default: None
platform
string
Default: salesforce
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
private_key
One of string(password), null
Private key as a string for Salesforce JSON web token access
Default: None
security_token
One of string(password), null
Security token for Salesforce username
Default: None
use_referenced_entities_as_upstreams
boolean
(Experimental) If enabled, referenced entities will be treated as upstream entities.
Default: False
username
One of string, null
Salesforce username
Default: None
env
string
The environment that all assets produced by this connector belong to
Default: PROD
domain
map(str,AllowDenyPattern)
A class to store allow deny regexes
domain.key.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
domain.key.allow.string
string
domain.key.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
domain.key.deny
array
List of regex patterns to exclude from ingestion.
Default: []
domain.key.deny.string
string
object_pattern
AllowDenyPattern
A class to store allow deny regexes
object_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
profile_pattern
AllowDenyPattern
A class to store allow deny regexes
profile_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
profiling
SalesforceProfilingConfig
profiling.enabled
boolean
Whether profiling should be done. Supports only table-level profiling at this stage
Default: False
profiling.operation_config
OperationConfig
profiling.operation_config.lower_freq_profile_enabled
boolean
Whether to do profiling at lower freq or not. This does not do any scheduling just adds additional checks to when not to run profiling.
Default: False
profiling.operation_config.profile_date_of_month
One of integer, null
Number between 1 to 31 for date of month (both inclusive). If not specified, defaults to Nothing and this field does not take affect.
Default: None
profiling.operation_config.profile_day_of_week
One of integer, null
Number between 0 to 6 for day of week (both inclusive). 0 is Monday and 6 is Sunday. If not specified, defaults to Nothing and this field does not take affect.
Default: None
stateful_ingestion
One of StatefulIngestionConfig, null
Stateful Ingestion Config
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False

Capabilities​

Use the Important Capabilities table above as the source of truth for supported features and whether additional configuration is required.

Limitations​

Module behavior is constrained by source APIs, permissions, and metadata exposed by the platform. Refer to capability notes for unsupported or conditional features.

  • This connector has only been tested with Salesforce Developer Edition.
  • This connector only supports table level profiling (Row and Column counts) as of now. Row counts are approximate as returned by Salesforce RecordCount REST API.
  • This integration does not support ingesting Salesforce External Objects

Troubleshooting​

If ingestion fails, validate credentials, permissions, connectivity, and scope filters first. Then review ingestion logs for source-specific errors and adjust configuration accordingly.

Code Coordinates​

  • Class Name: datahub.ingestion.source.salesforce.SalesforceSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for Salesforce, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.