Bundling Additional Connectors

To add connectors that aren't already bundled in your Remote Executor image, build a custom image that bundles the extra connector(s). The example Dockerfile below starts from an unlocked datahub-executor image and bundles the connectors you list. The image this produces is bundled but not locked, so you can keep extending it; if you need a hardened image that cannot install packages at runtime, see Optional: lock the image.

Example: Dockerfile.bundled

Save the following as Dockerfile.bundled in your build context (for example, the current directory). Do not name it Dockerfile unless you also change the -f flag in the build command below.

# syntax=docker/dockerfile:1
# check=skip=InvalidDefaultArgInFrom
# EXECUTOR_IMAGE is required via --build-arg (no useful default for customer registries).

ARG EXECUTOR_IMAGE
FROM ${EXECUTOR_IMAGE}

USER root

# Optional: omit to keep the base image's BUNDLED_CLI_VERSION (recommended).
# Only pass --build-arg BUNDLED_CLI_VERSION=... to pin a different CLI.
# Do not pass an empty build-arg — that clears the inherited value.
ARG BUNDLED_CLI_VERSION
ENV BUNDLED_CLI_VERSION=${BUNDLED_CLI_VERSION}

ARG BUNDLED_VENV_PLUGINS
ENV BUNDLED_VENV_PLUGINS=${BUNDLED_VENV_PLUGINS}

RUN /usr/local/bin/build_bundled_venvs_unified.sh

USER datahub

Build

docker build \
  --platform linux/amd64 \
  --build-arg EXECUTOR_IMAGE=<registry>/datahub-executor:<tag>-slim \
  --build-arg BUNDLED_VENV_PLUGINS=mysql,snowflake \
  -f Dockerfile.bundled .

Set --platform for a single-architecture image (the example uses linux/amd64, the common cloud default). For mixed amd64/arm64 fleets, see multi-arch under CPU architecture.

Build args:

• EXECUTOR_IMAGE — the unlocked executor image to build on (:tag or :tag-slim); not :tag-locked.
• BUNDLED_VENV_PLUGINS — comma-separated connectors to add.
• BUNDLED_CLI_VERSION (optional) — override the acryl-datahub pin. Defaults to the base image's BUNDLED_CLI_VERSION (or the installed acryl-datahub version if that ENV is unset). Omit this unless you intentionally want a different CLI than the executor image.

Each connector you list gets its own bundled environment; a connector already present in the base image is replaced, and everything else in the image is left unchanged. To bundle from a private/authenticated package index, add a BuildKit secret mount for your credentials to the bundle RUN step.

CPU architecture

Bundled venvs install architecture-specific Python wheels and native libraries. The image you run must contain venvs built for the same CPU architecture as the nodes that run the Remote Executor (or datahub-actions). Mixing arm64-built venvs onto amd64 nodes (or the reverse) causes import or binary failures.

Single-platform image

Build one architecture that matches your deployment. Pass a single --platform and an EXECUTOR_IMAGE that resolves to that arch (an arch-specific tag, or a multi-arch base that pulls the matching variant under --platform).

# amd64 cluster (common cloud default); also use this when building on Apple Silicon for amd64 nodes
docker build \
  --platform linux/amd64 \
  --build-arg EXECUTOR_IMAGE=<registry>/datahub-executor:<tag>-slim \
  --build-arg BUNDLED_VENV_PLUGINS=mysql,snowflake \
  -f Dockerfile.bundled \
  -t <registry>/datahub-executor-custom:<tag> \
  .

# arm64 cluster / arm64-only nodes
docker build \
  --platform linux/arm64 \
  --build-arg EXECUTOR_IMAGE=<registry>/datahub-executor:<tag>-slim \
  --build-arg BUNDLED_VENV_PLUGINS=mysql,snowflake \
  -f Dockerfile.bundled \
  -t <registry>/datahub-executor-custom:<tag> \
  .

Verify before deploying:

docker image inspect <registry>/datahub-executor-custom:<tag> --format '{{.Architecture}}'

Multi-arch image

If you run the executor on a mix of amd64 and arm64 nodes (or want one tag that works on both), build and push a multi-arch image with Buildx. Each platform gets its own layer set and its own bundled venvs; the registry stores a manifest list that clients pull by node architecture.

Requirements:

• A base EXECUTOR_IMAGE that itself is multi-arch (or that you can resolve per platform).
• Docker Buildx with QEMU/binfmt if you cross-build (for example building linux/amd64 on Apple Silicon).

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --build-arg EXECUTOR_IMAGE=<registry>/datahub-executor:<tag>-slim \
  --build-arg BUNDLED_VENV_PLUGINS=mysql,snowflake \
  -f Dockerfile.bundled \
  -t <registry>/datahub-executor-custom:<tag> \
  --push \
  .

--push is required for multi-platform manifests to land in the registry. Prefer a multi-arch tag when your fleet is mixed; use a single-platform build when every node shares one architecture (simpler and faster).

Optional: lock the image

Locking removes pip/uv and points the package indexes at an unreachable address, so no packages can be installed at runtime — useful for hardened or air-gapped deployments.

To lock, append these lines to the Dockerfile above (after the bundle RUN):

USER root
RUN sh /usr/local/bin/lock_image.sh
ENV UV_INDEX_URL=http://127.0.0.1:1/simple
ENV UV_DEFAULT_INDEX=http://127.0.0.1:1/simple
ENV PIP_INDEX_URL=http://127.0.0.1:1/simple
ENV PIP_EXTRA_INDEX_URL=""
ENV UV_EXTRA_INDEX_URL=""
USER datahub