Kafka Emitter

The Kafka emitter can be used to push metadata to DataHub.

DatahubKafkaEmitter

class datahub.emitter.kafka_emitter.DatahubKafkaEmitter(config)

Bases: Closeable, Emitter

• Parameters:config (KafkaEmitterConfig) –

close()

• Return type:None

emit(item, callback=None)

• Parameters:
  • item (Union[MetadataChangeEventClass, MetadataChangeProposalClass, MetadataChangeProposalWrapper]) –
  • callback (Optional[Callable[[Exception, str], None]])
• Return type:None

emit_mce_async(mce, callback)

• Parameters:
  • mce (MetadataChangeEventClass) –
  • callback (Callable[[Exception, str], None])
• Return type:None

emit_mcp_async(mcp, callback)

• Parameters:
  • mcp (Union[MetadataChangeProposalClass, MetadataChangeProposalWrapper]) –
  • callback (Callable[[Exception, str], None])
• Return type:None

flush()

• Return type:None

KafkaEmitterConfig

class datahub.emitter.kafka_emitter.KafkaEmitterConfig(**data)

Bases: ConfigModel

• Parameters:
  • data (Any)
  • connection (KafkaProducerConnectionConfig)
  • topic_routes (Dict [str,str])

connection : KafkaProducerConnectionConfig

model_config : ClassVar[ConfigDict] = {'_schema_extra': <function ConfigModel.Config._schema_extra>, 'extra': 'forbid', 'ignored_types': (<class 'cached_property.cached_property'>,), 'json_schema_extra': <function ConfigModel.Config._schema_extra>}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

topic_routes : Dict[str, str]

classmethod validate_topic_routes(v)

• Parameters:v (Dict[str, str])
• Return type:Dict[str, str]