datahub/metadata-ingestion/source-docs-template.md

Source Name

Integration Details

Concept Mapping

This ingestion source maps the following Source System Concepts to DataHub Concepts:

Source Concept	DataHub Concept	Notes
	Data Platform
	Dataset
	Data Job
	Data Flow
	Chart
	Dashboard
	User (a.k.a CorpUser)
	CorpGroup
	Domain
	Container
	Tag
	GlossaryTerm
	GlossaryNode
	Assertion
	DataProcess
	MlFeature
	MlFeatureTable
	MlModel
	MlModelDeployment
	MlPrimaryKey
	SchemaField
	DataHubPolicy
	DataHubIngestionSource
	DataHubSecret
	DataHubExecutionRequest
	DataHubREtention

Supported Capabilities

Capability	Status	Notes
Data Container	✅	Enabled by default
Detect Deleted Entities	✅	Requires recipe configuration
Data Domain	❌	Requires transformer
Dataset Profiling	✅	Requires acryl-datahub[source-usage-name]
Dataset Usage	✅	Requires acryl-datahub[source-usage-name]
Extract Descriptions	✅	Enabled by default
Extract Lineage	✅	Enabled by default
Extract Ownership	✅	Enabled by default
Extract Tags	❌	Requires transformer
Partition Support	❌	Not applicable to source
Platform Instance	❌	Not applicable to source
...

Metadata Ingestion Quickstart

Prerequisites

In order to ingest metadata from [Source Name], you will need:

• eg. Python version, source version, source access requirements
• eg. Steps to configure source access
• ...

Install the Plugin(s)

Run the following commands to install the relevant plugin(s):

pip install 'acryl-datahub[source-name]'

pip install 'acryl-datahub[source-usage-name]'

Configure the Ingestion Recipe(s)

Use the following recipe(s) to get started with ingestion.

For general pointers on writing and running a recipe, see our main recipe guide.

'acryl-datahub[source-name]'

source:
  type: source_name
  config:
    # Required fields
    option1: value1

sink:
  # sink configs

View All Recipe Configuartion Options

Field	Required	Default	Description
field1	✅	default_value	A required field with a default value
field2	❌	default_value	An optional field with a default value
field3	❌		An optional field without a default value
...

'acryl-datahub[source-usage-name]'

source:
  type: source-usage-name
  config:
    # Required Fields
    option1: value1

    # Options
    top_n_queries: 10

sink:
  # sink configs

View All Recipe Configuartion Options

Field	Required	Default	Description
field1	✅	default_value	A required field with a default value
field2	❌	default_value	An optional field with a default value
field3	❌		An optional field without a default value
...

Troubleshooting

[Common Issue]

[Provide description of common issues with this integration and steps to resolve]