mirror of https://github.com/datahub-project/datahub.git synced 2025-10-24 07:24:58 +00:00

docs: Ingestion Source Docs Template (#4275 )

* testing img.shield for status

* update to hyperlink

* changing link format

* adding status options

* updating prerequisities and quickstart

* update to ingestion docs

* updating template with collapse details

* adding linebreak between pip install commands

* Removed incomplete sentence

* typo fix

* pushing current changes

* testing logos in markdown table

* markdown table fix

* markdown table fix

* adding in additional logos

* transposing markdown table

* settling on final table format

* adding commented-out source template to sidebar.js

* moving reference sidebar and adding trailing comma

* fixing docs build

2022-03-30 15:36:39 -07:00

4.5 KiB

Raw Blame History

Source Name

Integration Details

Concept Mapping

This ingestion source maps the following Source System Concepts to DataHub Concepts:

Source Concept	DataHub Concept	Notes
	Data Platform
	Dataset
	Data Job
	Data Flow
	Chart
	Dashboard
	User (a.k.a CorpUser)
	CorpGroup
	Domain
	Container
	Tag
	GlossaryTerm
	GlossaryNode
	Assertion
	DataProcess
	MlFeature
	MlFeatureTable
	MlModel
	MlModelDeployment
	MlPrimaryKey
	SchemaField
	DataHubPolicy
	DataHubIngestionSource
	DataHubSecret
	DataHubExecutionRequest
	DataHubREtention

Supported Capabilities

Capability	Status	Notes
Data Container	✅	Enabled by default
Detect Deleted Entities	✅	Requires recipe configuration
Data Domain	❌	Requires transformer
Dataset Profiling	✅	Requires `acryl-datahub[source-usage-name]`
Dataset Usage	✅	Requires `acryl-datahub[source-usage-name]`
Extract Descriptions	✅	Enabled by default
Extract Lineage	✅	Enabled by default
Extract Ownership	✅	Enabled by default
Extract Tags	❌	Requires transformer
Partition Support	❌	Not applicable to source
Platform Instance	❌	Not applicable to source
...

Metadata Ingestion Quickstart

Prerequisites

In order to ingest metadata from [Source Name], you will need:

eg. Python version, source version, source access requirements
eg. Steps to configure source access
...

Install the Plugin(s)

Run the following commands to install the relevant plugin(s):

pip install 'acryl-datahub[source-name]'

pip install 'acryl-datahub[source-usage-name]'

Configure the Ingestion Recipe(s)

Use the following recipe(s) to get started with ingestion.

For general pointers on writing and running a recipe, see our main recipe guide.

`'acryl-datahub[source-name]'`

source:
  type: source_name
  config:
    # Required fields
    option1: value1

sink:
  # sink configs

View All Recipe Configuartion Options

Field	Required	Default	Description
`field1`	✅	`default_value`	A required field with a default value
`field2`	❌	`default_value`	An optional field with a default value
`field3`	❌		An optional field without a default value
...

`'acryl-datahub[source-usage-name]'`

source:
  type: source-usage-name
  config:
    # Required Fields
    option1: value1

    # Options
    top_n_queries: 10

sink:
  # sink configs