yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-04 07:34:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/metadata-ingestion/source-docs-template.md

152 lines
6.8 KiB
Markdown
Raw Permalink Normal View History

ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
# Source Name
docs: Ingestion Source Docs Template (#4275) * testing img.shield for status * update to hyperlink * changing link format * adding status options * updating prerequisities and quickstart * update to ingestion docs * updating template with collapse details * adding linebreak between pip install commands * Removed incomplete sentence * typo fix * pushing current changes * testing logos in markdown table * markdown table fix * markdown table fix * adding in additional logos * transposing markdown table * settling on final table format * adding commented-out source template to sidebar.js * moving reference sidebar and adding trailing comma * fixing docs build
2022-03-30 17:36:39 -05:00
<!-- Set Support Status -->
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
docs: Ingestion Source Docs Template (#4275) * testing img.shield for status * update to hyperlink * changing link format * adding status options * updating prerequisities and quickstart * update to ingestion docs * updating template with collapse details * adding linebreak between pip install commands * Removed incomplete sentence * typo fix * pushing current changes * testing logos in markdown table * markdown table fix * markdown table fix * adding in additional logos * transposing markdown table * settling on final table format * adding commented-out source template to sidebar.js * moving reference sidebar and adding trailing comma * fixing docs build
2022-03-30 17:36:39 -05:00
![Certified](https://img.shields.io/badge/support%20status-certified-brightgreen)
![Incubating](https://img.shields.io/badge/support%20status-incubating-blue)
![Testing](https://img.shields.io/badge/support%20status-testing-lightgrey)
## Integration Details
<!-- Plain-language description of what this integration is meant to do. -->
<!-- Include details about where metadata is extracted from (ie. logs, source API, manifest, etc.) -->
### Concept Mapping
<!-- This should be a manual mapping of concepts from the source to the DataHub Metadata Model -->
<!-- Authors should provide as much context as possible about how this mapping was generated, including assumptions made, known shortcuts, & any other caveats -->
This ingestion source maps the following Source System Concepts to DataHub Concepts:
<!-- Remove all unnecessary/irrevant DataHub Concepts -->
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
| Source Concept | DataHub Concept | Notes |
| -------------- | ---------------------------------------------------------------------- | ----- |
| | [Data Platform](docs/generated/metamodel/entities/dataPlatform.md) | |
| | [Dataset](docs/generated/metamodel/entities/dataset.md) | |
| | [Data Job](docs/generated/metamodel/entities/dataJob.md) | |
| | [Data Flow](docs/generated/metamodel/entities/dataFlow.md) | |
| | [Chart](docs/generated/metamodel/entities/chart.md) | |
| | [Dashboard](docs/generated/metamodel/entities/dashboard.md) | |
| | [User (a.k.a CorpUser)](docs/generated/metamodel/entities/corpuser.md) | |
| | CorpGroup | |
| | Domain | |
| | Container | |
| | Tag | |
| | GlossaryTerm | |
| | GlossaryNode | |
| | Assertion | |
| | DataProcess | |
| | MlFeature | |
| | MlFeatureTable | |
| | MlModel | |
| | MlModelDeployment | |
| | MlPrimaryKey | |
| | SchemaField | |
| | DataHubPolicy | |
| | DataHubIngestionSource | |
| | DataHubSecret | |
| | DataHubExecutionRequest | |
| | DataHubREtention | |
docs: Ingestion Source Docs Template (#4275) * testing img.shield for status * update to hyperlink * changing link format * adding status options * updating prerequisities and quickstart * update to ingestion docs * updating template with collapse details * adding linebreak between pip install commands * Removed incomplete sentence * typo fix * pushing current changes * testing logos in markdown table * markdown table fix * markdown table fix * adding in additional logos * transposing markdown table * settling on final table format * adding commented-out source template to sidebar.js * moving reference sidebar and adding trailing comma * fixing docs build
2022-03-30 17:36:39 -05:00
### Supported Capabilities
<!-- This should be an auto-generated table of supported DataHub features/functionality -->
<!-- Each capability should link out to a feature guide -->
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
| Capability | Status | Notes |
| ----------------------- | :----: | ------------------------------------------- |
| Data Container | ✅ | Enabled by default |
| Detect Deleted Entities | ✅ | Requires recipe configuration |
| Data Domain | ❌ | Requires transformer |
| Dataset Profiling | ✅ | Requires `acryl-datahub[source-usage-name]` |
| Dataset Usage | ✅ | Requires `acryl-datahub[source-usage-name]` |
| Extract Descriptions | ✅ | Enabled by default |
| Extract Lineage | ✅ | Enabled by default |
| Extract Ownership | ✅ | Enabled by default |
| Extract Tags | ❌ | Requires transformer |
| Partition Support | ❌ | Not applicable to source |
| Platform Instance | ❌ | Not applicable to source |
| ... | |
docs: Ingestion Source Docs Template (#4275) * testing img.shield for status * update to hyperlink * changing link format * adding status options * updating prerequisities and quickstart * update to ingestion docs * updating template with collapse details * adding linebreak between pip install commands * Removed incomplete sentence * typo fix * pushing current changes * testing logos in markdown table * markdown table fix * markdown table fix * adding in additional logos * transposing markdown table * settling on final table format * adding commented-out source template to sidebar.js * moving reference sidebar and adding trailing comma * fixing docs build
2022-03-30 17:36:39 -05:00
## Metadata Ingestion Quickstart
### Prerequisites
In order to ingest metadata from [Source Name], you will need:
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
- eg. Python version, source version, source access requirements
- eg. Steps to configure source access
- ...
docs: Ingestion Source Docs Template (#4275) * testing img.shield for status * update to hyperlink * changing link format * adding status options * updating prerequisities and quickstart * update to ingestion docs * updating template with collapse details * adding linebreak between pip install commands * Removed incomplete sentence * typo fix * pushing current changes * testing logos in markdown table * markdown table fix * markdown table fix * adding in additional logos * transposing markdown table * settling on final table format * adding commented-out source template to sidebar.js * moving reference sidebar and adding trailing comma * fixing docs build
2022-03-30 17:36:39 -05:00
### Install the Plugin(s)
Run the following commands to install the relevant plugin(s):
`pip install 'acryl-datahub[source-name]'`
`pip install 'acryl-datahub[source-usage-name]'`
### Configure the Ingestion Recipe(s)
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
Use the following recipe(s) to get started with ingestion.
docs: Ingestion Source Docs Template (#4275) * testing img.shield for status * update to hyperlink * changing link format * adding status options * updating prerequisities and quickstart * update to ingestion docs * updating template with collapse details * adding linebreak between pip install commands * Removed incomplete sentence * typo fix * pushing current changes * testing logos in markdown table * markdown table fix * markdown table fix * adding in additional logos * transposing markdown table * settling on final table format * adding commented-out source template to sidebar.js * moving reference sidebar and adding trailing comma * fixing docs build
2022-03-30 17:36:39 -05:00
_For general pointers on writing and running a recipe, see our [main recipe guide](../README.md#recipes)._
#### `'acryl-datahub[source-name]'`
```yml
source:
type: source_name
config:
# Required fields
option1: value1
sink:
# sink configs
```
<details>
<summary>View All Recipe Configuartion Options</summary>
| Field | Required | Default | Description |
| --- | :-: | :-: | --- |
| `field1` | ✅ | `default_value` | A required field with a default value |
| `field2` | ❌ | `default_value` | An optional field with a default value |
| `field3` | ❌ | | An optional field without a default value |
| ... | | |
</details>
#### `'acryl-datahub[source-usage-name]'`
```yml
source:
type: source-usage-name
config:
# Required Fields
option1: value1
# Options
top_n_queries: 10
sink:
# sink configs
```
<details>
<summary>View All Recipe Configuartion Options</summary>
| Field | Required | Default | Description |
| --- | :-: | :-: | --- |
| `field1` | ✅ | `default_value` | A required field with a default value |
| `field2` | ❌ | `default_value` | An optional field with a default value |
| `field3` | ❌ | | An optional field without a default value |
| ... | | |
</details>
## Troubleshooting
### [Common Issue]
[Provide description of common issues with this integration and steps to resolve]
Reference in New Issue Copy Permalink