yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-08-15 20:57:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(ingest): clarify adding source guide (#9161)

Browse Source
This commit is contained in:
Harshal Sheth 2023-11-06 12:47:07 -08:00 committed by GitHub
parent 81daae815a
commit 02156662b5
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 18 additions and 14 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

32
metadata-ingestion/adding-source.md
View File

@ -6,7 +6,7 @@ There are two ways of adding a metadata ingestion source.
2. You are writing the custom source for yourself and are not going to contribute back (yet). 2. You are writing the custom source for yourself and are not going to contribute back (yet).
If you are going for case (1) just follow the steps 1 to 9 below. In case you are building it for yourself you can skip If you are going for case (1) just follow the steps 1 to 9 below. In case you are building it for yourself you can skip
steps 4-9 (but maybe write tests and docs for yourself as well) and follow the documentation steps 4-8 (but maybe write tests and docs for yourself as well) and follow the documentation
on [how to use custom ingestion sources](../docs/how/add-custom-ingestion-source.md) on [how to use custom ingestion sources](../docs/how/add-custom-ingestion-source.md)
without forking Datahub. without forking Datahub.
@ -27,6 +27,7 @@ from `ConfigModel`. The [file source](./src/datahub/ingestion/source/file.py) is
We use [pydantic](https://pydantic-docs.helpmanual.io) conventions for documenting configuration flags. Use the `description` attribute to write rich documentation for your configuration field. We use [pydantic](https://pydantic-docs.helpmanual.io) conventions for documenting configuration flags. Use the `description` attribute to write rich documentation for your configuration field.
For example, the following code: For example, the following code:
```python ```python
from pydantic import Field from pydantic import Field
from datahub.api.configuration.common import ConfigModel from datahub.api.configuration.common import ConfigModel
@ -49,12 +50,10 @@ generates the following documentation:
<img width="70%" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/metadata-ingestion/generated_config_docs.png"/> <img width="70%" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/metadata-ingestion/generated_config_docs.png"/>
</p> </p>
:::note :::note
Inline markdown or code snippets are not yet supported for field level documentation. Inline markdown or code snippets are not yet supported for field level documentation.
::: :::
### 2. Set up the reporter ### 2. Set up the reporter
The reporter interface enables the source to report statistics, warnings, failures, and other information about the run. The reporter interface enables the source to report statistics, warnings, failures, and other information about the run.
@ -71,6 +70,8 @@ some [convenience methods](./src/datahub/emitter/mce_builder.py) for commonly us
### 4. Set up the dependencies ### 4. Set up the dependencies
Note: Steps 4-8 are only required if you intend to contribute the source back to the Datahub project.
Declare the source's pip dependencies in the `plugins` variable of the [setup script](./setup.py). Declare the source's pip dependencies in the `plugins` variable of the [setup script](./setup.py).
### 5. Enable discoverability ### 5. Enable discoverability
@ -119,37 +120,38 @@ from datahub.ingestion.api.decorators import (
@capability(SourceCapability.LINEAGE_COARSE, "Enabled by default") @capability(SourceCapability.LINEAGE_COARSE, "Enabled by default")
class FileSource(Source): class FileSource(Source):
""" """
The File Source can be used to produce all kinds of metadata from a generic metadata events file. The File Source can be used to produce all kinds of metadata from a generic metadata events file.
:::note :::note
Events in this file can be in MCE form or MCP form. Events in this file can be in MCE form or MCP form.
::: :::
""" """
... source code goes here ... source code goes here
``` ```
#### 7.2 Write custom documentation #### 7.2 Write custom documentation
- Create a copy of [`source-docs-template.md`](./source-docs-template.md) and edit all relevant components. - Create a copy of [`source-docs-template.md`](./source-docs-template.md) and edit all relevant components.
- Name the document as `<plugin.md>` and move it to `metadata-ingestion/docs/sources/<platform>/<plugin>.md`. For example for the Kafka platform, under the `kafka` plugin, move the document to `metadata-ingestion/docs/sources/kafka/kafka.md`. - Name the document as `<plugin.md>` and move it to `metadata-ingestion/docs/sources/<platform>/<plugin>.md`. For example for the Kafka platform, under the `kafka` plugin, move the document to `metadata-ingestion/docs/sources/kafka/kafka.md`.
- Add a quickstart recipe corresponding to the plugin under `metadata-ingestion/docs/sources/<platform>/<plugin>_recipe.yml`. For example, for the Kafka platform, under the `kafka` plugin, there is a quickstart recipe located at `metadata-ingestion/docs/sources/kafka/kafka_recipe.yml`. - Add a quickstart recipe corresponding to the plugin under `metadata-ingestion/docs/sources/<platform>/<plugin>_recipe.yml`. For example, for the Kafka platform, under the `kafka` plugin, there is a quickstart recipe located at `metadata-ingestion/docs/sources/kafka/kafka_recipe.yml`.
- To write platform-specific documentation (that is cross-plugin), write the documentation under `metadata-ingestion/docs/sources/<platform>/README.md`. For example, cross-plugin documentation for the BigQuery platform is located under `metadata-ingestion/docs/sources/bigquery/README.md`. - To write platform-specific documentation (that is cross-plugin), write the documentation under `metadata-ingestion/docs/sources/<platform>/README.md`. For example, cross-plugin documentation for the BigQuery platform is located under `metadata-ingestion/docs/sources/bigquery/README.md`.
#### 7.3 Viewing the Documentation #### 7.3 Viewing the Documentation
Documentation for the source can be viewed by running the documentation generator from the `docs-website` module. Documentation for the source can be viewed by running the documentation generator from the `docs-website` module.
##### Step 1: Build the Ingestion docs ##### Step 1: Build the Ingestion docs
```console ```console
# From the root of DataHub repo # From the root of DataHub repo
./gradlew :metadata-ingestion:docGen ./gradlew :metadata-ingestion:docGen
``` ```
If this finishes successfully, you will see output messages like: If this finishes successfully, you will see output messages like:
```console ```console
Ingestion Documentation Generation Complete Ingestion Documentation Generation Complete
############################################ ############################################
@ -170,7 +172,8 @@ Ingestion Documentation Generation Complete
You can also find documentation files generated at `./docs/generated/ingestion/sources` relative to the root of the DataHub repo. You should be able to locate your specific source's markdown file here and investigate it to make sure things look as expected. You can also find documentation files generated at `./docs/generated/ingestion/sources` relative to the root of the DataHub repo. You should be able to locate your specific source's markdown file here and investigate it to make sure things look as expected.
#### Step 2: Build the Entire Documentation #### Step 2: Build the Entire Documentation
To view how this documentation looks in the browser, there is one more step. Just build the entire docusaurus page from the `docs-website` module.
To view how this documentation looks in the browser, there is one more step. Just build the entire docusaurus page from the `docs-website` module.
```console ```console
# From the root of DataHub repo # From the root of DataHub repo
@ -178,6 +181,7 @@ To view how this documentation looks in the browser, there is one more step. Jus
``` ```
This will generate messages like: This will generate messages like:
```console ```console
... ...
> Task :docs-website:yarnGenerate > Task :docs-website:yarnGenerate
@ -219,15 +223,15 @@ BUILD SUCCESSFUL in 35s
36 actionable tasks: 16 executed, 20 up-to-date 36 actionable tasks: 16 executed, 20 up-to-date
``` ```
After this you need to run the following script from the `docs-website` module. After this you need to run the following script from the `docs-website` module.
```console ```console
cd docs-website cd docs-website
npm run serve npm run serve
``` ```
Now, browse to http://localhost:3000 or whichever port npm is running on, to browse the docs. Now, browse to http://localhost:3000 or whichever port npm is running on, to browse the docs.
Your source should show up on the left sidebar under `Metadata Ingestion / Sources`. Your source should show up on the left sidebar under `Metadata Ingestion / Sources`.
### 8. Add SQL Alchemy mapping (if applicable) ### 8. Add SQL Alchemy mapping (if applicable)