yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-10-11 17:08:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(ingest): add docs on pydantic compatibility (#11423)

Browse Source
This commit is contained in:
Harshal Sheth 2024-09-20 13:22:15 -07:00 committed by GitHub
parent ce0e9e2fb9
commit 1bea570b8d
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 16 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
docs/cli.md
View File

@ -34,9 +34,9 @@ datahub init
# authenticate your datahub CLI with your datahub instance # authenticate your datahub CLI with your datahub instance
``` ```
If you run into an error, try checking the [_common setup issues_](../metadata-ingestion/developing.md#Common-setup-issues). If you run into an error, try checking the [_common setup issues_](../metadata-ingestion/developing.md#common-setup-issues).
Other installation options such as installation from source and running the cli inside a container are available further below in the guide [here](#alternate-installation-options) Other installation options such as installation from source and running the cli inside a container are available further below in the guide [here](#alternate-installation-options).
## Starter Commands ## Starter Commands
@ -672,7 +672,6 @@ Old Entities Migrated = {'urn:li:dataset:(urn:li:dataPlatform:hive,logging_event
### Using docker ### Using docker
[![Docker Hub](https://img.shields.io/docker/pulls/acryldata/datahub-ingestion?style=plastic)](https://hub.docker.com/r/acryldata/datahub-ingestion) [![Docker Hub](https://img.shields.io/docker/pulls/acryldata/datahub-ingestion?style=plastic)](https://hub.docker.com/r/acryldata/datahub-ingestion)
[![datahub-ingestion docker](https://github.com/acryldata/datahub/workflows/datahub-ingestion%20docker/badge.svg)](https://github.com/acryldata/datahub/actions/workflows/docker-ingestion.yml)
If you don't want to install locally, you can alternatively run metadata ingestion within a Docker container. If you don't want to install locally, you can alternatively run metadata ingestion within a Docker container.
We have prebuilt images available on [Docker hub](https://hub.docker.com/r/acryldata/datahub-ingestion). All plugins will be installed and enabled automatically. We have prebuilt images available on [Docker hub](https://hub.docker.com/r/acryldata/datahub-ingestion). All plugins will be installed and enabled automatically.

16
metadata-ingestion/developing.md
View File

@ -55,7 +55,6 @@ logger.debug("this is the sample debug line")
#3. click on the `log` option #3. click on the `log` option
``` ```
> **P.S. if you are not able to see the log lines, then restart the `airflow scheduler` and rerun the DAG** > **P.S. if you are not able to see the log lines, then restart the `airflow scheduler` and rerun the DAG**
### (Optional) Set up your Python environment for developing on Dagster Plugin ### (Optional) Set up your Python environment for developing on Dagster Plugin
@ -70,6 +69,7 @@ datahub version # should print "DataHub CLI version: unavailable (installed in
``` ```
### (Optional) Set up your Python environment for developing on Prefect Plugin ### (Optional) Set up your Python environment for developing on Prefect Plugin
From the repository root: From the repository root:
```shell ```shell
@ -127,6 +127,18 @@ This sometimes happens if there's a version mismatch between the Kafka's C libra
</details> </details>
<details>
<summary>Conflict: acryl-datahub requires pydantic 1.10</summary>
The base `acryl-datahub` package supports both Pydantic 1.x and 2.x. However, some of our specific sources require Pydantic 1.x because of transitive dependencies.
If you're primarily using `acryl-datahub` for the SDKs, you can install `acryl-datahub` and some extras, like `acryl-datahub[sql-parser]`, without getting conflicts related to Pydantic versioning.
We recommend not installing full ingestion sources into your main environment (e.g. avoid having a dependency on `acryl-datahub[snowflake]` or other ingestion sources).
Instead, we recommend using UI-based ingestion or isolating the ingestion pipelines using [virtual environments](https://docs.python.org/3/library/venv.html). If you're using an orchestrator, they often have first-class support for virtual environments - here's an [example for Airflow](./schedule_docs/airflow.md).
</details>
### Using Plugins in Development ### Using Plugins in Development
The syntax for installing plugins is slightly different in development. For example: The syntax for installing plugins is slightly different in development. For example:
@ -286,4 +298,4 @@ tox -- --update-golden-files
# Update golden files for a specific environment. # Update golden files for a specific environment.
tox -e py310-airflow26 -- --update-golden-files tox -e py310-airflow26 -- --update-golden-files
``` ```