yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-10-13 09:54:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(ingest): add guidelines around proactive version pinning (#7534)

Browse Source
This commit is contained in:
Harshal Sheth 2023-03-10 13:16:01 -05:00 committed by GitHub
parent a653b8ad30
commit 66f255e805
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 8 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
metadata-ingestion/developing.md
View File

@ -107,10 +107,17 @@ Some other notes:
- Prefer a standalone helper method over a `@staticmethod`. - Prefer a standalone helper method over a `@staticmethod`.
- You probably should not be defining a `__hash__` method yourself. Using `@dataclass(frozen=True)` is a good way to get a hashable class. - You probably should not be defining a `__hash__` method yourself. Using `@dataclass(frozen=True)` is a good way to get a hashable class.
- Avoid global state. In sources, this includes instance variables that effectively function as "global" state for the source. - Avoid global state. In sources, this includes instance variables that effectively function as "global" state for the source.
- Avoid pinning version dependencies. The `acryl-datahub` package is frequently used as a library and hence installed alongside other tools. If we keep our dependencies pinned or too restrictive, we break such workflows.
- Avoid defining functions within other functions. This makes it harder to read and test the code. - Avoid defining functions within other functions. This makes it harder to read and test the code.
- When interacting with external APIs, parse the responses into a dataclass rather than operating directly on the response object. - When interacting with external APIs, parse the responses into a dataclass rather than operating directly on the response object.
## Dependency Management
The vast majority of our dependencies are not required by the "core" package but instead can be optionally installed using Python "extras". This allows us to keep the core package lightweight. We should be deliberate about adding new dependencies to the core framework.
Where possible, we should avoid pinning version dependencies. The `acryl-datahub` package is frequently used as a library and hence installed alongside other tools. If you need to restrict the version of a dependency, use a range like `>=1.2.3,<2.0.0` or a negative constraint like `>=1.2.3, !=1.2.7` instead. Every upper bound and negative constraint should be accompanied by a comment explaining why it's necessary.
Caveat: Some packages like Great Expectations and Airflow frequently make breaking changes. For such packages, it's ok to add a "defensive" upper bound with the current latest version, accompanied by a comment. It's critical that we revisit these upper bounds at least once a month and broaden them if possible.
## Guidelines for Ingestion Configs ## Guidelines for Ingestion Configs
We use [pydantic](https://pydantic-docs.helpmanual.io/) to define the ingestion configs. We use [pydantic](https://pydantic-docs.helpmanual.io/) to define the ingestion configs.