mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2026-01-06 12:36:56 +00:00
Docs: Python Version Correction and Sepertor removal (#18818)
Co-authored-by: Rounak Dhillon <rounakdhillon@Rounaks-MacBook-Air.local>
This commit is contained in:
parent
7917042ae9
commit
4be0bed056
@ -13,7 +13,7 @@ OpenMetadata being a full stack project, we use the following for development:
|
||||
- [Antlr 4.9.2](https://www.antlr.org/) - `sudo make install_antlr_cli`
|
||||
- [JQ](https://jqlang.github.io/jq/) - `brew install jq` (osx) `apt-get install jq` (Ubuntu)
|
||||
- [Maven 3.5.x or higher](https://maven.apache.org/install.html) - (with Java JDK 11)
|
||||
- [Python 3.7, 3.8 or 3.9](https://www.python.org/downloads/)
|
||||
- [Python 3.8 or 3.9](https://www.python.org/downloads/)
|
||||
- [Node 18.x](https://nodejs.org/en/download/)
|
||||
- [Yarn ^1.22.0](https://classic.yarnpkg.com/lang/en/docs/install/)
|
||||
- [Rpm (Optional, only to run RPM profile with maven)](https://macappstore.org/rpm/)
|
||||
|
||||
@ -13,7 +13,7 @@ OpenMetadata being a full stack project, we use the following for development:
|
||||
- [Antlr 4.9.2](https://www.antlr.org/) - `sudo make install_antlr_cli`
|
||||
- [JQ](https://jqlang.github.io/jq/) - `brew install jq` (osx) `apt-get install jq` (Ubuntu)
|
||||
- [Maven 3.5.x or higher](https://maven.apache.org/install.html) - (with Java JDK 11)
|
||||
- [Python 3.7, 3.8 or 3.9](https://www.python.org/downloads/)
|
||||
- [Python 3.8 or 3.9](https://www.python.org/downloads/)
|
||||
- [Node 18.x](https://nodejs.org/en/download/)
|
||||
- [Yarn ^1.22.0](https://classic.yarnpkg.com/lang/en/docs/install/)
|
||||
- [Rpm (Optional, only to run RPM profile with maven)](https://macappstore.org/rpm/)
|
||||
|
||||
@ -15,14 +15,10 @@ The main takes for developing a new connector are:
|
||||
|
||||
When developing a new database ingestion connector in OpenMetadata, ensure all necessary components are correctly configured. This guide outlines the steps required to define the connector's ingestion capabilities using a `service_spec.py` file.
|
||||
|
||||
---
|
||||
|
||||
### Why Use `service_spec.py`?
|
||||
|
||||
The `service_spec.py` file centralizes the definitions of sources, profilers, lineage, and other ingestion-related components for a connector. This approach helps standardize implementations across connectors, making it easier to manage ingestion workflows.
|
||||
|
||||
---
|
||||
|
||||
### Steps to Develop a New Connector
|
||||
|
||||
#### 1. Create the `service_spec.py` File
|
||||
@ -57,17 +53,13 @@ ServiceSpec = DefaultDatabaseSpec(
|
||||
|
||||
Replace the example classes (e.g., `BigquerySource`, `BigqueryLineageSource`, etc.) with those specific to your connector. Depending on the connector's features, you may include or exclude certain components like usage or profiling.
|
||||
|
||||
---
|
||||
|
||||
### Components of `service_spec.py`
|
||||
|
||||
- **`metadata_source_class`**: Defines the class for metadata ingestion.
|
||||
- **`lineage_source_class`**: Defines the class for lineage extraction.
|
||||
- **`usage_source_class`**: Tracks data usage patterns.
|
||||
- **`profiler_class`**: Profiles data for quality and insights.
|
||||
- **`sampler_class`**: Samples data for efficient ingestion.
|
||||
|
||||
---
|
||||
- **`sampler_class`**: Samples data for efficient ingestion.
|
||||
|
||||
### Example Workflow
|
||||
|
||||
@ -80,7 +72,6 @@ Define the `ServiceSpec` using the required classes, adjusting for your connecto
|
||||
#### Step 3: Verify Integration
|
||||
Run the ingestion workflow to test the connector and ensure all components are functioning correctly.
|
||||
|
||||
|
||||
## Service Topology
|
||||
|
||||
The Topology defines a series of Nodes and Stages that get executed in a hierarchical way and describe how we extract the needed data from the sources.
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user