Docs: Python Version Correction and Sepertor removal (#18818)

Co-authored-by: Rounak Dhillon <rounakdhillon@Rounaks-MacBook-Air.local>
This commit is contained in:
RounakDhillon 2024-12-05 12:02:47 +05:30 committed by GitHub
parent 7917042ae9
commit 4be0bed056
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 3 additions and 12 deletions

View File

@ -13,7 +13,7 @@ OpenMetadata being a full stack project, we use the following for development:
- [Antlr 4.9.2](https://www.antlr.org/) - `sudo make install_antlr_cli`
- [JQ](https://jqlang.github.io/jq/) - `brew install jq` (osx) `apt-get install jq` (Ubuntu)
- [Maven 3.5.x or higher](https://maven.apache.org/install.html) - (with Java JDK 11)
- [Python 3.7, 3.8 or 3.9](https://www.python.org/downloads/)
- [Python 3.8 or 3.9](https://www.python.org/downloads/)
- [Node 18.x](https://nodejs.org/en/download/)
- [Yarn ^1.22.0](https://classic.yarnpkg.com/lang/en/docs/install/)
- [Rpm (Optional, only to run RPM profile with maven)](https://macappstore.org/rpm/)

View File

@ -13,7 +13,7 @@ OpenMetadata being a full stack project, we use the following for development:
- [Antlr 4.9.2](https://www.antlr.org/) - `sudo make install_antlr_cli`
- [JQ](https://jqlang.github.io/jq/) - `brew install jq` (osx) `apt-get install jq` (Ubuntu)
- [Maven 3.5.x or higher](https://maven.apache.org/install.html) - (with Java JDK 11)
- [Python 3.7, 3.8 or 3.9](https://www.python.org/downloads/)
- [Python 3.8 or 3.9](https://www.python.org/downloads/)
- [Node 18.x](https://nodejs.org/en/download/)
- [Yarn ^1.22.0](https://classic.yarnpkg.com/lang/en/docs/install/)
- [Rpm (Optional, only to run RPM profile with maven)](https://macappstore.org/rpm/)

View File

@ -15,14 +15,10 @@ The main takes for developing a new connector are:
When developing a new database ingestion connector in OpenMetadata, ensure all necessary components are correctly configured. This guide outlines the steps required to define the connector's ingestion capabilities using a `service_spec.py` file.
---
### Why Use `service_spec.py`?
The `service_spec.py` file centralizes the definitions of sources, profilers, lineage, and other ingestion-related components for a connector. This approach helps standardize implementations across connectors, making it easier to manage ingestion workflows.
---
### Steps to Develop a New Connector
#### 1. Create the `service_spec.py` File
@ -57,17 +53,13 @@ ServiceSpec = DefaultDatabaseSpec(
Replace the example classes (e.g., `BigquerySource`, `BigqueryLineageSource`, etc.) with those specific to your connector. Depending on the connector's features, you may include or exclude certain components like usage or profiling.
---
### Components of `service_spec.py`
- **`metadata_source_class`**: Defines the class for metadata ingestion.
- **`lineage_source_class`**: Defines the class for lineage extraction.
- **`usage_source_class`**: Tracks data usage patterns.
- **`profiler_class`**: Profiles data for quality and insights.
- **`sampler_class`**: Samples data for efficient ingestion.
---
- **`sampler_class`**: Samples data for efficient ingestion.
### Example Workflow
@ -80,7 +72,6 @@ Define the `ServiceSpec` using the required classes, adjusting for your connecto
#### Step 3: Verify Integration
Run the ingestion workflow to test the connector and ensure all components are functioning correctly.
## Service Topology
The Topology defines a series of Nodes and Stages that get executed in a hierarchical way and describe how we extract the needed data from the sources.