Pere Miquel Brull 34fbe5d64c
Docs - Prepare 1.7 docs and 1.8 snapshot (#20882)
* DOCS - Prepare 1.7 Release and 1.8 SNAPSHOT

* DOCS - Prepare 1.7 Release and 1.8 SNAPSHOT
2025-04-18 12:12:17 +05:30

3.0 KiB

title slug
Source /sdk/python/build-connector/source

Source

The Source is the connector to external systems and outputs a record for downstream to process and push to OpenMetadata.

Source API

class Source(IterStep, ABC):
    """
    Abstract source implementation. The workflow will run
    its next_record and pass them to the next step.
    """

    metadata: OpenMetadata
    connection_obj: Any
    service_connection: Any
  
    # From the parent - Adding here just to showcase
    @abstractmethod
    def _iter(self) -> Iterable[Either]:
        """Main entrypoint to run through the Iterator"""

    @abstractmethod
    def prepare(self):
        pass

    @abstractmethod
    def test_connection(self) -> None:
        pass

prepare will be called through Python's init method. This will be a place where you could make connections to external sources or initiate the client library.

_iter is where the client can connect to an external resource and emit the data downstream.

test_connection is used (by OpenMetadata supported connectors ONLY) to validate permissions and connectivity before moving forward with the ingestion.

Example

A simple example of this implementation can be found in our demo Custom Connector here

For Consumers of Openmetadata-ingestion to define custom connectors in their own package with same namespace

As a consumer of Openmetadata-ingestion package, You can to add your custom connectors within the same namespace but in a different package repository.

Here is the situation

├─my_code_repository_package
  ├── src
      ├── my_other_relevant_code_package
      ├── metadata
      │   └── ingestion
      │       └── source
      │        └── database
      │         └── my_awesome_connector.py
      └── setup.py
├── openmetadata_ingestion
  ├── src
      ├── metadata
      │   └── ingestion
      │       └── source
      │        └── database
      │         └── existingSource1
      |         └── existingSource2
      |         └── ....
      └── setup.py

If you want my_awesome_connector.py to build as a source and run as a part of workflows defined in openmetadata_ingestion below are the steps.

First add your coustom project in PyCharm. {% image src="/images/v1.8/sdk/python/build-connector/add-project-in-pycharm.png" alt="Add project in pycharm" /%}

Now Go to IDE and Project Settings in PyCharm, inside that go to project section, and select python interpreter, Select virtual environment created for the project as python interpreter {% image src="/images/v1.8/sdk/python/build-connector/select-interpreter.png" alt="Select interpreter in pycharm" /%}

Now apply and okay that interpreter {% image src="/images/v1.8/sdk/python/build-connector/add-interpreter.png" alt="Select interpreter in pycharm" /%}