mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-07-06 08:37:22 +00:00
e2578d6be3
* GitBook: [#177] Documentation Update - Airflow * GitBook: [#195] Removing Cron from databaseServices * GitBook: [#196] Added trino * GitBook: [#197] removed cron from config * GitBook: [#198] Added Redash Documentation * GitBook: [#199] Added Bigquery Usage Documentation * GitBook: [#200] Added page link for presto * GitBook: [#201] Added Local Docker documentation * GitBook: [#202] Added Documentation for Local Docker Setup * GitBook: [#203] Added Git Command to clone Openmetadata in docs * GitBook: [#207] links update * GitBook: [#208] Updating Airflow Documentation * GitBook: [#210] Adding Python installation package under Airflow Lineage config * GitBook: [#211] Change the links to 0.5..0 * GitBook: [#213] Move buried connectors page up * GitBook: [#214] Update to connectors page * GitBook: [#215] Removed sub-categories * GitBook: [#212] Adding Discovery tutorial * GitBook: [#220] Updated steps to H2s. * GitBook: [#230] Complex queries * GitBook: [#231] Add lineage to feature overview * GitBook: [#232] Make feature overview headers verbs instead of nouns * GitBook: [#233] Add data reliability to features overview * GitBook: [#234] Add complex data types to feature overview * GitBook: [#235] Simplify and further distinguish discovery feature headers * GitBook: [#236] Add data importance to feature overview * GitBook: [#237] Break Connectors into its own section * GitBook: [#238] Reorganize first section of docs. * GitBook: [#239] Add connectors to feature overview * GitBook: [#240] Organize layout of feature overview into feature categories as agreed with Harsha. * GitBook: [#242] Make overview paragraph more descriptive. * GitBook: [#243] Create a link to Connectors section from feature overview. * GitBook: [#244] Add "discover data through association" to feature overview. * GitBook: [#245] Update importance and owners gifs * GitBook: [#246] Include a little more descriptive documentation for key features. * GitBook: [#248] Small tweaks to intro paragraph. * GitBook: [#249] Clean up data profiler paragraph. * GitBook: [#250] Promote Complex Data Types to its own feature. * GitBook: [#251] Update to advanced search * GitBook: [#252] Update Roadmap * GitBook: [#254] Remove old features page (text and screenshot based). * GitBook: [#255] Remove references to removed page. * GitBook: [#256] Add Descriptions and Tags section to feature overview. * GitBook: [#257] Update title for "Know Your Data" Co-authored-by: Ayush Shah <ayush.shah@deuexsolutions.com> Co-authored-by: Suresh Srinivas <suresh@getcollate.io> Co-authored-by: Shannon Bradshaw <shannon.bradshaw@arrikto.com> Co-authored-by: OpenMetadata <github@harsha.io>
82 lines
2.9 KiB
Markdown
82 lines
2.9 KiB
Markdown
# Source
|
|
|
|
The Source is the connector to external systems and outputs a record for downstream to process and push to OpenMetadata.
|
|
|
|
## Source API
|
|
|
|
```python
|
|
@dataclass # type: ignore[misc]
|
|
class Source(Closeable, metaclass=ABCMeta):
|
|
ctx: WorkflowContext
|
|
|
|
@classmethod
|
|
@abstractmethod
|
|
def create(cls, config_dict: dict, metadata_config_dict: dict, ctx: WorkflowContext) -> "Source":
|
|
pass
|
|
|
|
@abstractmethod
|
|
def prepare(self):
|
|
pass
|
|
|
|
@abstractmethod
|
|
def next_record(self) -> Iterable[Record]:
|
|
pass
|
|
|
|
@abstractmethod
|
|
def get_status(self) -> SourceStatus:
|
|
pass
|
|
```
|
|
|
|
**create** method is used to create an instance of Source
|
|
|
|
**prepare** will be called through Python's init method. This will be a place where you could make connections to external sources or initiate the client library
|
|
|
|
**next_record** is where the client can connect to an external resource and emit the data downstream
|
|
|
|
**get_status** is for the [workflow](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/api/workflow.py) to call and report the status of the source such as how many records its processed any failures or warnings
|
|
|
|
## Example
|
|
|
|
A simple example of this implementation is
|
|
|
|
```python
|
|
class SampleTablesSource(Source):
|
|
|
|
def __init__(self, config: SampleTableSourceConfig, metadata_config: MetadataServerConfig, ctx):
|
|
super().__init__(ctx)
|
|
self.status = SampleTableSourceStatus()
|
|
self.config = config
|
|
self.metadata_config = metadata_config
|
|
self.client = REST(metadata_config)
|
|
self.service_json = json.load(open(config.sample_schema_folder + "/service.json", 'r'))
|
|
self.database = json.load(open(config.sample_schema_folder + "/database.json", 'r'))
|
|
self.tables = json.load(open(config.sample_schema_folder + "/tables.json", 'r'))
|
|
self.service = get_service_or_create(self.service_json, metadata_config)
|
|
|
|
@classmethod
|
|
def create(cls, config_dict, metadata_config_dict, ctx):
|
|
config = SampleTableSourceConfig.parse_obj(config_dict)
|
|
metadata_config = MetadataServerConfig.parse_obj(metadata_config_dict)
|
|
return cls(config, metadata_config, ctx)
|
|
|
|
def prepare(self):
|
|
pass
|
|
|
|
def next_record(self) -> Iterable[OMetaDatabaseAndTable]:
|
|
db = DatabaseEntity(id=uuid.uuid4(),
|
|
name=self.database['name'],
|
|
description=self.database['description'],
|
|
service=EntityReference(id=self.service.id, type=self.config.service_type))
|
|
for table in self.tables['tables']:
|
|
table_metadata = TableEntity(**table)
|
|
table_and_db = OMetaDatabaseAndTable(table=table_metadata, database=db)
|
|
self.status.scanned(table_metadata.name.__root__)
|
|
yield table_and_db
|
|
|
|
def close(self):
|
|
pass
|
|
|
|
def get_status(self):
|
|
return self.status
|
|
```
|