mgorsk1 98850ab5cc
feat: OpenLineage integration (#15317)
* 🎉 Init OpenLineage connector

Co-authored-by: dechoma <dominik.choma@gmail.com>

* MLH - make linter happy

* review fixes

* 🐛 Fix path for ol event in tests

* 🐛 Fix path for ol event in tests

* Update ingestion/setup.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/models.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* review fixes 2

* linter

* review

* review

* make linter happy

* fix test_yield_pipeline_lineage_details test

* make linter happy

* fix tests

* fix tests 2

---------

Co-authored-by: dechoma <dominik.choma@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-03-12 08:39:25 +01:00

89 lines
1.7 KiB
Python

# Copyright 2021 Collate
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Openlineage Source Model module
"""
from dataclasses import dataclass
from enum import Enum
from typing import Any, Dict, List
@dataclass
class OpenLineageEvent:
"""
An object containing data extracted from raw OpenLineage event. Used as a basis for all abstract methods of
OpenlineageSource connector.
"""
run_facet: Dict
job: Dict
event_type: str
inputs: List[Any]
outputs: List[Any]
@dataclass
class TableFQN:
"""
Fully Qualified Name of a Table.
"""
value: str
@dataclass
class ColumnFQN:
"""
Fully Qualified Name of a Column.
"""
value: str
@dataclass
class LineageNode:
"""
A node being a part of Lineage information.
"""
uuid: str
fqn: TableFQN
node_type: str = "table"
@dataclass
class LineageEdge:
"""
An object describing connection of two nodes in the Lineage information.
"""
from_node: LineageNode
to_node: LineageNode
@dataclass
class TableDetails:
"""
Minimal table information.
"""
schema: str
name: str
class EventType(str, Enum):
"""
List of used OpenLineage event types.
"""
COMPLETE = "COMPLETE"