In this guide, we will use the Python SDK to create and fetch Lineage information.
For simplicity, we are going to create lineage between Tables. However, this would work with ANY entity.
You can find the Lineage Entity defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/type/entityLineage.json),
as well as the Entity defining the payload to add a new lineage: [AddLineage](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/api/lineage/addLineage.json).
{% note %}
Note that in OpenMetadata, the Lineage information is just a possible relationship between Entities. Other types
of relationships for example could be:
- Contains (a Database contains Schemas, which at the same time contain Tables),
- or Ownership of any asset.
The point being, any Entity existent in OpenMetadata can be related to any other via Lineage.
{% /note %}
In the following sections we will:
- Create a Database Service, a Database, a Schema and two Tables,
- Add Lineage between both Tables,
- Get the Lineage information back.
A **prerequisite** for this section is to have previously gone through the following [docs](/sdk/python).
## Creating the Entities
To prepare the necessary ingredients, execute the following steps.
All functions that we are going to use related to Lineage can be found in [here](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/ometa/mixins/lineage_mixin.py)
### 1. Preparing the Client
```python
from metadata.generated.schema.entity.services.connections.metadata.openMetadataConnection import (
OpenMetadataConnection,
)
from metadata.generated.schema.security.client.openMetadataJWTClientConfig import (
OpenMetadataJWTClientConfig,
)
from metadata.ingestion.ometa.ometa_api import OpenMetadata
server_config = OpenMetadataConnection(
hostPort="http://localhost:8585/api",
authProvider="openmetadata",
securityConfig=OpenMetadataJWTClientConfig(
jwtToken="<token>"
),
)
metadata = OpenMetadata(server_config)
assert metadata.health_check() # Will fail if we cannot reach the server
```
### 2. Creating the Database Service
We are mocking a MySQL instance. Note how we need to pass the right configuration class `MysqlConnection`, as a
parameter for the generic `DatabaseConnection` type.
```python
from metadata.generated.schema.api.services.createDatabaseService import (
CreateDatabaseServiceRequest,
)
from metadata.generated.schema.entity.services.connections.database.common.basicAuth import (
BasicAuth,
)
from metadata.generated.schema.entity.services.connections.database.mysqlConnection import (
MysqlConnection,
)
from metadata.generated.schema.entity.services.databaseService import (
You can also get lineage by ID using the `get_lineage_by_id` method, which accepts `entity_id` instead of `fqn`.
{% /note %}
# Lineage Details
Note how when adding lineage information we give to the API an [AddLineage](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/api/lineage/addLineage.json)
Request. This is composed of an Entity Edge, whose definition you can find [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/type/entityLineage.json#L75).
In a nutshell, an Entity Edge has:
1. The Entity Reference as the lineage origin,
2. The Entity Reference as the lineage destination,
3. Optionally, Lineage Details.
In the Lineage Details property we can pass further information specific about Table to Table lineage:
-`sqlQuery` specifying the transformation,
- An array of `columnsLineage` as an object with an array of source and destination columns, as well as their own specific transformation function,
- Optionally, the Entity Reference of a Pipeline powering the transformation from Table A to Table B.
The API call will be exactly the same as before, but now we will add more ingredients when defining our objects. Let's
see how to do that and play with the possible combinations:
First, import the required classes and create a new table:
```python
from metadata.generated.schema.type.entityLineage import (
The UI currently supports showing the column lineage information. Data about the SQL queries and the Pipeline Entities
will be surfaced soon. Thanks!
{% /note %}
# Automated SQL lineage
In case you want OpenMetadata to identify the lineage based on the sql query, then you can make use of the method `add_lineage_by_query` of the python SDK to parser the sql and generate the lineage in OpenMetadata.
follow the below code snippet for the example:
```python
from metadata.generated.schema.entity.services.databaseService import DatabaseService
sql="insert into target_table(id) as select id from source_table" # your sql query
)
```
Above example would create a lineage between `target_table` and `source_table` within `my_service` database service.
## Automated SQL lineage via CLI
To create the automated sql lineage via CLI, you need to make sure that you have installed the openmetadata-ingestion package in your local environment using command `pip install openmetadata-ingestion`.
Once that is done you will have to prepare a yaml file as follows.
```yaml
serviceName: local_mysql
query: insert into target_table(id) as select id from source_table
# filePath: test.sql
# parseTimeout: 360 # timeout in seconds
workflowConfig:
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
openMetadataServerConfig:
hostPort: <OpenMetadatahostandport>
authProvider: <OpenMetadataauthprovider>
```
- **serviceName**: Name of the database service which contains the table involved in query.
- **query**: You can specify the raw sql query within the yaml file itself.
- **filePath**: In case the query is too big then you can also save query in a file and pass the path to the file in this field.
- **parseTimeout**: Timeout for the lineage parsing process.
- **workflowConfig**: The main property here is the openMetadataServerConfig, where you can define the host and security provider of your OpenMetadata installation.
Once the yaml file is prepare you can run the command
