yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-08-31 12:39:01 +00:00
Code Issues Packages Projects Releases Wiki Activity

Ingestion Workflow Markdown Docs (#11123)

Browse Source 
* Ingestion Workflow Markdown Docs

* Update Section related changes

* Enable Debug Log Id

* address comments
This commit is contained in:
Mayur Singal 2023-04-21 10:45:38 +05:30 committed by GitHub
parent d1d45ea944
commit a72cdf4b61
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
10 changed files with 395 additions and 19 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

93
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Dashboard/workflows/metadata.md Normal file
View File

@ -0,0 +1,93 @@
# Metadata
DashboardService Metadata Pipeline Configuration.
## Properties
$$section
### Dashboard Filter Pattern $(id="dashboardFilterPattern")
Dashboard filter patterns to control whether or not to include dashboard as part of metadata ingestion.
**Include**: Explicitly include dashboards by adding a list of comma-separated regular expressions to the `Include` field. OpenMetadata will include all dashboards with names matching one or more of the supplied regular expressions. All other dashboards will be excluded.
for example, to include only those dashboards for which the name starts with the word `demo`, add regex pattern in include field as `^demo.*`.
**Exclude**: Explicitly exclude dashboards by adding a list of comma-separated regular expressions to the `Exclude` field. OpenMetadata will exclude all dashboards with names matching one or more of the supplied regular expressions. All other dashboards will be included.
for example, to exclude all dashboards with the name containing the word `demo`, add regex pattern in exclude field as `.*demo.*`.
Checkout [this](https://docs.open-metadata.org/connectors/ingestion/workflows/metadata/filter-patterns/database) document for more examples on filter patterns
$$
$$section
### Chart Filter Pattern $(id="chartFilterPattern")
Chart filter patterns to control whether or not to include chart as part of metadata ingestion.
**Include**: Explicitly include charts by adding a list of comma-separated regular expressions to the `Include` field. OpenMetadata will include all charts with names matching one or more of the supplied regular expressions. All other charts will be excluded.
for example, to include only those charts for which the name starts with the word `demo`, add regex pattern in include field as `^demo.*`.
**Exclude**: Explicitly exclude charts by adding a list of comma-separated regular expressions to the `Exclude` field. OpenMetadata will exclude all charts with names matching one or more of the supplied regular expressions. All other charts will be included.
for example, to exclude all charts with the name containing the word `demo`, add regex pattern in exclude field as `.*demo.*`.
Checkout [this](https://docs.open-metadata.org/connectors/ingestion/workflows/metadata/filter-patterns/database) document for more examples on filter patterns
$$
$$section
### Datamodel Filter Pattern $(id="dataModelFilterPattern")
Datamodel filter patterns to control whether or not to include Datamodel as part of metadata ingestion.
**Include**: Explicitly include data models by adding a list of comma-separated regular expressions to the `Include` field. OpenMetadata will include all data models with names matching one or more of the supplied regular expressions. All other data models will be excluded.
for example, to include only those datamodels for which the name starts with the word `demo`, add regex pattern in include field as `^demo.*`.
**Exclude**: Explicitly exclude data models by adding a list of comma-separated regular expressions to the `Exclude` field. OpenMetadata will exclude all data models with names matching one or more of the supplied regular expressions. All other data models will be included.
for example, to exclude all datamodels with the name containing the word `demo`, add regex pattern in exclude field as `.*demo.*`.
Checkout [this](https://docs.open-metadata.org/connectors/ingestion/workflows/metadata/filter-patterns/database) document for more examples on filter patterns
$$
$$section
### Database Service Name $(id="dbServiceNames")
Enter the name of Database Service which is already ingested in OpenMetadata to create lineage between dashboards and database tables.
$$
$$section
### Enable Debug Logs $(id="loggerLevel")
Set the `Enable Debug Log` toggle to set the default log level to debug, these logs can be viewed later in Airflow.
$$
$$section
### Include Owners $(id="includeOwners")
Set the `Include Owner` toggle to control whether to include owners to the ingested entity if its email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten.
$$
$$section
### Include Tags $(id="includeTags")
Set the Include tags toggle to control whether or not to include tags as part of metadata ingestion.
$$
$$section
### Include Data Models $(id="includeDataModels")
Set the Include tags toggle to control whether or not to include data models as part of metadata ingestion.
$$
$$section
### Mark Deleted Dashboard $(id="markDeletedDashboards")
Optional configuration to soft delete 'dashboards' in OpenMetadata if the source 'dashboards' are deleted. After deleting, all the associated entities like lineage, etc., with that 'dashboard' will be deleted.
$$

4
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Database/workflows/lineage.md
View File

@ -16,11 +16,14 @@ $$
Depending on the number of queries ran in the service, this can become an expensive operation. We offer two ways of
limiting the number of parsed queries:
$$section
## Query Log Duration $(id="queryLogDuration")
This is the value in **days** to filter out past queries. For example, being today `2023/04/19`, if we set this value
as 2, we would be listing queries from `2023/04/17` until `2023/04/19` (included).
$$
$$section
## Result Limit $(id="resultLimit")
Another way to limit data is by adding a maximum number of records to process. This value works as:
@ -30,3 +33,4 @@ SELECT xyz FROM query_history limit <resultLimit>
```
This value will take precedence over the `Query Log Duration`.
$$

121
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Database/workflows/metadata.md
View File

@ -3,32 +3,117 @@
DatabaseService Metadata Pipeline Configuration.
## Properties
$$section
### Database Filter Pattern $(id="databaseFilterPattern")
Database filter patterns to control whether or not to include database as part of metadata ingestion.
**Include**: Explicitly include databases by adding a list of comma-separated regular expressions to the `Include` field. OpenMetadata will include all databases with names matching one or more of the supplied regular expressions. All other databases will be excluded.
for example, to include only those databases for which the name starts with the word `demo`, add regex pattern in include field as `^demo.*`.
**Exclude**: Explicitly exclude databases by adding a list of comma-separated regular expressions to the `Exclude` field. OpenMetadata will exclude all databases with names matching one or more of the supplied regular expressions. All other databases will be included.
for example, to exclude all databases with the name containing the word `demo`, add regex pattern in exclude field as `.*demo.*`.
Checkout [this](https://docs.open-metadata.org/connectors/ingestion/workflows/metadata/filter-patterns/database#database-filter-pattern) document for examples on database filter pattern
$$
$$section
### Schema Filter Pattern $(id="schemaFilterPattern")
Schema filter patterns are used to control whether or not to include schemas as part of metadata ingestion.
**Include**: Explicitly include schemas by adding a list of comma-separated regular expressions to the `Include` field. OpenMetadata will include all schemas with names matching one or more of the supplied regular expressions. All other schemas will be excluded.
for example, to include only those schemas for which the name starts with the word `demo`, add regex pattern in include field as `^demo.*`.
**Exclude**: Explicitly exclude schemas by adding a list of comma-separated regular expressions to the `Exclude` field. OpenMetadata will exclude all schemas with names matching one or more of the supplied regular expressions. All other schemas will be included.
for example, to exclude all schemas with the name containing the word `demo`, add regex pattern in exclude field as `.*demo.*`.
Checkout [this](https://docs.open-metadata.org/connectors/ingestion/workflows/metadata/filter-patterns/database#database-filter-pattern) document for examples on schema filter pattern
$$
$$section
### Table Filter Pattern $(id="tableFilterPattern")
Table filter patterns are used to control whether or not to include tables as part of metadata ingestion.
**Include**: Explicitly include tables by adding a list of comma-separated regular expressions to the `Include` field. OpenMetadata will include all tables with names matching one or more of the supplied regular expressions. All other tables will be excluded.
for example, to include only those tables for which the name starts with the word `demo`, add regex pattern in include field as `^demo.*`.
**Exclude**: Explicitly exclude tables by adding a list of comma-separated regular expressions to the `Exclude` field. OpenMetadata will exclude all tables with names matching one or more of the supplied regular expressions. All other tables will be included.
for example, to exclude all tables with the name containing the word `demo`, add regex pattern in exclude field as `.*demo.*`.
Checkout [this](https://docs.open-metadata.org/connectors/ingestion/workflows/metadata/filter-patterns/database#table-filter-pattern) document for examples on table filter pattern
$$
$$section
### Use FQN For Filtering $(id="useFqnForFiltering")
This flag set when you want to apply the filter on fully qualified name (e.g `service_name.db_name.schema_name.table_name`) instead of applying the filter to raw name of entity (e.g `table_name`).
This Flag is useful in scenario when you have schema with same name in different databases, or table with same name in different schemas and you want to filter out one of them.
Checkout [this](https://docs.open-metadata.org/connectors/ingestion/workflows/metadata/filter-patterns/database#table-filter-pattern) document for examples on how to use this field.
$$
$$section
### Include Views $(id="includeViews")
Set the `Include Views` toggle to control whether or not to include views as part of metadata ingestion.
$$
$$section
### Include Tags $(id="includeTags")
Set the `Include Tags` toggle to control whether or not to include tags as part of metadata ingestion.
$$
$$section
### Enable Debug Logs $(id="loggerLevel")
Set the `Enable Debug Log` toggle to set the default log level to debug, these logs can be viewed later in Airflow.
$$
$$section
### Mark Deleted Tables $(id="markDeletedTables")
This is an optional configuration for enabling soft deletion of tables. When this option is enabled, only tables that have been deleted from the source will be soft deleted, and this will apply solely to the schema that is currently being ingested via the pipeline. Any related entities such as test suites or lineage information that were associated with those tables will also be deleted.
Following are some examples of scenarios in which table will get soft deleted if this flag is enabled.
- If you have not applied any filter patterns for this ingestion pipeline then if any table was deleted from the data source, then the same table will be soft deleted from OpenMetadata as well.
- If you have applied a `Schema Filter Pattern` to include `SchemaA` then any table deleted from `SchemaA` will also be soft deleted from Openmetadata.
- If `TableA` was already ingested in OpenMetadata then later you apply a `Table Filter Pattern` to exclude `TableA` then `TableA` will get soft deleted from OpenMetadata.
Following are some examples of scenarios in which table will **NOT** get soft deleted if this flag is enabled.
- If you already have `SchemaA` & `SchemaB` ingested in OpenMetadata then later you apply a `Schema Filter Pattern` to exclude `SchemaB`, then no table from `SchemaB` will be deleted due to this ingestion pipeline.
- If you already have `SchemaA` & `SchemaB` ingested in OpenMetadata and for this ingestion pipeline you have applies a `Schema Filter Pattern` to include only `SchemaA` then any table deleted from `SchemaB` will not be deleted due to this ingestion pipeline.
$$
$$section
### Mark All Deleted Tables $(id="markAllDeletedTables")
This is an optional configuration for enabling soft deletion of tables. When this option is enabled, only tables that have been deleted from the source will be soft deleted, and this will apply to all the schemas available in the data source. Any related entities such as test suites or lineage information that were associated with those tables will also be deleted. Do not enable this option when you have multiple metadata ingestion pipelines. Also make sure to enable the markDeletedTables option for this to work.
$$
$$section
### Schema Filter Pattern $(id="schemaFilterPattern")
Regex to only fetch tables or databases that matches the pattern.
$$
$$section
### Table Filter Pattern $(id="tableFilterPattern")
Regex exclude tables or databases that matches the pattern.
$$
$$section
### Database Filter Pattern $(id="databaseFilterPattern")
Regex to only fetch databases that matches the pattern.
**It is recommended to be cautious while enabling this flag if you have multiple ingestion pipelines running for the same service, Because it is possible that a pipelines might delete the tables ingested by other pipeline.**
Following are some examples of scenarios in which table will get soft deleted if this flag is enabled.
- If you have not applied any filter patterns for this ingestion pipeline then if any table was deleted from the data source, then the same table will be soft deleted from OpenMetadata as well.
- If you have applied a Schema Filter Pattern to include `SchemaA` then any table deleted from `SchemaA` will also be soft deleted from Openmetadata.
- If `TableA` was already ingested in OpenMetadata then later you apply a `Table Filter Pattern` to exclude `TableA` then `TableA` will get soft deleted from OpenMetadata.
- If you already have `SchemaA` & `SchemaB` ingested in OpenMetadata then later you apply a `Schema Filter Pattern` to exclude `SchemaB`, then all table from `SchemaB` will be deleted due to this ingestion pipeline.
$$

38
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Database/workflows/profiler.md
View File

@ -2,28 +2,60 @@
This workflow allows you to profile your table assets an gain insight into their structure (e.g. of metrics computed: `max`, `min`, `mean`, etc. The full list can be found [here](https://docs.open-metadata.org/connectors/ingestion/workflows/profiler/metrics)). We recommend to check the [best practices](https://docs.open-metadata.org/connectors/ingestion/workflows/profiler#profiler-best-practices) before creating a profiler workflow.
## Properties
$$section
### Database Filter Pattern $(id="databaseFilterPattern")
Regex to only fetch databases that matches the pattern.
$$
$$section
### Schema Filter Pattern $(id="schemaFilterPattern")
Regex to only fetch schema that matches the pattern.
$$
$$section
### Table Filter Pattern $(id="tableFilterPattern")
Regex exclude tables that matches the pattern.
$$
### Process PII Sensitive $(id="processPiiSensitive")
Optional configuration to automatically tag columns that might contain sensitive information. If `generateSampleData` is enabled, OpenMetadata will leverage machine learning to infer which column may contain PII sensitive data. If disabled, OpenMetadata will infer from the column name.
$$section
### Profile Sample $(id="profileSample")
Percentage of data or number of rows to use when sampling tables. If left as is, the profiler will run against the entire table.
$$
$$section
### Profile Sample Type $(id="profileSampleType")
Profile sample type can be set to either:
* percentage: this will use a percentage to sample sample the table (e.g. table has 100 rows and we set sample percentage tp 50%, the profiler will use 50 random rows to compute the metrics)
* row count: this will use a number of rows to sample the table (e.g. table has 100 rows and we set row count to 10, the profiler will use 10 random rows to compute the metrics)
$$
$$section
### Thread Count $(id="threadCount")
Number of thread that will be used when computing the profiler metrics. A number set to high can have negative effect on performance. We recommend to use the default value unless you have a good understanding of multithreading.
$$
$$section
### Timeout in Seconds $(id="timeoutSeconds")
This will set the duration a profiling job against a table should wait before interrupting its execution and moving on to profiling the next table. It is important to note that the profiler will wait for the hanging query to terminiate before killing the execution. If there is a risk for your profiling job to hang, it is important to also set a query/connection timeout on your database engine. The default value for the profiler timeout is 12-hours.
$$
$$section
### Ingest Sample Data $(id="generateSampleData")
Set the Ingest Sample Data toggle to control whether or not to ingest sample data as part of profiler ingestion. If this toggle is enabled, 100 rows will be ingested by default.
$$
$$section
### Enable Debug Logs $(id="loggerLevel")
Set the `Enable Debug Log` toggle to set the default log level to debug, these logs can be viewed later in Airflow.
$$
$$section
### Auto Tag PII $(id="processPiiSensitive")
Set the `Auto Tag PII` toggle to control whether or not automatically tag columns that might contain sensitive information as part of profiler ingestion.
If `Ingest Sample Data` is enabled, OpenMetadata will leverage machine learning to infer which column may contain PII sensitive data. If disabled, OpenMetadata will infer from the column name.
$$

4
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Database/workflows/usage.md
View File

@ -19,11 +19,14 @@ frequently joined tables.
Depending on the number of queries ran in the service, this can become an expensive operation. We offer two ways of
limiting the number of parsed queries:
$$section
## Query Log Duration $(id="queryLogDuration")
This is the value in **days** to filter out past queries. For example, being today `2023/04/19`, if we set this value
as 2, we would be listing queries from `2023/04/17` until `2023/04/19` (included).
$$
$$section
## Result Limit $(id="resultLimit")
Another way to limit data is by adding a maximum number of records to process. This value works as:
@ -33,3 +36,4 @@ SELECT xyz FROM query_history limit <resultLimit>
```
This value will take precedence over the `Query Log Duration`.
$$

40
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Messaging/workflows/metadata.md Normal file
View File

@ -0,0 +1,40 @@
# Metadata
MessagingService Metadata Pipeline Configuration.
## Properties
$$section
### Topic Filter Pattern $(id="topicFilterPattern")
Topic filter patterns to control whether or not to include topic as part of metadata ingestion.
**Include**: Explicitly include topics by adding a list of comma-separated regular expressions to the `Include` field. OpenMetadata will include all topics with names matching one or more of the supplied regular expressions. All other topics will be excluded.
for example, to include only those topics for which the name starts with the word `demo`, add regex pattern in include field as `^demo.*`.
**Exclude**: Explicitly exclude topics by adding a list of comma-separated regular expressions to the `Exclude` field. OpenMetadata will exclude all topics with names matching one or more of the supplied regular expressions. All other topics will be included.
for example, to exclude all topics with the name containing the word `demo`, add regex pattern in exclude field as `.*demo.*`.
Checkout [this](https://docs.open-metadata.org/connectors/ingestion/workflows/metadata/filter-patterns/database) document for more examples on filter patterns
$$
$$section
### Ingest Sample Data $(id="generateSampleData")
Set the Ingest Sample Data toggle to control whether or not to topic sample data as part of metadata ingestion.
$$
$$section
### Enable Debug Logs $(id="loggerLevel")
Set the `Enable Debug Log` toggle to set the default log level to debug, these logs can be viewed later in Airflow.
$$
$$section
### Mark Deleted Topics $(id="markDeletedTopics")
Optional configuration to soft delete 'topics' in OpenMetadata if the source 'topics' are deleted. After deleting, all the associated entities like lineage, etc., with that 'topic' will be deleted.
$$

10
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Metadata/workflows/metadata.md Normal file
View File

@ -0,0 +1,10 @@
# Metadata
MetadataService Metadata Pipeline Configuration.
$$section
### Enable Debug Logs $(id="loggerLevel")
Set the `Enable Debug Log` toggle to set the default log level to debug, these logs can be viewed later in Airflow.
$$

34
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Mlmodel/workflows/metadata.md Normal file
View File

@ -0,0 +1,34 @@
# Metadata
MessagingService Metadata Pipeline Configuration.
## Properties
$$section
### ML Model Filter Pattern $(id="mlModelFilterPattern")
ML model filter patterns to control whether or not to include model as part of metadata ingestion.
**Include**: Explicitly include models by adding a list of comma-separated regular expressions to the `Include` field. OpenMetadata will include all models with names matching one or more of the supplied regular expressions. All other models will be excluded.
for example, to include only those models for which the name starts with the word `demo`, add regex pattern in include field as `^demo.*`.
**Exclude**: Explicitly exclude models by adding a list of comma-separated regular expressions to the `Exclude` field. OpenMetadata will exclude all models with names matching one or more of the supplied regular expressions. All other models will be included.
for example, to exclude all models with the name containing the word `demo`, add regex pattern in exclude field as `.*demo.*`.
Checkout [this](https://docs.open-metadata.org/connectors/ingestion/workflows/metadata/filter-patterns/database) document for more examples on filter patterns
$$
$$section
### Enable Debug Logs $(id="loggerLevel")
Set the `Enable Debug Log` toggle to set the default log level to debug, these logs can be viewed later in Airflow.
$$
$$section
### Mark Deleted Ml Model $(id="markDeletedMlModels")
Optional configuration to soft delete 'ML models' in OpenMetadata if the source 'ML models' are deleted. After deleting, all the associated entities like lineage, etc., with that 'ML model' will be deleted.
$$

46
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Pipeline/workflows/metadata.md Normal file
View File

@ -0,0 +1,46 @@
# Metadata
PipelineService Metadata Pipeline Configuration.
## Properties
$$section
### Pipeline Filter Pattern $(id="pipelineFilterPattern")
Pipeline filter patterns to control whether or not to include pipeline as part of metadata ingestion.
**Include**: Explicitly include pipelines by adding a list of comma-separated regular expressions to the `Include` field. OpenMetadata will include all pipelines with names matching one or more of the supplied regular expressions. All other pipelines will be excluded.
for example, to include only those pipelines for which the name starts with the word `demo`, add regex pattern in include field as `^demo.*`.
**Exclude**: Explicitly exclude pipelines by adding a list of comma-separated regular expressions to the `Exclude` field. OpenMetadata will exclude all pipelines with names matching one or more of the supplied regular expressions. All other pipelines will be included.
for example, to exclude all pipelines with the name containing the word `demo`, add regex pattern in exclude field as `.*demo.*`.
Checkout [this](https://docs.open-metadata.org/connectors/ingestion/workflows/metadata/filter-patterns/database) document for more examples on filter patterns
$$
$$section
### Include Lineage $(id="includeLineage")
Set the Include Lineage toggle to control whether or not to include lineage between pipelines and data sources as part of metadata ingestion.
$$
$$section
### Enable Debug Logs $(id="loggerLevel")
Set the `Enable Debug Log` toggle to set the default log level to debug, these logs can be viewed later in Airflow.
$$
$$section
### Include Tags $(id="includeTags")
Set the Include tags toggle to control whether or not to include tags as part of metadata ingestion.
$$
$$section
### Mark Deleted Pipelines $(id="markDeletedPipeline")
Optional configuration to soft delete 'pipelines' in OpenMetadata if the source 'pipelines' are deleted. After deleting, all the associated entities like lineage, etc., with that 'pipeline' will be deleted.
$$

28
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Storage/workflows/metadata.md Normal file
View File

@ -0,0 +1,28 @@
# Metadata
StorageService Metadata Pipeline Configuration.
## Properties
$$section
### Container Filter Pattern $(id="containerFilterPattern")
Container filter patterns to control whether or not to include container as part of metadata ingestion.
**Include**: Explicitly include containers by adding a list of comma-separated regular expressions to the `Include` field. OpenMetadata will include all containers with names matching one or more of the supplied regular expressions. All other containers will be excluded.
for example, to include only those containers for which the name starts with the word `demo`, add regex pattern in include field as `^demo.*`.
**Exclude**: Explicitly exclude containers by adding a list of comma-separated regular expressions to the `Exclude` field. OpenMetadata will exclude all containers with names matching one or more of the supplied regular expressions. All other containers will be included.
for example, to exclude all containers with the name containing the word `demo`, add regex pattern in exclude field as `.*demo.*`.
Checkout [this](https://docs.open-metadata.org/connectors/ingestion/workflows/metadata/filter-patterns/database) document for more examples on filter patterns
$$
$$section
### Enable Debug Logs $(id="loggerLevel")
Set the `Enable Debug Log` toggle to set the default log level to debug, these logs can be viewed later in Airflow.
$$