yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-07-09 10:09:09 +00:00
Code Issues Packages Projects Releases Wiki Activity
OpenMetadata/openmetadata-docs/content/v1.6.x/main-concepts/metadata-standard/schemas/metadataIngestion/databaseServiceQueryLineagePipeline.md
tarunpandey23 69342d4783
Feat: update generate docs schema (#19372) 
* Updated generate_docs_schema

* Updated generate_docs_schema

* Updated generate_docs_schema

* applied generate_docs_schema script

* Revert the changes in this file

---------

Co-authored-by: Tarun <tarun.p@deuexsolutions.com>
Co-authored-by: Prajwal214 <167504578+Prajwal214@users.noreply.github.com>
2025-01-15 17:46:55 +05:30

2.9 KiB
Raw Permalink Blame History

title slug
databaseServiceQueryLineagePipeline /main-concepts/metadata-standard/schemas/metadataingestion/databaseservicequerylineagepipeline

DatabaseServiceQueryLineagePipeline

DatabaseService Query Lineage Pipeline Configuration.

Properties

  • type: Pipeline type. Refer to #/definitions/databaseLineageConfigType. Default: "DatabaseLineage".
  • queryLogDuration (integer): Configuration to tune how far we want to look back in query logs to process lineage data. Default: 1.
  • queryLogFilePath (string): Configuration to set the file path for query logs.
  • resultLimit (integer): Configuration to set the limit for query logs. Default: 1000.
  • parsingTimeoutLimit (integer): Configuration to set the timeout for parsing the query in seconds. Default: 300.
  • filterCondition (string): Configuration the condition to filter the query history.
  • schemaFilterPattern: Regex to only fetch tables or databases that matches the pattern. Refer to ../type/filterPattern.json#/definitions/filterPattern.
  • tableFilterPattern: Regex exclude tables or databases that matches the pattern. Refer to ../type/filterPattern.json#/definitions/filterPattern.
  • databaseFilterPattern: Regex to only fetch databases that matches the pattern. Refer to ../type/filterPattern.json#/definitions/filterPattern.
  • overrideViewLineage (boolean): Set the 'Override View Lineage' toggle to control whether to override the existing view lineage. Default: false.
  • processViewLineage (boolean): Set the 'Process View Lineage' toggle to control whether to process view lineage. Default: true.
  • processQueryLineage (boolean): Set the 'Process Query Lineage' toggle to control whether to process query lineage. Default: true.
  • processStoredProcedureLineage (boolean): Set the 'Process Stored ProcedureLog Lineage' toggle to control whether to process stored procedure lineage. Default: true.
  • threads (integer): Number of Threads to use in order to parallelize lineage ingestion. Minimum: 1. Default: 1.
  • processCrossDatabaseLineage (boolean): Set the 'Process Cross Database Lineage' toggle to control whether to process table lineage across different databases. Default: false.
  • crossDatabaseServiceNames (array): Set 'Cross Database Service Names' to process lineage with the database.
    • Items (string)

Definitions

  • databaseLineageConfigType (string): Database Source Config Usage Pipeline type. Must be one of: ["DatabaseLineage"]. Default: "DatabaseLineage".

Documentation file automatically generated at 2025-01-15 09:05:25.266839+00:00.