diff --git a/docs/connectors/mariadb.md b/docs/connectors/mariadb.md index 8c6f992f906..4416faa67ac 100644 --- a/docs/connectors/mariadb.md +++ b/docs/connectors/mariadb.md @@ -40,7 +40,10 @@ metadata ingest -c ./examples/workflows/mariadb.json "password": "openmetadata_password", "database": "openmetadata_db", "service_name": "local_mysql", - "filter_pattern": { + "table_filter_pattern": { + "excludes": ["demo.*","orders.*"] + }, + "schema_filter_pattern": { "excludes": ["mysql.*", "information_schema.*", "performance_schema.*", "sys.*"] } } @@ -52,10 +55,11 @@ metadata ingest -c ./examples/workflows/mariadb.json 1. **username** - pass the MariaDB username. 2. **password** - password for the username 3. **service\_name** - Service Name for this MariaDB cluster. If you added MariaDB cluster through OpenMetadata UI, make sure the service name matches the same. -4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata -5. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you the newly ingested data. -6. **data\_profiler\_offset** - Specify offset. -7. **data\_profiler\_limit** - Specify limit. +4. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +5. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. +6. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you the newly ingested data. +7. **data\_profiler\_offset** - Specify offset. +8. **data\_profiler\_limit** - Specify limit. ## Publish to OpenMetadata @@ -73,8 +77,11 @@ Add optionally `pii` processor and `metadata-rest` sink along with `metadata-ser "password": "openmetadata_password", "database": "openmetadata_db", "service_name": "local_mysql", - "filter_pattern": { - "excludes": ["mysql.*", "information_schema.*", "performance_schema.*", "sys.*"] + "table_filter_pattern": { + "excludes": ["demo.*","orders.*"] + }, + "schema_filter_pattern": { + "excludes": ["information_schema.*"] } } }, diff --git a/docs/openmetadata/connectors/athena.md b/docs/openmetadata/connectors/athena.md index d718f09bf2f..f9fcf54f346 100644 --- a/docs/openmetadata/connectors/athena.md +++ b/docs/openmetadata/connectors/athena.md @@ -44,7 +44,8 @@ pip install 'openmetadata-ingestion[athena]' 1. **username** - pass the Athena username. We recommend creating a user with read-only permissions to all the databases in your Athena installation 2. **password** - password for the username 3. **service\_name** - Service Name for this Athena cluster. If you added the Athena cluster through OpenMetadata UI, make sure the service name matches the same. -4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata +4. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +5. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. ## Publish to OpenMetadata diff --git a/docs/openmetadata/connectors/bigquery-usage.md b/docs/openmetadata/connectors/bigquery-usage.md index 3fbdd69d65e..6ca322f6b78 100644 --- a/docs/openmetadata/connectors/bigquery-usage.md +++ b/docs/openmetadata/connectors/bigquery-usage.md @@ -69,8 +69,9 @@ metadata ingest -c ./examples/workflows/bigquery_usage.json 1. **username** - pass the Bigquery username. 2. **password** - the password for the Bigquery username. 3. **service\_name** - Service Name for this Bigquery cluster. If you added the Bigquery cluster through OpenMetadata UI, make sure the service name matches the same. -4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata. -5. **database -** Database name from where data is to be fetched. +4. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. +5. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +6. **database -** Database name from where data is to be fetched. ### Publish to OpenMetadata diff --git a/docs/openmetadata/connectors/bigquery.md b/docs/openmetadata/connectors/bigquery.md index 7dbca54400d..4f38a6389e6 100644 --- a/docs/openmetadata/connectors/bigquery.md +++ b/docs/openmetadata/connectors/bigquery.md @@ -63,7 +63,10 @@ metadata ingest -c ./examples/workflows/bigquery.json "options": { "credentials_path": "examples/creds/bigquery-cred.json" }, - "filter_pattern": { + "table_filter_pattern": { + "excludes": ["demo.*","orders.*"] + }, + "schema_filter_pattern": { "excludes": [ "[\\w]*cloudaudit.*", "[\\w]*logging_googleapis_com.*", @@ -78,11 +81,12 @@ metadata ingest -c ./examples/workflows/bigquery.json 1. **username** - pass the Bigquery username. 2. **password** - the password for the Bigquery username. 3. **service\_name** - Service Name for this Bigquery cluster. If you added the Bigquery cluster through OpenMetadata UI, make sure the service name matches the same. -4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata. -5. **database -** Database name from where data is to be fetched. -6. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you the newly ingested data. -7. **data\_profiler\_offset** - Specify offset. -8. **data\_profiler\_limit** - Specify limit. +4. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. +5. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +6. **database -** Database name from where data is to be fetched. +7. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you the newly ingested data. +8. **data\_profiler\_offset** - Specify offset. +9. **data\_profiler\_limit** - Specify limit. ### Publish to OpenMetadata @@ -106,7 +110,10 @@ Add `metadata-rest` sink along with `metadata-server` config "options": { "credentials_path": "examples/creds/bigquery-cred.json" }, - "filter_pattern": { + "table_filter_pattern": { + "excludes": ["demo.*","orders.*"] + }, + "schema_filter_pattern": { "excludes": [ "[\\w]*cloudaudit.*", "[\\w]*logging_googleapis_com.*", diff --git a/docs/openmetadata/connectors/hive.md b/docs/openmetadata/connectors/hive.md index 9dd93383412..f4060b6e553 100644 --- a/docs/openmetadata/connectors/hive.md +++ b/docs/openmetadata/connectors/hive.md @@ -43,10 +43,11 @@ pip install 'openmetadata-ingestion[hive]' {% endcode %} 1. **service\_name** - Service Name for this Hive cluster. If you added the Hive cluster through OpenMetadata UI, make sure the service name matches the same. -2. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata -3. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you the newly ingested data. -4. **data\_profiler\_offset** - Specify offset. -5. **data\_profiler\_limit** - Specify limit. +2. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +3. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. +4. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you the newly ingested data. +5. **data\_profiler\_offset** - Specify offset. +6. **data\_profiler\_limit** - Specify limit. ## Publish to OpenMetadata diff --git a/docs/openmetadata/connectors/mssql.md b/docs/openmetadata/connectors/mssql.md index 083e4e1154d..eea2ca2d00c 100644 --- a/docs/openmetadata/connectors/mssql.md +++ b/docs/openmetadata/connectors/mssql.md @@ -42,8 +42,11 @@ metadata ingest -c ./examples/workflows/mssql.json "query": "select top 50 * from {}.{}", "username": "sa", "password": "test!Password", - "filter_pattern": { - "excludes": ["catalog_test.*"] + "table_filter_pattern": { + "excludes": ["demo.*","orders.*"] + }, + "schema_filter_pattern": { + "excludes": ["information_schema.*"] } } }, @@ -55,8 +58,9 @@ metadata ingest -c ./examples/workflows/mssql.json 2. **password** - the password for the mssql username. 3. **service\_name** - Service Name for this mssql cluster. If you added the mssql cluster through OpenMetadata UI, make sure the service name matches the same. 4. **host\_port** - Hostname and Port number where the service is being initialized. -5. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata -6. **database** - Database name from where data is to be fetched from. +5. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +6. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. +7. **database** - Database name from where data is to be fetched from. ## Publish to OpenMetadata @@ -76,8 +80,11 @@ Add `metadata-rest` sink along with `metadata-server` config "query": "select top 50 * from {}.{}", "username": "sa", "password": "test!Password", - "filter_pattern": { - "excludes": ["catalog_test.*"] + "table_filter_pattern": { + "excludes": ["catalog_test.*","orders.*"] + }, + "schema_filter_pattern": { + "excludes": ["information_schema.*"] } } }, diff --git a/docs/openmetadata/connectors/mysql.md b/docs/openmetadata/connectors/mysql.md index 91094451b73..d235f673c6d 100644 --- a/docs/openmetadata/connectors/mysql.md +++ b/docs/openmetadata/connectors/mysql.md @@ -43,8 +43,11 @@ metadata ingest -c ./examples/workflows/mysql.json "data_profiler_enabled": "true", "data_profiler_offset": "0", "data_profiler_limit": "50000", - "filter_pattern": { - "excludes": ["mysql.*", "information_schema.*", "performance_schema.*", "sys.*"] + "table_filter_pattern": { + "excludes": ["demo.*","orders.*"] + }, + "schema_filter_pattern": { + "excludes": ["information_schema.*"] } } }, @@ -55,10 +58,11 @@ metadata ingest -c ./examples/workflows/mysql.json 1. **username** - pass the MySQL username. We recommend creating a user with read-only permissions to all the databases in your MySQL installation 2. **password** - password for the username 3. **service\_name** - Service Name for this MySQL cluster. If you added MySQL cluster through OpenMetadata UI, make sure the service name matches the same. -4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata -5. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you the newly ingested data. -6. **data\_profiler\_offset** - Specify offset. -7. **data\_profiler\_limit** - Specify limit. +4. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +5. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. +6. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you the newly ingested data. +7. **data\_profiler\_offset** - Specify offset. +8. **data\_profiler\_limit** - Specify limit. ## Publish to OpenMetadata @@ -79,7 +83,10 @@ Add `metadata-rest` sink along with `metadata-server` config "data_profiler_enabled": "true", "data_profiler_offset": "0", "data_profiler_limit": "50000", - "filter_pattern": { + "table_filter_pattern": { + "excludes": ["demo.*","orders.*"] + }, + "schema_filter_pattern": { "excludes": ["mysql.*", "information_schema.*", "performance_schema.*", "sys.*"] } } diff --git a/docs/openmetadata/connectors/oracle.md b/docs/openmetadata/connectors/oracle.md index 7a23f8eda2f..64672141cfb 100644 --- a/docs/openmetadata/connectors/oracle.md +++ b/docs/openmetadata/connectors/oracle.md @@ -48,7 +48,8 @@ pip install 'openmetadata-ingestion[oracle]' 3. **host\_port** - Host Port where Oracle Instance is initiated 4. **service\_name** - Service Name for this Oracle cluster. If you added Oracle cluster through OpenMetadata UI, make sure the service name matches the same. 5. **oracle\_service\_name -** Oracle Service Name (TNS alias) -6. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata +6. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +7. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. ## Publish to OpenMetadata diff --git a/docs/openmetadata/connectors/postgres.md b/docs/openmetadata/connectors/postgres.md index fccb470dbaf..93f0525ea82 100644 --- a/docs/openmetadata/connectors/postgres.md +++ b/docs/openmetadata/connectors/postgres.md @@ -53,11 +53,12 @@ metadata ingest -c ./examples/workflows/postgres.json 1. **username** - pass the Postgres username. 2. **password** - the password for the Postgres username. 3. **service\_name** - Service Name for this Postgres cluster. If you added the Postgres cluster through OpenMetadata UI, make sure the service name matches the same. -4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata. -5. **database -** Database name from where data is to be fetched. -6. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you the newly ingested data. -7. **data\_profiler\_offset** - Specify offset. -8. **data\_profiler\_limit** - Specify limit. +4. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. +5. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +6. **database -** Database name from where data is to be fetched. +7. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you the newly ingested data. +8. **data\_profiler\_offset** - Specify offset. +9. **data\_profiler\_limit** - Specify limit. ## Publish to OpenMetadata diff --git a/docs/openmetadata/connectors/presto.md b/docs/openmetadata/connectors/presto.md index 3bcffdb0d74..65ab0cebd9c 100644 --- a/docs/openmetadata/connectors/presto.md +++ b/docs/openmetadata/connectors/presto.md @@ -46,7 +46,8 @@ metadata ingest -c ./examples/workflows/presto.json 2. **password** - password for the username 3. **host\_port** - host and port of the Presto cluster 4. **service\_name** - Service Name for this Presto cluster. If you added the Presto cluster through OpenMetadata UI, make sure the service name matches the same. -5. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata +5. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +6. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. ## Publish to OpenMetadata diff --git a/docs/openmetadata/connectors/redshift-usage.md b/docs/openmetadata/connectors/redshift-usage.md index ed538343cbb..2d03b03d203 100644 --- a/docs/openmetadata/connectors/redshift-usage.md +++ b/docs/openmetadata/connectors/redshift-usage.md @@ -52,7 +52,8 @@ metadata ingest -c ./examples/workflows/redshift_usage.json 1. **username** - pass the Redshift username. We recommend creating a user with read-only permissions to all the databases in your Redshift installation 2. **password** - password for the username 3. **service\_name** - Service Name for this Redshift cluster. If you added the Redshift cluster through OpenMetadata UI, make sure the service name matches the same. -4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata +4. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +5. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. ## Publish to OpenMetadata diff --git a/docs/openmetadata/connectors/redshift.md b/docs/openmetadata/connectors/redshift.md index 4db6aae4769..157ad726329 100644 --- a/docs/openmetadata/connectors/redshift.md +++ b/docs/openmetadata/connectors/redshift.md @@ -42,8 +42,11 @@ metadata ingest -c ./examples/workflows/redshift.json "data_profiler_enabled": "true", "data_profiler_offset": "0", "data_profiler_limit": "50000", - "filter_pattern": { - "excludes": ["information_schema.*", "[\\w]*event_vw.*"] + "table_filter_pattern": { + "excludes": ["demo.*","orders.*"] + }, + "schema_filter_pattern": { + "excludes": ["information_schema.*"] } }, ... @@ -53,7 +56,8 @@ metadata ingest -c ./examples/workflows/redshift.json 1. **username** - pass the Redshift username. 2. **password** - the password for the Redshift username. 3. **service\_name** - Service Name for this Redshift cluster. If you added the Redshift cluster through OpenMetadata UI, make sure the service name matches the same. -4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata. +4. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. +5. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. 5. **database -** Database name from where data is to be fetched.‌ 6. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you the newly ingested data. 7. **data\_profiler\_offset** - Specify offset. @@ -79,8 +83,11 @@ Add `metadata-rest` sink along with `metadata-server` config "data_profiler_enabled": "true", "data_profiler_offset": "0", "data_profiler_limit": "50000", - "filter_pattern": { - "excludes": ["information_schema.*", "[\\w]*event_vw.*"] + "table_filter_pattern": { + "excludes": ["demo.*","orders.*"] + }, + "schema_filter_pattern": { + "excludes": ["information_schema.*"] } }, "sink": { diff --git a/docs/openmetadata/connectors/salesforce.md b/docs/openmetadata/connectors/salesforce.md index 29aaf13def2..01c04708979 100644 --- a/docs/openmetadata/connectors/salesforce.md +++ b/docs/openmetadata/connectors/salesforce.md @@ -53,7 +53,8 @@ metadata ingest -c ./examples/workflows/salesforce.json 3. **security\_token** - pass the security token. 4. **sobject\_name** - pass the salesforce object name. 5. **service\_name** - Service Name for this Salesforce cluster. If you added Salesforce cluster through OpenMetadata UI, make sure the service name matches the same. -6. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata +6. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +7. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. ## Publish to OpenMetadata diff --git a/docs/openmetadata/connectors/snowflake-usage.md b/docs/openmetadata/connectors/snowflake-usage.md index db3b1bfa530..37c552082dc 100644 --- a/docs/openmetadata/connectors/snowflake-usage.md +++ b/docs/openmetadata/connectors/snowflake-usage.md @@ -51,8 +51,9 @@ metadata ingest -c ./examples/workflows/snowflake_usage.json 1. **username** - pass the Snowflake username. 2. **password** - the password for the Snowflake username. 3. **service\_name** - Service Name for this Snowflake cluster. If you added the Snowflake cluster through OpenMetadata UI, make sure the service name matches the same. -4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata. -5. **database -** Database name from where data is to be fetched. +4. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. +5. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +6. **database -** Database name from where data is to be fetched. ### Publish to OpenMetadata diff --git a/docs/openmetadata/connectors/snowflake.md b/docs/openmetadata/connectors/snowflake.md index c7fd82bff32..0b6ca9f077d 100644 --- a/docs/openmetadata/connectors/snowflake.md +++ b/docs/openmetadata/connectors/snowflake.md @@ -45,10 +45,11 @@ metadata ingest -c ./examples/workflows/snowflake.json "data_profiler_enabled": "true", "data_profiler_offset": "0", "data_profiler_limit": "50000", - "filter_pattern": { - "excludes": [ - "tpcds_sf100tcl" - ] + "table_filter_pattern": { + "excludes": ["demo.*","orders.*"] + }, + "schema_filter_pattern": { + "excludes": ["information_schema.*"] } } }, @@ -58,11 +59,12 @@ metadata ingest -c ./examples/workflows/snowflake.json 1. **username** - pass the Snowflake username. 2. **password** - the password for the Snowflake username. 3. **service\_name** - Service Name for this Snowflake cluster. If you added the Snowflake cluster through OpenMetadata UI, make sure the service name matches the same. -4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata. -5. **database -** Database name from where data is to be fetched. -6. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you with the newly ingested data. -7. **data\_profiler\_offset** - Specify offset. -8. **data\_profiler\_limit** - Specify limit. +4. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. +5. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +6. **database -** Database name from where data is to be fetched. +7. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you with the newly ingested data. +8. **data\_profiler\_offset** - Specify offset. +9. **data\_profiler\_limit** - Specify limit. ### SSO Configuration @@ -136,10 +138,11 @@ Add `metadata-rest` sink along with `metadata-server` config "data_profiler_enabled": "true", "data_profiler_offset": "0", "data_profiler_limit": "50000", - "filter_pattern": { - "excludes": [ - "tpcds_sf100tcl" - ] + "table_filter_pattern": { + "excludes": ["demo.*","orders.*"] + }, + "schema_filter_pattern": { + "excludes": ["information_schema.*"] } } }, diff --git a/docs/openmetadata/connectors/trino.md b/docs/openmetadata/connectors/trino.md index 9f870e4b630..f096a10ebfe 100644 --- a/docs/openmetadata/connectors/trino.md +++ b/docs/openmetadata/connectors/trino.md @@ -47,7 +47,8 @@ metadata ingest -c ./examples/workflows/trino.json 2. **password** - password for the username 3. **host\_port** - host and port of the Trino cluster 4. **service\_name** - Service Name for this Trino cluster. If you added the Trino cluster through OpenMetadata UI, make sure the service name matches the same. -5. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata +5. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +6. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. ## Publish to OpenMetadata diff --git a/docs/openmetadata/connectors/vertica.md b/docs/openmetadata/connectors/vertica.md index 638f92c67c0..3789cece2c6 100644 --- a/docs/openmetadata/connectors/vertica.md +++ b/docs/openmetadata/connectors/vertica.md @@ -40,8 +40,11 @@ metadata ingest -c ./examples/workflows/vertica.json "password": "openmetadata_password", "database": "openmetadata_db", "service_name": "local_vertica", - "filter_pattern": { - "excludes": [] + "table_filter_pattern": { + "excludes": ["demo.*","orders.*"] + }, + "schema_filter_pattern": { + "excludes": ["information_schema.*"] } } }, @@ -52,7 +55,8 @@ metadata ingest -c ./examples/workflows/vertica.json 1. \*\*username \*\*- pass the Vertica username. 2. **password** - password for the username. 3. **service\_name** - Service Name for this Vertica cluster. If you added Vertica cluster through OpenMetadata UI, make sure the service name matches the same. -4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata +4. **table\_filter\_pattern** - It contains includes, excludes options to choose which pattern of tables you want to ingest into OpenMetadata. +5. **schema\_filter\_pattern** - It contains includes, excludes options to choose which pattern of schemas you want to ingest into OpenMetadata. ### Publish to OpenMetadata @@ -70,8 +74,11 @@ Add `metadata-rest` sink along with `metadata-server` config "password": "openmetadata_password", "database": "openmetadata_db", "service_name": "local_vertica", - "filter_pattern": { - "excludes": [] + "table_filter_pattern": { + "excludes": ["demo.*","orders.*"] + }, + "schema_filter_pattern": { + "excludes": ["information_schema.*"] } } }, diff --git a/ingestion/tests/unit/mysql_test.json b/ingestion/tests/unit/mysql_test.json index 0cbbeed2192..ad75fc11125 100644 --- a/ingestion/tests/unit/mysql_test.json +++ b/ingestion/tests/unit/mysql_test.json @@ -6,7 +6,7 @@ "password": "openmetadata_password", "database": "openmetadata_db", "service_name": "local_mysql_test", - "filter_pattern": { + "schema_filter_pattern": { "excludes": ["mysql.*", "information_schema.*", "performance_schema.*", "sys.*"] } }