mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-12-25 14:38:29 +00:00
Missing docs added (#15763)
This commit is contained in:
Shilpa Vernekar
GitHub
parent
dd53f45cc6
commit
f28214e0c0
@ -0,0 +1,413 @@
|
||||
---
|
||||
title: Database Filter Patterns
|
||||
slug: /connectors/ingestion/workflows/metadata/filter-patterns/database
|
||||
---
|
||||
|
||||
# Database Filter Patterns
|
||||
|
||||
|
||||
## Configuring Filters
|
||||
|
||||
One can configure the metadata ingestion filter for database source using four configuration fields which are `Database Filter Pattern`,
|
||||
`Schema Filter Pattern`, `Table Filter Pattern` & `Use FQN For Filtering`. In this document we will learn about each field in detail
|
||||
along with many examples.
|
||||
|
||||
### Configuring Filters via UI
|
||||
|
||||
Filters can be configured in UI while adding an ingestion pipeline through `Add Metadata Ingestion` page.
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/features/ingestion/workflows/metadata/filter-patterns/database-filter-patterns.webp"
|
||||
alt="Database Filter Pattern Fields"
|
||||
caption="Database Filter Pattern Fields"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI
|
||||
|
||||
Filters can be configured in CLI in connection configuration within `source.sourceConfig.config` field as described below.
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- database1
|
||||
- database2
|
||||
excludes:
|
||||
- database3
|
||||
- database4
|
||||
schemaFilterPattern:
|
||||
includes:
|
||||
- schema1
|
||||
- schema2
|
||||
excludes:
|
||||
- schema3
|
||||
- schema4
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- table1
|
||||
- table2
|
||||
excludes:
|
||||
- table3
|
||||
- table4
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Use FQN For Filtering
|
||||
|
||||
This flag set when you want to apply the filter on fully qualified name (e.g. service_name.db_name.schema_name.table_name)
|
||||
instead of applying the filter to raw name of entity (e.g. table_name). This Flag is useful in scenario when you have schema
|
||||
with same name in different databases or table with same name in different schemas, and you want to filter out one of them. This will be explained further in detail in this document.
|
||||
|
||||
|
||||
### Database Filter Pattern
|
||||
|
||||
Database filter patterns to control whether to include database as part of metadata ingestion.
|
||||
- **Include**: Explicitly include databases by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all databases with names matching one or more of the supplied regular expressions. All other databases will be excluded.
|
||||
- **Exclude**: Explicitly exclude databases by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all databases with names matching one or more of the supplied regular expressions. All other databases will be included.
|
||||
|
||||
#### Example 1
|
||||
|
||||
```yaml
|
||||
Snowflake_Prod # Snowflake Service Name
|
||||
│
|
||||
└─── SNOWFLAKE # DB Name
|
||||
│
|
||||
└─── SNOWFLAKE_SAMPLE_DATA # DB Name
|
||||
│
|
||||
└─── TEST_SNOWFLAKEDB # DB Name
|
||||
│
|
||||
└─── DUMMY_DB # DB Name
|
||||
│
|
||||
└─── ECOMMERCE # DB Name
|
||||
```
|
||||
|
||||
Let's say we want to ingest metadata from a snowflake instance which contains multiple databases as described above.
|
||||
In this example we want to ingest all databases which contains `SNOWFLAKE` in name, then the filter pattern
|
||||
applied would be `.*SNOWFLAKE.*` in the include field. This will result in ingestion of database `SNOWFLAKE`, `SNOWFLAKE_SAMPLE_DATA`
|
||||
and `TEST_SNOWFLAKEDB`.
|
||||
|
||||
### Configuring Filters via UI for Example 1
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-1.webp"
|
||||
alt="Database Filter Pattern Example 1"
|
||||
caption="Database Filter Pattern Example 1"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 1
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- .*SNOWFLAKE.*
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Example 2
|
||||
|
||||
In this example we want to ingest all databases which starts with `SNOWFLAKE` in name, then the filter pattern
|
||||
applied would be `^SNOWFLAKE.*` in the include field. This will result in ingestion of database `SNOWFLAKE` & `SNOWFLAKE_SAMPLE_DATA`.
|
||||
|
||||
### Configuring Filters via UI for Example 2
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-2.webp"
|
||||
alt="Database Filter Pattern Example 2"
|
||||
caption="Database Filter Pattern Example 2"
|
||||
/%}
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 2
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- ^SNOWFLAKE.*
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Example 3
|
||||
|
||||
In this example we want to ingest all databases for which the name starts with `SNOWFLAKE` OR ends with `DB` , then the filter pattern
|
||||
applied would be `^SNOWFLAKE` & `DB$` in the include field. This will result in ingestion of database `SNOWFLAKE`, `SNOWFLAKE_SAMPLE_DATA`, `TEST_SNOWFLAKEDB` & `DUMMY_DB`.
|
||||
|
||||
### Configuring Filters via UI for Example 3
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-3.webp"
|
||||
alt="Database Filter Pattern Example 3"
|
||||
caption="Database Filter Pattern Example 3"
|
||||
/%}
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 3
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- ^SNOWFLAKE.*
|
||||
- .*DB$
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Example 4
|
||||
|
||||
In this example we want to ingest only the `SNOWFLAKE` database then the filter pattern applied would be `^SNOWFLAKE$`.
|
||||
|
||||
### Configuring Filters via UI for Example 4
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-4.webp"
|
||||
alt="Database Filter Pattern Example 4"
|
||||
caption="Database Filter Pattern Example 4"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 4
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- ^SNOWFLAKE$
|
||||
```
|
||||
|
||||
|
||||
### Schema Filter Pattern
|
||||
|
||||
Schema filter patterns are used to control whether to include schemas as part of metadata ingestion.
|
||||
- **Include**: Explicitly include schemas by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all schemas with names matching one or more of the supplied regular expressions. All other schemas will be excluded.
|
||||
- **Exclude**: Explicitly exclude schemas by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all schemas with names matching one or more of the supplied regular expressions. All other schemas will be included.
|
||||
|
||||
#### Example 1
|
||||
|
||||
```yaml
|
||||
Snowflake_Prod # Snowflake Service Name
|
||||
│
|
||||
└─── SNOWFLAKE # DB Name
|
||||
│ │
|
||||
│ └─── PUBLIC # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF1 # Schema Name
|
||||
│ │
|
||||
│ └─── INFORMATION_SCHEMA # Schema Name
|
||||
│
|
||||
└─── SNOWFLAKE_SAMPLE_DATA # DB Name
|
||||
│ │
|
||||
│ └─── PUBLIC # Schema Name
|
||||
│ │
|
||||
│ └─── INFORMATION_SCHEMA # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF1 # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF10 # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF100 # Schema Name
|
||||
```
|
||||
|
||||
In this example we want to ingest all schema within any database with name `PUBLIC`, then the schema filter pattern
|
||||
applied would be `^PUBLIC$` in the include field. This will result in ingestion of schemas `SNOWFLAKE.PUBLIC` & `SNOWFLAKE_SAMPLE_DATA.PUBLIC`
|
||||
|
||||
|
||||
### Configuring Filters via UI for Example 1
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-1.webp"
|
||||
alt="Schema Filter Pattern Example 1"
|
||||
caption="Schema Filter Pattern Example 1"
|
||||
/%}
|
||||
|
||||
### Configuring Filters via CLI for Example 1
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
schemaFilterPattern:
|
||||
includes:
|
||||
- ^PUBLIC$
|
||||
```
|
||||
|
||||
|
||||
|
||||
#### Example 2
|
||||
|
||||
|
||||
In this example we want to ingest all schema within any database except schema with name `PUBLIC` available in `SNOWFLAKE_SAMPLE_DATA`.
|
||||
Notice that we have two schemas available with name `PUBLIC` one is available in database `SNOWFLAKE_SAMPLE_DATA.PUBLIC` and other is `SNOWFLAKE.PUBLIC`. As per the constraint of this example all the schemas including `SNOWFLAKE.PUBLIC` but we need to skip `SNOWFLAKE_SAMPLE_DATA.PUBLIC`. to do that we will need to set `useFqnForFiltering` flag to true by doing this the filter pattern will be applied to fully qualified name instead of raw table name. A fully qualified name(FQN) of schema is combination of service name, database name & schema name joined with `.`. In this example fully qualified name of the `SNOWFLAKE_SAMPLE_DATA.PUBLIC` schema will be `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC`, so we will need to apply an exclude filter pattern `^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.PUBLIC$` and set `useFqnForFiltering` to true.
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via UI for Example 2
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-2.webp"
|
||||
alt="Schema Filter Pattern Example 2"
|
||||
caption="Schema Filter Pattern Example 2"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 2
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
schemaFilterPattern:
|
||||
excludes:
|
||||
- ^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.PUBLIC$
|
||||
```
|
||||
|
||||
|
||||
#### Example 3
|
||||
|
||||
In this example we want to ingest `SNOWFLAKE.PUBLIC` & all the schemas in `SNOWFLAKE_SAMPLE_DATA` that starts with `TPCH_` i.e `SNOWFLAKE_SAMPLE_DATA.TPCH_1`, `SNOWFLAKE_SAMPLE_DATA.TPCH_10` & `SNOWFLAKE_SAMPLE_DATA.TPCH_100`. To achieve this an include schema filter will be applied with pattern `^Snowflake_Prod\.SNOWFLAKE\.PUBLIC$` & `^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.TPCH_.*`, we need to set `useFqnForFiltering` as true as we want to apply filter on FQN.
|
||||
|
||||
|
||||
### Configuring Filters via UI for Example 3
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-3.webp"
|
||||
alt="Schema Filter Pattern Example 3"
|
||||
caption="Schema Filter Pattern Example 3"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 3
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
schemaFilterPattern:
|
||||
includes:
|
||||
- ^Snowflake_Prod\.SNOWFLAKE\.PUBLIC$
|
||||
- ^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.TPCH_.*
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Table Filter Pattern
|
||||
|
||||
Table filter patterns are used to control whether or not to include tables as part of metadata ingestion.
|
||||
- **Include**: Explicitly include tables by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all tables with names matching one or more of the supplied regular expressions. All other tables will be excluded.
|
||||
- **Exclude**: Explicitly exclude tables by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all tables with names matching one or more of the supplied regular expressions. All other tables will be included.
|
||||
|
||||
|
||||
#### Example 1
|
||||
|
||||
```yaml
|
||||
Snowflake_Prod # Snowflake Service Name
|
||||
│
|
||||
└─── SNOWFLAKE_SAMPLE_DATA # DB Name
|
||||
│ │
|
||||
│ └─── PUBLIC # Schema Name
|
||||
│ │ │
|
||||
│ │ └─── CUSTOMER # Table Name
|
||||
│ │ │
|
||||
│ │ └─── CUSTOMER_ADDRESS # Table Name
|
||||
│ │ │
|
||||
│ │ └─── CUSTOMER_DEMOGRAPHICS # Table Name
|
||||
│ │ │
|
||||
│ │ └─── CALL_CENTER # Table Name
|
||||
│ │
|
||||
│ └─── INFORMATION # Schema Name
|
||||
│ │
|
||||
│ └─── ORDERS # Table Name
|
||||
│ │
|
||||
│ └─── REGION # Table Name
|
||||
│ │
|
||||
│ └─── CUSTOMER # Table Name
|
||||
│
|
||||
└─── SNOWFLAKE # DB Name
|
||||
│
|
||||
└─── PUBLIC # Schema Name
|
||||
│
|
||||
└─── CUSTOMER # Table Name
|
||||
```
|
||||
|
||||
#### Example 1
|
||||
|
||||
In this example we want to ingest table with name `CUSTOMER` within any schema and database. In this case we need to apply include table filter pattern `^CUSTOMER$`. This will result in ingestion of tables `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC.CUSTOMER`, `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.INFORMATION.CUSTOMER` & `Snowflake_Prod.SNOWFLAKE.PUBLIC.CUSTOMER`
|
||||
|
||||
### Configuring Filters via UI for Example 1
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/features/ingestion/workflows/metadata/filter-patterns/table-filter-example-1.webp"
|
||||
alt="Table Filter Pattern Example 1"
|
||||
caption="Table Filter Pattern Example 1"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 1
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- ^CUSTOMER$
|
||||
```
|
||||
|
||||
|
||||
#### Example 2
|
||||
|
||||
In this example we want to ingest table with name `CUSTOMER` within `PUBLIC` schema of any database. In this case we need to apply include table filter pattern `.*\.PUBLIC\.CUSTOMER$` this will also require to set the `useFqnForFiltering` flag as true as we want to apply filter on FQN. This will result in ingestion of tables `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC.CUSTOMER` & `Snowflake_Prod.SNOWFLAKE.PUBLIC.CUSTOMER`
|
||||
|
||||
### Configuring Filters via UI for Example 2
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/features/ingestion/workflows/metadata/filter-patterns/table-filter-example-2.webp"
|
||||
alt="Table Filter Pattern Example 2"
|
||||
caption="Table Filter Pattern Example 2"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 2
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- .*\.PUBLIC\.CUSTOMER$
|
||||
```
|
||||
@ -0,0 +1,26 @@
|
||||
---
|
||||
title: Metadata Ingestion Filter Patterns
|
||||
slug: /connectors/ingestion/workflows/metadata/filter-patterns
|
||||
---
|
||||
|
||||
# Metadata Ingestion Filter Patterns
|
||||
|
||||
The ingestion filter patterns are very useful when you have a lot of metadata available in your data source but
|
||||
some metadata might not be useful or relevant to produce any insights or discover data for ex. you might want to
|
||||
filter out the log tables while ingesting metadata.
|
||||
|
||||
Configuring these metadata filters with OpenMetadata is very easy, which uses regex for matching and filtering the metadata.
|
||||
Following documents will guide you on how to configure filters based on the type of data source
|
||||
|
||||
{%inlineCalloutContainer%}
|
||||
|
||||
{%inlineCallout
|
||||
bold="Database Filter Patterns"
|
||||
icon="cable"
|
||||
href="/connectors/ingestion/workflows/metadata/filter-patterns/database" %}
|
||||
Learn more about how to configure filters for database sources.
|
||||
{%/inlineCallout%}
|
||||
|
||||
{%/inlineCalloutContainer%}
|
||||
|
||||
|
||||
@ -54,7 +54,7 @@ caption="Landing Page Announcement Display (Top Right)"
|
||||
Furthermore, users can react with emojis and reply to the announcements from both the Activity Feed in the homepage and from the data asset page.
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/how-to-guides/user-guide-for-data-stewards/react.png"
|
||||
src="/images/v1.1/how-to-guides/user-guide-for-data-stewards/react.webp"
|
||||
alt="Reply or React to an Announcement"
|
||||
caption="Reply or React to an Announcement"
|
||||
/%}
|
||||
|
||||
@ -0,0 +1,32 @@
|
||||
---
|
||||
title: Conversation Threads
|
||||
slug: /how-to-guides/data-collaboration/conversation
|
||||
---
|
||||
|
||||
# Conversation Threads
|
||||
|
||||
As part of Data Collaboration feature of OpenMetadata, Conversation Threads were one of the earliest features introduced to enable users to easily ask any questions that they might have around a data asset. In the 0.9.0 release of OpenMetadata, we introduced the ability to reply and create entire conversation
|
||||
threads around all the various activities across any data asset. In release 0.11.0, we took it to the next level by extending conversation threads to Tasks and adding support for reactions with emojis.
|
||||
|
||||
Across OpenMetadata, users can start conversations around description, column description, tags, announcements, glossary terms of a data asset by clicking the chat icon as shown in the screen-shot below.
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/how-to-guides/collaboration/chat3.webp"
|
||||
alt="Start a Conversation"
|
||||
caption="Start a Conversation"
|
||||
/%}
|
||||
|
||||
Users can also reply or react with emojis for any Conversation by hovering over the conversation.
|
||||
{% image
|
||||
src="/images/v1.1/how-to-guides/collaboration/chat4.webp"
|
||||
alt="Reply or React to a Conversation"
|
||||
caption="Reply or React to a Conversation"
|
||||
/%}
|
||||
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
bold="Tasks"
|
||||
icon="MdArrowForward"
|
||||
href="/how-to-guides/data-collaboration/tasks"%}
|
||||
Add requests to create or update descriptions and tags.
|
||||
{%/inlineCallout%}
|
||||
@ -0,0 +1,31 @@
|
||||
---
|
||||
title: Create Tasks
|
||||
slug: /how-to-guides/data-collaboration/tasks
|
||||
---
|
||||
|
||||
# Create Tasks
|
||||
|
||||
Tasks are an extension to the Conversation Threads feature where users can create tasks to
|
||||
request to create or update description or tags for a data asset. Tasks are assgined to the owner of the data asset by default. If there are no owners, the task can be assigned to an appropriate user or team.
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/how-to-guides/collaboration/task.webp"
|
||||
alt="Start a Conversation"
|
||||
caption="Start a Conversation"
|
||||
/%}
|
||||
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
bold="How to Request for Description"
|
||||
icon="MdArrowForward"
|
||||
href="/how-to-guides/data-collaboration/request-description"%}
|
||||
Request for a description and discuss the same within OpenMetadata
|
||||
{%/inlineCallout%}
|
||||
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
bold="How to Request for Tags"
|
||||
icon="MdArrowForward"
|
||||
href="/how-to-guides/data-collaboration/request-tags"%}
|
||||
Request for tags and discuss about the same, all within OpenMetadata.
|
||||
{%/inlineCallout%}
|
||||
@ -54,7 +54,7 @@ caption="Landing Page Announcement Display (Top Right)"
|
||||
Furthermore, users can react with emojis and reply to the announcements from both the Activity Feed in the homepage and from the data asset page.
|
||||
|
||||
{% image
|
||||
src="/images/v1.1/how-to-guides/user-guide-for-data-stewards/react.png"
|
||||
src="/images/v1.1/how-to-guides/user-guide-for-data-stewards/react.webp"
|
||||
alt="Reply or React to an Announcement"
|
||||
caption="Reply or React to an Announcement"
|
||||
/%}
|
||||
|
||||
@ -0,0 +1,413 @@
|
||||
---
|
||||
title: Database Filter Patterns
|
||||
slug: /connectors/ingestion/workflows/metadata/filter-patterns/database
|
||||
---
|
||||
|
||||
# Database Filter Patterns
|
||||
|
||||
|
||||
## Configuring Filters
|
||||
|
||||
One can configure the metadata ingestion filter for database source using four configuration fields which are `Database Filter Pattern`,
|
||||
`Schema Filter Pattern`, `Table Filter Pattern` & `Use FQN For Filtering`. In this document we will learn about each field in detail
|
||||
along with many examples.
|
||||
|
||||
### Configuring Filters via UI
|
||||
|
||||
Filters can be configured in UI while adding an ingestion pipeline through `Add Metadata Ingestion` page.
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/features/ingestion/workflows/metadata/filter-patterns/database-filter-patterns.webp"
|
||||
alt="Database Filter Pattern Fields"
|
||||
caption="Database Filter Pattern Fields"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI
|
||||
|
||||
Filters can be configured in CLI in connection configuration within `source.sourceConfig.config` field as described below.
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- database1
|
||||
- database2
|
||||
excludes:
|
||||
- database3
|
||||
- database4
|
||||
schemaFilterPattern:
|
||||
includes:
|
||||
- schema1
|
||||
- schema2
|
||||
excludes:
|
||||
- schema3
|
||||
- schema4
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- table1
|
||||
- table2
|
||||
excludes:
|
||||
- table3
|
||||
- table4
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Use FQN For Filtering
|
||||
|
||||
This flag set when you want to apply the filter on fully qualified name (e.g service_name.db_name.schema_name.table_name)
|
||||
instead of applying the filter to raw name of entity (e.g table_name). This Flag is useful in scenario when you have schema
|
||||
with same name in different databases or table with same name in different schemas and you want to filter out one of them. This will be explained further in detail in this document.
|
||||
|
||||
|
||||
### Database Filter Pattern
|
||||
|
||||
Database filter patterns to control whether or not to include database as part of metadata ingestion.
|
||||
- **Include**: Explicitly include databases by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all databases with names matching one or more of the supplied regular expressions. All other databases will be excluded.
|
||||
- **Exclude**: Explicitly exclude databases by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all databases with names matching one or more of the supplied regular expressions. All other databases will be included.
|
||||
|
||||
#### Example 1
|
||||
|
||||
```yaml
|
||||
Snowflake_Prod # Snowflake Service Name
|
||||
│
|
||||
└─── SNOWFLAKE # DB Name
|
||||
│
|
||||
└─── SNOWFLAKE_SAMPLE_DATA # DB Name
|
||||
│
|
||||
└─── TEST_SNOWFLAKEDB # DB Name
|
||||
│
|
||||
└─── DUMMY_DB # DB Name
|
||||
│
|
||||
└─── ECOMMERCE # DB Name
|
||||
```
|
||||
|
||||
Let's say we want to ingest metadata from a snowflake instance which contains multiple databases as described above.
|
||||
In this example we want to ingest all databases which contains `SNOWFLAKE` in name, then the filter pattern
|
||||
applied would be `.*SNOWFLAKE.*` in the include field. This will result in ingestion of database `SNOWFLAKE`, `SNOWFLAKE_SAMPLE_DATA`
|
||||
and `TEST_SNOWFLAKEDB`.
|
||||
|
||||
### Configuring Filters via UI for Example 1
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-1.webp"
|
||||
alt="Database Filter Pattern Example 1"
|
||||
caption="Database Filter Pattern Example 1"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 1
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- .*SNOWFLAKE.*
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Example 2
|
||||
|
||||
In this example we want to ingest all databases which starts with `SNOWFLAKE` in name, then the filter pattern
|
||||
applied would be `^SNOWFLAKE.*` in the include field. This will result in ingestion of database `SNOWFLAKE` & `SNOWFLAKE_SAMPLE_DATA`.
|
||||
|
||||
### Configuring Filters via UI for Example 2
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-2.webp"
|
||||
alt="Database Filter Pattern Example 2"
|
||||
caption="Database Filter Pattern Example 2"
|
||||
/%}
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 2
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- ^SNOWFLAKE.*
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Example 3
|
||||
|
||||
In this example we want to ingest all databases for which the name starts with `SNOWFLAKE` OR ends with `DB` , then the filter pattern
|
||||
applied would be `^SNOWFLAKE` & `DB$` in the include field. This will result in ingestion of database `SNOWFLAKE`, `SNOWFLAKE_SAMPLE_DATA`, `TEST_SNOWFLAKEDB` & `DUMMY_DB`.
|
||||
|
||||
### Configuring Filters via UI for Example 3
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-3.webp"
|
||||
alt="Database Filter Pattern Example 3"
|
||||
caption="Database Filter Pattern Example 3"
|
||||
/%}
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 3
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- ^SNOWFLAKE.*
|
||||
- .*DB$
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Example 4
|
||||
|
||||
In this example we want to ingest only the `SNOWFLAKE` database then the filter pattern applied would be `^SNOWFLAKE$`.
|
||||
|
||||
### Configuring Filters via UI for Example 4
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-4.webp"
|
||||
alt="Database Filter Pattern Example 4"
|
||||
caption="Database Filter Pattern Example 4"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 4
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- ^SNOWFLAKE$
|
||||
```
|
||||
|
||||
|
||||
### Schema Filter Pattern
|
||||
|
||||
Schema filter patterns are used to control whether or not to include schemas as part of metadata ingestion.
|
||||
- **Include**: Explicitly include schemas by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all schemas with names matching one or more of the supplied regular expressions. All other schemas will be excluded.
|
||||
- **Exclude**: Explicitly exclude schemas by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all schemas with names matching one or more of the supplied regular expressions. All other schemas will be included.
|
||||
|
||||
#### Example 1
|
||||
|
||||
```yaml
|
||||
Snowflake_Prod # Snowflake Service Name
|
||||
│
|
||||
└─── SNOWFLAKE # DB Name
|
||||
│ │
|
||||
│ └─── PUBLIC # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF1 # Schema Name
|
||||
│ │
|
||||
│ └─── INFORMATION_SCHEMA # Schema Name
|
||||
│
|
||||
└─── SNOWFLAKE_SAMPLE_DATA # DB Name
|
||||
│ │
|
||||
│ └─── PUBLIC # Schema Name
|
||||
│ │
|
||||
│ └─── INFORMATION_SCHEMA # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF1 # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF10 # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF100 # Schema Name
|
||||
```
|
||||
|
||||
In this example we want to ingest all schema within any database with name `PUBLIC`, then the schema filter pattern
|
||||
applied would be `^PUBLIC$` in the include field. This will result in ingestion of schemas `SNOWFLAKE.PUBLIC` & `SNOWFLAKE_SAMPLE_DATA.PUBLIC`
|
||||
|
||||
|
||||
### Configuring Filters via UI for Example 1
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-1.webp"
|
||||
alt="Schema Filter Pattern Example 1"
|
||||
caption="Schema Filter Pattern Example 1"
|
||||
/%}
|
||||
|
||||
### Configuring Filters via CLI for Example 1
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
schemaFilterPattern:
|
||||
includes:
|
||||
- ^PUBLIC$
|
||||
```
|
||||
|
||||
|
||||
|
||||
#### Example 2
|
||||
|
||||
|
||||
In this example we want to ingest all schema within any database except schema with name `PUBLIC` available in `SNOWFLAKE_SAMPLE_DATA`.
|
||||
Notice that we have two schemas available with name `PUBLIC` one is available in database `SNOWFLAKE_SAMPLE_DATA.PUBLIC` and other is `SNOWFLAKE.PUBLIC`. As per the constraint of this example all the schemas including `SNOWFLAKE.PUBLIC` but we need to skip `SNOWFLAKE_SAMPLE_DATA.PUBLIC`. to do that we will need to set `useFqnForFiltering` flag to true by doing this the filter pattern will be applied to fully qualified name instead of raw table name. A fully qualified name(FQN) of schema is combination of service name, database name & schema name joined with `.`. In this example fully qualified name of the `SNOWFLAKE_SAMPLE_DATA.PUBLIC` schema will be `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC`, so we will need to apply a exclude filter pattern `^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.PUBLIC$` and set `useFqnForFiltering` to true.
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via UI for Example 2
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-2.webp"
|
||||
alt="Schema Filter Pattern Example 2"
|
||||
caption="Schema Filter Pattern Example 2"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 2
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
schemaFilterPattern:
|
||||
excludes:
|
||||
- ^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.PUBLIC$
|
||||
```
|
||||
|
||||
|
||||
#### Example 3
|
||||
|
||||
In this example we want to ingest `SNOWFLAKE.PUBLIC` & all the schemas in `SNOWFLAKE_SAMPLE_DATA` that starts with `TPCH_` i.e `SNOWFLAKE_SAMPLE_DATA.TPCH_1`, `SNOWFLAKE_SAMPLE_DATA.TPCH_10` & `SNOWFLAKE_SAMPLE_DATA.TPCH_100`. To achieve this an include schema filter will be applied with pattern `^Snowflake_Prod\.SNOWFLAKE\.PUBLIC$` & `^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.TPCH_.*`, we need to set `useFqnForFiltering` as true as we want to apply filter on FQN.
|
||||
|
||||
|
||||
### Configuring Filters via UI for Example 3
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-3.webp"
|
||||
alt="Schema Filter Pattern Example 3"
|
||||
caption="Schema Filter Pattern Example 3"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 3
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
schemaFilterPattern:
|
||||
includes:
|
||||
- ^Snowflake_Prod\.SNOWFLAKE\.PUBLIC$
|
||||
- ^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.TPCH_.*
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Table Filter Pattern
|
||||
|
||||
Table filter patterns are used to control whether or not to include tables as part of metadata ingestion.
|
||||
- **Include**: Explicitly include tables by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all tables with names matching one or more of the supplied regular expressions. All other tables will be excluded.
|
||||
- **Exclude**: Explicitly exclude tables by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all tables with names matching one or more of the supplied regular expressions. All other tables will be included.
|
||||
|
||||
|
||||
#### Example 1
|
||||
|
||||
```yaml
|
||||
Snowflake_Prod # Snowflake Service Name
|
||||
│
|
||||
└─── SNOWFLAKE_SAMPLE_DATA # DB Name
|
||||
│ │
|
||||
│ └─── PUBLIC # Schema Name
|
||||
│ │ │
|
||||
│ │ └─── CUSTOMER # Table Name
|
||||
│ │ │
|
||||
│ │ └─── CUSTOMER_ADDRESS # Table Name
|
||||
│ │ │
|
||||
│ │ └─── CUSTOMER_DEMOGRAPHICS # Table Name
|
||||
│ │ │
|
||||
│ │ └─── CALL_CENTER # Table Name
|
||||
│ │
|
||||
│ └─── INFORMATION # Schema Name
|
||||
│ │
|
||||
│ └─── ORDERS # Table Name
|
||||
│ │
|
||||
│ └─── REGION # Table Name
|
||||
│ │
|
||||
│ └─── CUSTOMER # Table Name
|
||||
│
|
||||
└─── SNOWFLAKE # DB Name
|
||||
│
|
||||
└─── PUBLIC # Schema Name
|
||||
│
|
||||
└─── CUSTOMER # Table Name
|
||||
```
|
||||
|
||||
#### Example 1
|
||||
|
||||
In this example we want to ingest table with name `CUSTOMER` within any schema and database. In this case we need to apply include table filter pattern `^CUSTOMER$`. This will result in ingestion of tables `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC.CUSTOMER`, `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.INFORMATION.CUSTOMER` & `Snowflake_Prod.SNOWFLAKE.PUBLIC.CUSTOMER`
|
||||
|
||||
### Configuring Filters via UI for Example 1
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/features/ingestion/workflows/metadata/filter-patterns/table-filter-example-1.webp"
|
||||
alt="Table Filter Pattern Example 1"
|
||||
caption="Table Filter Pattern Example 1"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 1
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- ^CUSTOMER$
|
||||
```
|
||||
|
||||
|
||||
#### Example 2
|
||||
|
||||
In this example we want to ingest table with name `CUSTOMER` within `PUBLIC` schema of any database. In this case we need to apply include table filter pattern `.*\.PUBLIC\.CUSTOMER$` this will also require to set the `useFqnForFiltering` flag as true as we want to apply filter on FQN. This will result in ingestion of tables `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC.CUSTOMER` & `Snowflake_Prod.SNOWFLAKE.PUBLIC.CUSTOMER`
|
||||
|
||||
### Configuring Filters via UI for Example 2
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/features/ingestion/workflows/metadata/filter-patterns/table-filter-example-2.webp"
|
||||
alt="Table Filter Pattern Example 2"
|
||||
caption="Table Filter Pattern Example 2"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 2
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- .*\.PUBLIC\.CUSTOMER$
|
||||
```
|
||||
@ -0,0 +1,26 @@
|
||||
---
|
||||
title: Metadata Ingestion Filter Patterns
|
||||
slug: /connectors/ingestion/workflows/metadata/filter-patterns
|
||||
---
|
||||
|
||||
# Metadata Ingestion Filter Patterns
|
||||
|
||||
The ingestion filter patterns are very useful when you have a lot of metadata available in your data source but
|
||||
some metadata might not be useful or relevant to produce any insights or discover data for ex. you might want to
|
||||
filter out the log tables while ingesting metadata.
|
||||
|
||||
Configuring these metadata filters with OpenMetadata is very easy, which uses regex for matching and filtering the metadata.
|
||||
Following documents will guide you on how to configure filters based on the type of data source
|
||||
|
||||
{%inlineCalloutContainer%}
|
||||
|
||||
{%inlineCallout
|
||||
bold="Database Filter Patterns"
|
||||
icon="cable"
|
||||
href="/connectors/ingestion/workflows/metadata/filter-patterns/database" %}
|
||||
Learn more about how to configure filters for database sources.
|
||||
{%/inlineCallout%}
|
||||
|
||||
{%/inlineCalloutContainer%}
|
||||
|
||||
|
||||
@ -54,7 +54,7 @@ caption="Landing Page Announcement Display (Top Right)"
|
||||
Furthermore, users can react with emojis and reply to the announcements from both the Activity Feed in the homepage and from the data asset page.
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/how-to-guides/user-guide-for-data-stewards/react.png"
|
||||
src="/images/v1.2/how-to-guides/user-guide-for-data-stewards/react.webp"
|
||||
alt="Reply or React to an Announcement"
|
||||
caption="Reply or React to an Announcement"
|
||||
/%}
|
||||
|
||||
@ -0,0 +1,32 @@
|
||||
---
|
||||
title: Conversation Threads
|
||||
slug: /how-to-guides/data-collaboration/conversation
|
||||
---
|
||||
|
||||
# Conversation Threads
|
||||
|
||||
As part of Data Collaboration feature of OpenMetadata, Conversation Threads were one of the earliest features introduced to enable users to easily ask any questions that they might have around a data asset. In the 0.9.0 release of OpenMetadata, we introduced the ability to reply and create entire conversation
|
||||
threads around all the various activities across any data asset. In release 0.11.0, we took it to the next level by extending conversation threads to Tasks and adding support for reactions with emojis.
|
||||
|
||||
Across OpenMetadata, users can start conversations around description, column description, tags, announcements, glossary terms of a data asset by clicking the chat icon as shown in the screen-shot below.
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/how-to-guides/collaboration/chat3.webp"
|
||||
alt="Start a Conversation"
|
||||
caption="Start a Conversation"
|
||||
/%}
|
||||
|
||||
Users can also reply or react with emojis for any Conversation by hovering over the conversation.
|
||||
{% image
|
||||
src="/images/v1.2/how-to-guides/collaboration/chat4.webp"
|
||||
alt="Reply or React to a Conversation"
|
||||
caption="Reply or React to a Conversation"
|
||||
/%}
|
||||
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
bold="Tasks"
|
||||
icon="MdArrowForward"
|
||||
href="/how-to-guides/data-collaboration/tasks"%}
|
||||
Add requests to create or update descriptions and tags.
|
||||
{%/inlineCallout%}
|
||||
@ -0,0 +1,31 @@
|
||||
---
|
||||
title: Create Tasks
|
||||
slug: /how-to-guides/data-collaboration/tasks
|
||||
---
|
||||
|
||||
# Create Tasks
|
||||
|
||||
Tasks are an extension to the Conversation Threads feature where users can create tasks to
|
||||
request to create or update description or tags for a data asset. Tasks are assgined to the owner of the data asset by default. If there are no owners, the task can be assigned to an appropriate user or team.
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/how-to-guides/collaboration/task.webp"
|
||||
alt="Start a Conversation"
|
||||
caption="Start a Conversation"
|
||||
/%}
|
||||
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
bold="How to Request for Description"
|
||||
icon="MdArrowForward"
|
||||
href="/how-to-guides/data-collaboration/request-description"%}
|
||||
Request for a description and discuss the same within OpenMetadata
|
||||
{%/inlineCallout%}
|
||||
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
bold="How to Request for Tags"
|
||||
icon="MdArrowForward"
|
||||
href="/how-to-guides/data-collaboration/request-tags"%}
|
||||
Request for tags and discuss about the same, all within OpenMetadata.
|
||||
{%/inlineCallout%}
|
||||
@ -54,7 +54,7 @@ caption="Landing Page Announcement Display (Top Right)"
|
||||
Furthermore, users can react with emojis and reply to the announcements from both the Activity Feed in the homepage and from the data asset page.
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/how-to-guides/user-guide-for-data-stewards/react.png"
|
||||
src="/images/v1.2/how-to-guides/user-guide-for-data-stewards/react.webp"
|
||||
alt="Reply or React to an Announcement"
|
||||
caption="Reply or React to an Announcement"
|
||||
/%}
|
||||
|
||||
@ -0,0 +1,413 @@
|
||||
---
|
||||
title: Database Filter Patterns
|
||||
slug: /connectors/ingestion/workflows/metadata/filter-patterns/database
|
||||
---
|
||||
|
||||
# Database Filter Patterns
|
||||
|
||||
|
||||
## Configuring Filters
|
||||
|
||||
One can configure the metadata ingestion filter for database source using four configuration fields which are `Database Filter Pattern`,
|
||||
`Schema Filter Pattern`, `Table Filter Pattern` & `Use FQN For Filtering`. In this document we will learn about each field in detail
|
||||
along with many examples.
|
||||
|
||||
### Configuring Filters via UI
|
||||
|
||||
Filters can be configured in UI while adding an ingestion pipeline through `Add Metadata Ingestion` page.
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/features/ingestion/workflows/metadata/filter-patterns/database-filter-patterns.webp"
|
||||
alt="Database Filter Pattern Fields"
|
||||
caption="Database Filter Pattern Fields"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI
|
||||
|
||||
Filters can be configured in CLI in connection configuration within `source.sourceConfig.config` field as described below.
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- database1
|
||||
- database2
|
||||
excludes:
|
||||
- database3
|
||||
- database4
|
||||
schemaFilterPattern:
|
||||
includes:
|
||||
- schema1
|
||||
- schema2
|
||||
excludes:
|
||||
- schema3
|
||||
- schema4
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- table1
|
||||
- table2
|
||||
excludes:
|
||||
- table3
|
||||
- table4
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Use FQN For Filtering
|
||||
|
||||
This flag set when you want to apply the filter on fully qualified name (e.g service_name.db_name.schema_name.table_name)
|
||||
instead of applying the filter to raw name of entity (e.g table_name). This Flag is useful in scenario when you have schema
|
||||
with same name in different databases or table with same name in different schemas and you want to filter out one of them. This will be explained further in detail in this document.
|
||||
|
||||
|
||||
### Database Filter Pattern
|
||||
|
||||
Database filter patterns to control whether or not to include database as part of metadata ingestion.
|
||||
- **Include**: Explicitly include databases by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all databases with names matching one or more of the supplied regular expressions. All other databases will be excluded.
|
||||
- **Exclude**: Explicitly exclude databases by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all databases with names matching one or more of the supplied regular expressions. All other databases will be included.
|
||||
|
||||
#### Example 1
|
||||
|
||||
```yaml
|
||||
Snowflake_Prod # Snowflake Service Name
|
||||
│
|
||||
└─── SNOWFLAKE # DB Name
|
||||
│
|
||||
└─── SNOWFLAKE_SAMPLE_DATA # DB Name
|
||||
│
|
||||
└─── TEST_SNOWFLAKEDB # DB Name
|
||||
│
|
||||
└─── DUMMY_DB # DB Name
|
||||
│
|
||||
└─── ECOMMERCE # DB Name
|
||||
```
|
||||
|
||||
Let's say we want to ingest metadata from a snowflake instance which contains multiple databases as described above.
|
||||
In this example we want to ingest all databases which contains `SNOWFLAKE` in name, then the filter pattern
|
||||
applied would be `.*SNOWFLAKE.*` in the include field. This will result in ingestion of database `SNOWFLAKE`, `SNOWFLAKE_SAMPLE_DATA`
|
||||
and `TEST_SNOWFLAKEDB`.
|
||||
|
||||
### Configuring Filters via UI for Example 1
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-1.webp"
|
||||
alt="Database Filter Pattern Example 1"
|
||||
caption="Database Filter Pattern Example 1"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 1
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- .*SNOWFLAKE.*
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Example 2
|
||||
|
||||
In this example we want to ingest all databases which starts with `SNOWFLAKE` in name, then the filter pattern
|
||||
applied would be `^SNOWFLAKE.*` in the include field. This will result in ingestion of database `SNOWFLAKE` & `SNOWFLAKE_SAMPLE_DATA`.
|
||||
|
||||
### Configuring Filters via UI for Example 2
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-2.webp"
|
||||
alt="Database Filter Pattern Example 2"
|
||||
caption="Database Filter Pattern Example 2"
|
||||
/%}
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 2
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- ^SNOWFLAKE.*
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Example 3
|
||||
|
||||
In this example we want to ingest all databases for which the name starts with `SNOWFLAKE` OR ends with `DB` , then the filter pattern
|
||||
applied would be `^SNOWFLAKE` & `DB$` in the include field. This will result in ingestion of database `SNOWFLAKE`, `SNOWFLAKE_SAMPLE_DATA`, `TEST_SNOWFLAKEDB` & `DUMMY_DB`.
|
||||
|
||||
### Configuring Filters via UI for Example 3
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-3.webp"
|
||||
alt="Database Filter Pattern Example 3"
|
||||
caption="Database Filter Pattern Example 3"
|
||||
/%}
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 3
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- ^SNOWFLAKE.*
|
||||
- .*DB$
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Example 4
|
||||
|
||||
In this example we want to ingest only the `SNOWFLAKE` database then the filter pattern applied would be `^SNOWFLAKE$`.
|
||||
|
||||
### Configuring Filters via UI for Example 4
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-4.webp"
|
||||
alt="Database Filter Pattern Example 4"
|
||||
caption="Database Filter Pattern Example 4"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 4
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- ^SNOWFLAKE$
|
||||
```
|
||||
|
||||
|
||||
### Schema Filter Pattern
|
||||
|
||||
Schema filter patterns are used to control whether or not to include schemas as part of metadata ingestion.
|
||||
- **Include**: Explicitly include schemas by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all schemas with names matching one or more of the supplied regular expressions. All other schemas will be excluded.
|
||||
- **Exclude**: Explicitly exclude schemas by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all schemas with names matching one or more of the supplied regular expressions. All other schemas will be included.
|
||||
|
||||
#### Example 1
|
||||
|
||||
```yaml
|
||||
Snowflake_Prod # Snowflake Service Name
|
||||
│
|
||||
└─── SNOWFLAKE # DB Name
|
||||
│ │
|
||||
│ └─── PUBLIC # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF1 # Schema Name
|
||||
│ │
|
||||
│ └─── INFORMATION_SCHEMA # Schema Name
|
||||
│
|
||||
└─── SNOWFLAKE_SAMPLE_DATA # DB Name
|
||||
│ │
|
||||
│ └─── PUBLIC # Schema Name
|
||||
│ │
|
||||
│ └─── INFORMATION_SCHEMA # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF1 # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF10 # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF100 # Schema Name
|
||||
```
|
||||
|
||||
In this example we want to ingest all schema within any database with name `PUBLIC`, then the schema filter pattern
|
||||
applied would be `^PUBLIC$` in the include field. This will result in ingestion of schemas `SNOWFLAKE.PUBLIC` & `SNOWFLAKE_SAMPLE_DATA.PUBLIC`
|
||||
|
||||
|
||||
### Configuring Filters via UI for Example 1
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-1.webp"
|
||||
alt="Schema Filter Pattern Example 1"
|
||||
caption="Schema Filter Pattern Example 1"
|
||||
/%}
|
||||
|
||||
### Configuring Filters via CLI for Example 1
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
schemaFilterPattern:
|
||||
includes:
|
||||
- ^PUBLIC$
|
||||
```
|
||||
|
||||
|
||||
|
||||
#### Example 2
|
||||
|
||||
|
||||
In this example we want to ingest all schema within any database except schema with name `PUBLIC` available in `SNOWFLAKE_SAMPLE_DATA`.
|
||||
Notice that we have two schemas available with name `PUBLIC` one is available in database `SNOWFLAKE_SAMPLE_DATA.PUBLIC` and other is `SNOWFLAKE.PUBLIC`. As per the constraint of this example all the schemas including `SNOWFLAKE.PUBLIC` but we need to skip `SNOWFLAKE_SAMPLE_DATA.PUBLIC`. to do that we will need to set `useFqnForFiltering` flag to true by doing this the filter pattern will be applied to fully qualified name instead of raw table name. A fully qualified name(FQN) of schema is combination of service name, database name & schema name joined with `.`. In this example fully qualified name of the `SNOWFLAKE_SAMPLE_DATA.PUBLIC` schema will be `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC`, so we will need to apply a exclude filter pattern `^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.PUBLIC$` and set `useFqnForFiltering` to true.
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via UI for Example 2
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-2.webp"
|
||||
alt="Schema Filter Pattern Example 2"
|
||||
caption="Schema Filter Pattern Example 2"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 2
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
schemaFilterPattern:
|
||||
excludes:
|
||||
- ^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.PUBLIC$
|
||||
```
|
||||
|
||||
|
||||
#### Example 3
|
||||
|
||||
In this example we want to ingest `SNOWFLAKE.PUBLIC` & all the schemas in `SNOWFLAKE_SAMPLE_DATA` that starts with `TPCH_` i.e `SNOWFLAKE_SAMPLE_DATA.TPCH_1`, `SNOWFLAKE_SAMPLE_DATA.TPCH_10` & `SNOWFLAKE_SAMPLE_DATA.TPCH_100`. To achieve this an include schema filter will be applied with pattern `^Snowflake_Prod\.SNOWFLAKE\.PUBLIC$` & `^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.TPCH_.*`, we need to set `useFqnForFiltering` as true as we want to apply filter on FQN.
|
||||
|
||||
|
||||
### Configuring Filters via UI for Example 3
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-3.webp"
|
||||
alt="Schema Filter Pattern Example 3"
|
||||
caption="Schema Filter Pattern Example 3"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 3
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
schemaFilterPattern:
|
||||
includes:
|
||||
- ^Snowflake_Prod\.SNOWFLAKE\.PUBLIC$
|
||||
- ^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.TPCH_.*
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Table Filter Pattern
|
||||
|
||||
Table filter patterns are used to control whether or not to include tables as part of metadata ingestion.
|
||||
- **Include**: Explicitly include tables by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all tables with names matching one or more of the supplied regular expressions. All other tables will be excluded.
|
||||
- **Exclude**: Explicitly exclude tables by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all tables with names matching one or more of the supplied regular expressions. All other tables will be included.
|
||||
|
||||
|
||||
#### Example 1
|
||||
|
||||
```yaml
|
||||
Snowflake_Prod # Snowflake Service Name
|
||||
│
|
||||
└─── SNOWFLAKE_SAMPLE_DATA # DB Name
|
||||
│ │
|
||||
│ └─── PUBLIC # Schema Name
|
||||
│ │ │
|
||||
│ │ └─── CUSTOMER # Table Name
|
||||
│ │ │
|
||||
│ │ └─── CUSTOMER_ADDRESS # Table Name
|
||||
│ │ │
|
||||
│ │ └─── CUSTOMER_DEMOGRAPHICS # Table Name
|
||||
│ │ │
|
||||
│ │ └─── CALL_CENTER # Table Name
|
||||
│ │
|
||||
│ └─── INFORMATION # Schema Name
|
||||
│ │
|
||||
│ └─── ORDERS # Table Name
|
||||
│ │
|
||||
│ └─── REGION # Table Name
|
||||
│ │
|
||||
│ └─── CUSTOMER # Table Name
|
||||
│
|
||||
└─── SNOWFLAKE # DB Name
|
||||
│
|
||||
└─── PUBLIC # Schema Name
|
||||
│
|
||||
└─── CUSTOMER # Table Name
|
||||
```
|
||||
|
||||
#### Example 1
|
||||
|
||||
In this example we want to ingest table with name `CUSTOMER` within any schema and database. In this case we need to apply include table filter pattern `^CUSTOMER$`. This will result in ingestion of tables `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC.CUSTOMER`, `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.INFORMATION.CUSTOMER` & `Snowflake_Prod.SNOWFLAKE.PUBLIC.CUSTOMER`
|
||||
|
||||
### Configuring Filters via UI for Example 1
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/features/ingestion/workflows/metadata/filter-patterns/table-filter-example-1.webp"
|
||||
alt="Table Filter Pattern Example 1"
|
||||
caption="Table Filter Pattern Example 1"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 1
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- ^CUSTOMER$
|
||||
```
|
||||
|
||||
|
||||
#### Example 2
|
||||
|
||||
In this example we want to ingest table with name `CUSTOMER` within `PUBLIC` schema of any database. In this case we need to apply include table filter pattern `.*\.PUBLIC\.CUSTOMER$` this will also require to set the `useFqnForFiltering` flag as true as we want to apply filter on FQN. This will result in ingestion of tables `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC.CUSTOMER` & `Snowflake_Prod.SNOWFLAKE.PUBLIC.CUSTOMER`
|
||||
|
||||
### Configuring Filters via UI for Example 2
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/features/ingestion/workflows/metadata/filter-patterns/table-filter-example-2.webp"
|
||||
alt="Table Filter Pattern Example 2"
|
||||
caption="Table Filter Pattern Example 2"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 2
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- .*\.PUBLIC\.CUSTOMER$
|
||||
```
|
||||
@ -0,0 +1,26 @@
|
||||
---
|
||||
title: Metadata Ingestion Filter Patterns
|
||||
slug: /connectors/ingestion/workflows/metadata/filter-patterns
|
||||
---
|
||||
|
||||
# Metadata Ingestion Filter Patterns
|
||||
|
||||
The ingestion filter patterns are very useful when you have a lot of metadata available in your data source but
|
||||
some metadata might not be useful or relevant to produce any insights or discover data for ex. you might want to
|
||||
filter out the log tables while ingesting metadata.
|
||||
|
||||
Configuring these metadata filters with OpenMetadata is very easy, which uses regex for matching and filtering the metadata.
|
||||
Following documents will guide you on how to configure filters based on the type of data source
|
||||
|
||||
{%inlineCalloutContainer%}
|
||||
|
||||
{%inlineCallout
|
||||
bold="Database Filter Patterns"
|
||||
icon="cable"
|
||||
href="/connectors/ingestion/workflows/metadata/filter-patterns/database" %}
|
||||
Learn more about how to configure filters for database sources.
|
||||
{%/inlineCallout%}
|
||||
|
||||
{%/inlineCalloutContainer%}
|
||||
|
||||
|
||||
@ -27,7 +27,7 @@ Navigate to Observability > Alerts.
|
||||
|
||||
Once on the Observability alerts page, click on button **Add Alert** in the top right corner.
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/admin-guide/alert1.png"
|
||||
src="/images/v1.3/how-to-guides/admin-guide/alert1.webp"
|
||||
alt="Create Observability Alerts"
|
||||
caption="Create Observability Alerts"
|
||||
/%}
|
||||
@ -39,7 +39,7 @@ As with most configuration in OpenMetadata you will first need to add a **Name**
|
||||
1. **Source**: Specify the data asset for which the alert must be set up. Alerts can be set up for Container, Ingestion Pipeline, Pipeline, Table, Test Case, Test Suite, and Topic.
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/admin-guide/source.png"
|
||||
src="/images/v1.3/how-to-guides/admin-guide/source.webp"
|
||||
alt="Specify the Data Asset"
|
||||
caption="Specify the Data Asset"
|
||||
/%}
|
||||
@ -47,13 +47,13 @@ caption="Specify the Data Asset"
|
||||
2. **Filter**: Specify the change events to narrow the scope of your alerts. Filters are based on source selected. For example, if the source is a Table, then you can filter by Table Name, Domain, or Owner Name. Use the toggle button to **Include** or **Exclude** the selected filter option. Selecting **Include** will result in an alert being sent if the condition is met. If the Include toggle is turned off, then the condition will be silenced and an alert will not be sent for the denied event. You can add multiple filters.
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/admin-guide/filter.png"
|
||||
src="/images/v1.3/how-to-guides/admin-guide/filter.webp"
|
||||
alt="Define the Filter"
|
||||
caption="Define the Filter"
|
||||
/%}
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/admin-guide/filter2.png"
|
||||
src="/images/v1.3/how-to-guides/admin-guide/filter2.webp"
|
||||
alt="Include/Exclude the Filter"
|
||||
caption="Include/Exclude the Filter"
|
||||
/%}
|
||||
@ -69,7 +69,7 @@ caption="Include/Exclude the Filter"
|
||||
Multiple triggers can be added.
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/admin-guide/trigger.png"
|
||||
src="/images/v1.3/how-to-guides/admin-guide/trigger.webp"
|
||||
alt="Select Important Trigger Events"
|
||||
caption="Select Important Trigger Events"
|
||||
/%}
|
||||
@ -77,7 +77,7 @@ caption="Select Important Trigger Events"
|
||||
4. **Destination**: Specify the destination for the alerts. OpenMetadata supports sending alerts to the users within OpenMetadata, i.e., **Internal** such as Admins, Followers, Owners, Teams and Users.
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/admin-guide/internal.png"
|
||||
src="/images/v1.3/how-to-guides/admin-guide/internal.webp"
|
||||
alt="Send Alerts to the Users and Teams within OpenMetadata"
|
||||
caption="Send Alerts to the Users and Teams within OpenMetadata"
|
||||
/%}
|
||||
@ -85,7 +85,7 @@ caption="Send Alerts to the Users and Teams within OpenMetadata"
|
||||
Alerts can be sent to **External** sources like **Email, G Chat, Generic Webhooks, MS Teams, and Slack**. Add the endpoint URL for the destinations. Alerts can be sent to multiple destinations.
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/admin-guide/external.png"
|
||||
src="/images/v1.3/how-to-guides/admin-guide/external.webp"
|
||||
alt="Send Alerts to Email, Slack, Teams, G Chat, & Webhooks"
|
||||
caption="Send Alerts to Email, Slack, Teams, G Chat, & Webhooks"
|
||||
/%}
|
||||
@ -97,13 +97,13 @@ In addition to setting up observability alerts, you can also set up notification
|
||||
Navigate to Settings > Notifications. Once on the Notifications page, click on button **Add Alert** in the top right corner.
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/admin-guide/notify.png"
|
||||
src="/images/v1.3/how-to-guides/admin-guide/notify.webp"
|
||||
alt="Add Notifications for Change Events"
|
||||
caption="Add Notifications for Change Events"
|
||||
/%}
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/admin-guide/alert2.png"
|
||||
src="/images/v1.3/how-to-guides/admin-guide/alert2.webp"
|
||||
alt="Add an Alert"
|
||||
caption="Add an Alert"
|
||||
/%}
|
||||
@ -115,7 +115,7 @@ Add a **Name** and **Description** for your notification. Configure the notifica
|
||||
1. **Source**: Specify the data asset for which the notification must be set up. Notifications can be set up for Announcements, Chart, Conversation, Dashboard, Dashboard Data Model, Dashboard Service, Database, Database Schema, Database Service, Glossary, Glossary Term, Ingestion Pipeline, Location, Messaging Service, Metadata Service, ML Model, ML Model Service, Pipeline, Pipeline Service, Storage Service, Table, Tag, Tag Category, Task, Topic, or for all the data assets.
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/admin-guide/source2.png"
|
||||
src="/images/v1.3/how-to-guides/admin-guide/source2.webp"
|
||||
alt="Specify the Data Asset"
|
||||
caption="Specify the Data Asset"
|
||||
/%}
|
||||
@ -125,7 +125,7 @@ caption="Specify the Data Asset"
|
||||
Use the toggle button to **Include** or **Exclude** the selected filter option. Selecting **Include** will result in a notification being sent if the condition is met. If the Include toggle is turned off, then the condition will be silenced and a notification will not be sent for the denied event. You can add multiple filters.
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/admin-guide/filter3.png"
|
||||
src="/images/v1.3/how-to-guides/admin-guide/filter3.webp"
|
||||
alt="Define the Filter"
|
||||
caption="Define the Filter"
|
||||
/%}
|
||||
@ -140,7 +140,7 @@ To enable email alerts you will need to ensure that you have an SMTP server avai
|
||||
To update the details from the UI, navigate to Settings > Customize OpenMetadata > Email
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/admin-guide/email.png"
|
||||
src="/images/v1.3/how-to-guides/admin-guide/email.webp"
|
||||
alt="Email Configuration"
|
||||
caption="Email Configuration"
|
||||
/%}
|
||||
|
||||
@ -54,7 +54,7 @@ caption="Landing Page Announcement Display (Top Right)"
|
||||
Furthermore, users can react with emojis and reply to the announcements from both the Activity Feed in the homepage and from the data asset page.
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/user-guide-for-data-stewards/react.png"
|
||||
src="/images/v1.3/how-to-guides/user-guide-for-data-stewards/react.webp"
|
||||
alt="Reply or React to an Announcement"
|
||||
caption="Reply or React to an Announcement"
|
||||
/%}
|
||||
|
||||
@ -0,0 +1,32 @@
|
||||
---
|
||||
title: Conversation Threads
|
||||
slug: /how-to-guides/data-collaboration/conversation
|
||||
---
|
||||
|
||||
# Conversation Threads
|
||||
|
||||
As part of Data Collaboration feature of OpenMetadata, Conversation Threads were one of the earliest features introduced to enable users to easily ask any questions that they might have around a data asset. In the 0.9.0 release of OpenMetadata, we introduced the ability to reply and create entire conversation
|
||||
threads around all the various activities across any data asset. In release 0.11.0, we took it to the next level by extending conversation threads to Tasks and adding support for reactions with emojis.
|
||||
|
||||
Across OpenMetadata, users can start conversations around description, column description, tags, announcements, glossary terms of a data asset by clicking the chat icon as shown in the screen-shot below.
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/collaboration/chat3.webp"
|
||||
alt="Start a Conversation"
|
||||
caption="Start a Conversation"
|
||||
/%}
|
||||
|
||||
Users can also reply or react with emojis for any Conversation by hovering over the conversation.
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/collaboration/chat4.webp"
|
||||
alt="Reply or React to a Conversation"
|
||||
caption="Reply or React to a Conversation"
|
||||
/%}
|
||||
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
bold="Tasks"
|
||||
icon="MdArrowForward"
|
||||
href="/how-to-guides/data-collaboration/tasks"%}
|
||||
Add requests to create or update descriptions and tags.
|
||||
{%/inlineCallout%}
|
||||
@ -0,0 +1,31 @@
|
||||
---
|
||||
title: Create Tasks
|
||||
slug: /how-to-guides/data-collaboration/tasks
|
||||
---
|
||||
|
||||
# Create Tasks
|
||||
|
||||
Tasks are an extension to the Conversation Threads feature where users can create tasks to
|
||||
request to create or update description or tags for a data asset. Tasks are assgined to the owner of the data asset by default. If there are no owners, the task can be assigned to an appropriate user or team.
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/collaboration/task.webp"
|
||||
alt="Start a Conversation"
|
||||
caption="Start a Conversation"
|
||||
/%}
|
||||
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
bold="How to Request for Description"
|
||||
icon="MdArrowForward"
|
||||
href="/how-to-guides/data-collaboration/request-description"%}
|
||||
Request for a description and discuss the same within OpenMetadata
|
||||
{%/inlineCallout%}
|
||||
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
bold="How to Request for Tags"
|
||||
icon="MdArrowForward"
|
||||
href="/how-to-guides/data-collaboration/request-tags"%}
|
||||
Request for tags and discuss about the same, all within OpenMetadata.
|
||||
{%/inlineCallout%}
|
||||
@ -54,7 +54,7 @@ caption="Landing Page Announcement Display (Top Right)"
|
||||
Furthermore, users can react with emojis and reply to the announcements from both the Activity Feed in the homepage and from the data asset page.
|
||||
|
||||
{% image
|
||||
src="/images/v1.3/how-to-guides/user-guide-for-data-stewards/react.png"
|
||||
src="/images/v1.3/how-to-guides/user-guide-for-data-stewards/react.webp"
|
||||
alt="Reply or React to an Announcement"
|
||||
caption="Reply or React to an Announcement"
|
||||
/%}
|
||||
|
||||
@ -0,0 +1,413 @@
|
||||
---
|
||||
title: Database Filter Patterns
|
||||
slug: /connectors/ingestion/workflows/metadata/filter-patterns/database
|
||||
---
|
||||
|
||||
# Database Filter Patterns
|
||||
|
||||
|
||||
## Configuring Filters
|
||||
|
||||
One can configure the metadata ingestion filter for database source using four configuration fields which are `Database Filter Pattern`,
|
||||
`Schema Filter Pattern`, `Table Filter Pattern` & `Use FQN For Filtering`. In this document we will learn about each field in detail
|
||||
along with many examples.
|
||||
|
||||
### Configuring Filters via UI
|
||||
|
||||
Filters can be configured in UI while adding an ingestion pipeline through `Add Metadata Ingestion` page.
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/features/ingestion/workflows/metadata/filter-patterns/database-filter-patterns.webp"
|
||||
alt="Database Filter Pattern Fields"
|
||||
caption="Database Filter Pattern Fields"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI
|
||||
|
||||
Filters can be configured in CLI in connection configuration within `source.sourceConfig.config` field as described below.
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- database1
|
||||
- database2
|
||||
excludes:
|
||||
- database3
|
||||
- database4
|
||||
schemaFilterPattern:
|
||||
includes:
|
||||
- schema1
|
||||
- schema2
|
||||
excludes:
|
||||
- schema3
|
||||
- schema4
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- table1
|
||||
- table2
|
||||
excludes:
|
||||
- table3
|
||||
- table4
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Use FQN For Filtering
|
||||
|
||||
This flag set when you want to apply the filter on fully qualified name (e.g service_name.db_name.schema_name.table_name)
|
||||
instead of applying the filter to raw name of entity (e.g table_name). This Flag is useful in scenario when you have schema
|
||||
with same name in different databases or table with same name in different schemas and you want to filter out one of them. This will be explained further in detail in this document.
|
||||
|
||||
|
||||
### Database Filter Pattern
|
||||
|
||||
Database filter patterns to control whether or not to include database as part of metadata ingestion.
|
||||
- **Include**: Explicitly include databases by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all databases with names matching one or more of the supplied regular expressions. All other databases will be excluded.
|
||||
- **Exclude**: Explicitly exclude databases by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all databases with names matching one or more of the supplied regular expressions. All other databases will be included.
|
||||
|
||||
#### Example 1
|
||||
|
||||
```yaml
|
||||
Snowflake_Prod # Snowflake Service Name
|
||||
│
|
||||
└─── SNOWFLAKE # DB Name
|
||||
│
|
||||
└─── SNOWFLAKE_SAMPLE_DATA # DB Name
|
||||
│
|
||||
└─── TEST_SNOWFLAKEDB # DB Name
|
||||
│
|
||||
└─── DUMMY_DB # DB Name
|
||||
│
|
||||
└─── ECOMMERCE # DB Name
|
||||
```
|
||||
|
||||
Let's say we want to ingest metadata from a snowflake instance which contains multiple databases as described above.
|
||||
In this example we want to ingest all databases which contains `SNOWFLAKE` in name, then the filter pattern
|
||||
applied would be `.*SNOWFLAKE.*` in the include field. This will result in ingestion of database `SNOWFLAKE`, `SNOWFLAKE_SAMPLE_DATA`
|
||||
and `TEST_SNOWFLAKEDB`.
|
||||
|
||||
### Configuring Filters via UI for Example 1
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-1.webp"
|
||||
alt="Database Filter Pattern Example 1"
|
||||
caption="Database Filter Pattern Example 1"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 1
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- .*SNOWFLAKE.*
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Example 2
|
||||
|
||||
In this example we want to ingest all databases which starts with `SNOWFLAKE` in name, then the filter pattern
|
||||
applied would be `^SNOWFLAKE.*` in the include field. This will result in ingestion of database `SNOWFLAKE` & `SNOWFLAKE_SAMPLE_DATA`.
|
||||
|
||||
### Configuring Filters via UI for Example 2
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-2.webp"
|
||||
alt="Database Filter Pattern Example 2"
|
||||
caption="Database Filter Pattern Example 2"
|
||||
/%}
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 2
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- ^SNOWFLAKE.*
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Example 3
|
||||
|
||||
In this example we want to ingest all databases for which the name starts with `SNOWFLAKE` OR ends with `DB` , then the filter pattern
|
||||
applied would be `^SNOWFLAKE` & `DB$` in the include field. This will result in ingestion of database `SNOWFLAKE`, `SNOWFLAKE_SAMPLE_DATA`, `TEST_SNOWFLAKEDB` & `DUMMY_DB`.
|
||||
|
||||
### Configuring Filters via UI for Example 3
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-3.webp"
|
||||
alt="Database Filter Pattern Example 3"
|
||||
caption="Database Filter Pattern Example 3"
|
||||
/%}
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 3
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- ^SNOWFLAKE.*
|
||||
- .*DB$
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Example 4
|
||||
|
||||
In this example we want to ingest only the `SNOWFLAKE` database then the filter pattern applied would be `^SNOWFLAKE$`.
|
||||
|
||||
### Configuring Filters via UI for Example 4
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-4.webp"
|
||||
alt="Database Filter Pattern Example 4"
|
||||
caption="Database Filter Pattern Example 4"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 4
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
databaseFilterPattern:
|
||||
includes:
|
||||
- ^SNOWFLAKE$
|
||||
```
|
||||
|
||||
|
||||
### Schema Filter Pattern
|
||||
|
||||
Schema filter patterns are used to control whether or not to include schemas as part of metadata ingestion.
|
||||
- **Include**: Explicitly include schemas by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all schemas with names matching one or more of the supplied regular expressions. All other schemas will be excluded.
|
||||
- **Exclude**: Explicitly exclude schemas by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all schemas with names matching one or more of the supplied regular expressions. All other schemas will be included.
|
||||
|
||||
#### Example 1
|
||||
|
||||
```yaml
|
||||
Snowflake_Prod # Snowflake Service Name
|
||||
│
|
||||
└─── SNOWFLAKE # DB Name
|
||||
│ │
|
||||
│ └─── PUBLIC # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF1 # Schema Name
|
||||
│ │
|
||||
│ └─── INFORMATION_SCHEMA # Schema Name
|
||||
│
|
||||
└─── SNOWFLAKE_SAMPLE_DATA # DB Name
|
||||
│ │
|
||||
│ └─── PUBLIC # Schema Name
|
||||
│ │
|
||||
│ └─── INFORMATION_SCHEMA # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF1 # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF10 # Schema Name
|
||||
│ │
|
||||
│ └─── TPCH_SF100 # Schema Name
|
||||
```
|
||||
|
||||
In this example we want to ingest all schema within any database with name `PUBLIC`, then the schema filter pattern
|
||||
applied would be `^PUBLIC$` in the include field. This will result in ingestion of schemas `SNOWFLAKE.PUBLIC` & `SNOWFLAKE_SAMPLE_DATA.PUBLIC`
|
||||
|
||||
|
||||
### Configuring Filters via UI for Example 1
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-1.webp"
|
||||
alt="Schema Filter Pattern Example 1"
|
||||
caption="Schema Filter Pattern Example 1"
|
||||
/%}
|
||||
|
||||
### Configuring Filters via CLI for Example 1
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: false
|
||||
schemaFilterPattern:
|
||||
includes:
|
||||
- ^PUBLIC$
|
||||
```
|
||||
|
||||
|
||||
|
||||
#### Example 2
|
||||
|
||||
|
||||
In this example we want to ingest all schema within any database except schema with name `PUBLIC` available in `SNOWFLAKE_SAMPLE_DATA`.
|
||||
Notice that we have two schemas available with name `PUBLIC` one is available in database `SNOWFLAKE_SAMPLE_DATA.PUBLIC` and other is `SNOWFLAKE.PUBLIC`. As per the constraint of this example all the schemas including `SNOWFLAKE.PUBLIC` but we need to skip `SNOWFLAKE_SAMPLE_DATA.PUBLIC`. to do that we will need to set `useFqnForFiltering` flag to true by doing this the filter pattern will be applied to fully qualified name instead of raw table name. A fully qualified name(FQN) of schema is combination of service name, database name & schema name joined with `.`. In this example fully qualified name of the `SNOWFLAKE_SAMPLE_DATA.PUBLIC` schema will be `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC`, so we will need to apply a exclude filter pattern `^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.PUBLIC$` and set `useFqnForFiltering` to true.
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via UI for Example 2
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-2.webp"
|
||||
alt="Schema Filter Pattern Example 2"
|
||||
caption="Schema Filter Pattern Example 2"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 2
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
schemaFilterPattern:
|
||||
excludes:
|
||||
- ^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.PUBLIC$
|
||||
```
|
||||
|
||||
|
||||
#### Example 3
|
||||
|
||||
In this example we want to ingest `SNOWFLAKE.PUBLIC` & all the schemas in `SNOWFLAKE_SAMPLE_DATA` that starts with `TPCH_` i.e `SNOWFLAKE_SAMPLE_DATA.TPCH_1`, `SNOWFLAKE_SAMPLE_DATA.TPCH_10` & `SNOWFLAKE_SAMPLE_DATA.TPCH_100`. To achieve this an include schema filter will be applied with pattern `^Snowflake_Prod\.SNOWFLAKE\.PUBLIC$` & `^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.TPCH_.*`, we need to set `useFqnForFiltering` as true as we want to apply filter on FQN.
|
||||
|
||||
|
||||
### Configuring Filters via UI for Example 3
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-3.webp"
|
||||
alt="Schema Filter Pattern Example 3"
|
||||
caption="Schema Filter Pattern Example 3"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 3
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
schemaFilterPattern:
|
||||
includes:
|
||||
- ^Snowflake_Prod\.SNOWFLAKE\.PUBLIC$
|
||||
- ^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.TPCH_.*
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Table Filter Pattern
|
||||
|
||||
Table filter patterns are used to control whether or not to include tables as part of metadata ingestion.
|
||||
- **Include**: Explicitly include tables by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all tables with names matching one or more of the supplied regular expressions. All other tables will be excluded.
|
||||
- **Exclude**: Explicitly exclude tables by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all tables with names matching one or more of the supplied regular expressions. All other tables will be included.
|
||||
|
||||
|
||||
#### Example 1
|
||||
|
||||
```yaml
|
||||
Snowflake_Prod # Snowflake Service Name
|
||||
│
|
||||
└─── SNOWFLAKE_SAMPLE_DATA # DB Name
|
||||
│ │
|
||||
│ └─── PUBLIC # Schema Name
|
||||
│ │ │
|
||||
│ │ └─── CUSTOMER # Table Name
|
||||
│ │ │
|
||||
│ │ └─── CUSTOMER_ADDRESS # Table Name
|
||||
│ │ │
|
||||
│ │ └─── CUSTOMER_DEMOGRAPHICS # Table Name
|
||||
│ │ │
|
||||
│ │ └─── CALL_CENTER # Table Name
|
||||
│ │
|
||||
│ └─── INFORMATION # Schema Name
|
||||
│ │
|
||||
│ └─── ORDERS # Table Name
|
||||
│ │
|
||||
│ └─── REGION # Table Name
|
||||
│ │
|
||||
│ └─── CUSTOMER # Table Name
|
||||
│
|
||||
└─── SNOWFLAKE # DB Name
|
||||
│
|
||||
└─── PUBLIC # Schema Name
|
||||
│
|
||||
└─── CUSTOMER # Table Name
|
||||
```
|
||||
|
||||
#### Example 1
|
||||
|
||||
In this example we want to ingest table with name `CUSTOMER` within any schema and database. In this case we need to apply include table filter pattern `^CUSTOMER$`. This will result in ingestion of tables `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC.CUSTOMER`, `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.INFORMATION.CUSTOMER` & `Snowflake_Prod.SNOWFLAKE.PUBLIC.CUSTOMER`
|
||||
|
||||
### Configuring Filters via UI for Example 1
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/features/ingestion/workflows/metadata/filter-patterns/table-filter-example-1.webp"
|
||||
alt="Table Filter Pattern Example 1"
|
||||
caption="Table Filter Pattern Example 1"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 1
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- ^CUSTOMER$
|
||||
```
|
||||
|
||||
|
||||
#### Example 2
|
||||
|
||||
In this example we want to ingest table with name `CUSTOMER` within `PUBLIC` schema of any database. In this case we need to apply include table filter pattern `.*\.PUBLIC\.CUSTOMER$` this will also require to set the `useFqnForFiltering` flag as true as we want to apply filter on FQN. This will result in ingestion of tables `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC.CUSTOMER` & `Snowflake_Prod.SNOWFLAKE.PUBLIC.CUSTOMER`
|
||||
|
||||
### Configuring Filters via UI for Example 2
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/features/ingestion/workflows/metadata/filter-patterns/table-filter-example-2.webp"
|
||||
alt="Table Filter Pattern Example 2"
|
||||
caption="Table Filter Pattern Example 2"
|
||||
/%}
|
||||
|
||||
|
||||
|
||||
### Configuring Filters via CLI for Example 2
|
||||
|
||||
```yaml
|
||||
sourceConfig:
|
||||
config:
|
||||
...
|
||||
useFqnForFiltering: true
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- .*\.PUBLIC\.CUSTOMER$
|
||||
```
|
||||
@ -0,0 +1,26 @@
|
||||
---
|
||||
title: Metadata Ingestion Filter Patterns
|
||||
slug: /connectors/ingestion/workflows/metadata/filter-patterns
|
||||
---
|
||||
|
||||
# Metadata Ingestion Filter Patterns
|
||||
|
||||
The ingestion filter patterns are very useful when you have a lot of metadata available in your data source but
|
||||
some metadata might not be useful or relevant to produce any insights or discover data for ex. you might want to
|
||||
filter out the log tables while ingesting metadata.
|
||||
|
||||
Configuring these metadata filters with OpenMetadata is very easy, which uses regex for matching and filtering the metadata.
|
||||
Following documents will guide you on how to configure filters based on the type of data source
|
||||
|
||||
{%inlineCalloutContainer%}
|
||||
|
||||
{%inlineCallout
|
||||
bold="Database Filter Patterns"
|
||||
icon="cable"
|
||||
href="/connectors/ingestion/workflows/metadata/filter-patterns/database" %}
|
||||
Learn more about how to configure filters for database sources.
|
||||
{%/inlineCallout%}
|
||||
|
||||
{%/inlineCalloutContainer%}
|
||||
|
||||
|
||||
@ -0,0 +1,30 @@
|
||||
---
|
||||
title: Metadata Ingestion - Incremental Extraction - BigQuery
|
||||
slug: /connectors/ingestion/workflows/metadata/incremental-extraction/bigquery
|
||||
---
|
||||
|
||||
# Metadata Ingestion - Incremental Extraction - BigQuery
|
||||
|
||||
## Approach
|
||||
|
||||
In order to implement the Incremental Extraction for BigQuery we rely on [`Cloud Logging`](https://cloud.google.com/bigquery/docs/reference/auditlogs) to get the latest DDL changes.
|
||||
|
||||
We then proceed to get the Table Name from the `resourceName` property and we set it to be deleted if `tableDeletion` is present on the payload.
|
||||
|
||||
## Requisites
|
||||
|
||||
The credentials need to have the [Logs Viewer](https://cloud.google.com/logging/docs/access-control#logging.viewer) role.
|
||||
|
||||
## Used Query
|
||||
|
||||
```sql
|
||||
protoPayload.metadata.@type="type.googleapis.com/google.cloud.audit.BigQueryAuditMetadata"
|
||||
AND (
|
||||
protoPayload.methodName = ("google.cloud.bigquery.v2.TableService.UpdateTable" OR "google.cloud.bigquery.v2.TableService.InsertTable" OR "google.cloud.bigquery.v2.TableService.PatchTable" OR "google.cloud.bigquery.v2.TableService.DeleteTable")
|
||||
OR
|
||||
(protoPayload.methodName = "google.cloud.bigquery.v2.JobService.InsertJob" AND (protoPayload.metadata.tableCreation:* OR protoPayload.metadata.tableChange:* OR protoPayload.metadata.tableDeletion:*))
|
||||
)
|
||||
AND resource.labels.project_id = "{project}"
|
||||
AND resource.labels.dataset_id = "{dataset}"
|
||||
AND timestamp >= "{start_date}"
|
||||
```
|
||||
@ -0,0 +1,59 @@
|
||||
---
|
||||
title: Metadata Ingestion - Incremental Extraction
|
||||
slug: /connectors/ingestion/workflows/metadata/incremental-extraction
|
||||
---
|
||||
|
||||
# Metadata Ingestion - Incremental Extraction
|
||||
|
||||
The default Metadata Ingestion roughly follows these steps:
|
||||
|
||||
1. Fetch all the information from the Source.
|
||||
2. Compare the information with the OpenMetadata information to update it properly.
|
||||
3. Compare the information with the OpenMetadata information to delete entities that were deleted.
|
||||
|
||||
While on one hand this is a great simple way of doing things that works for most use cases since at every ingestion pipeline run we get the whole Source state, on other hand this is fetching and comparing a lot of data without need since if there were no structural changes we already know there is nothing to update on OpenMetadata.
|
||||
|
||||
We implemented the Incremental Extraction feature to improve the performance by diminishing the extraction and comparison of uneeded data.
|
||||
|
||||
How this is done depends a lot on the Source itself, but the general idea is to follow these steps:
|
||||
|
||||
1. Fetch the last successful pipeline run.
|
||||
2. Add a small safety margin.
|
||||
3. Get all the structural changes since then.
|
||||
4. Flag deleted entities.
|
||||
5. Fetch/Compare only the entities with structural changes.
|
||||
6. Delete entities flagged for deletion.
|
||||
|
||||
## External Ingestion
|
||||
|
||||
When using the Incremental Extraction feature with External Ingestions (ingesting using YAML files instead of setting it up from the UI), you must pass the ingestion pipeline fully qualified name to the configuration.
|
||||
|
||||
This should be `{service_name}{pipeline_name}`
|
||||
|
||||
**Example:**
|
||||
|
||||
```yaml
|
||||
source:
|
||||
serviceName: my_service
|
||||
# ...
|
||||
# Other configurations
|
||||
# ...
|
||||
ingestionPipelineFQN: my_service.my_pipeline
|
||||
```
|
||||
|
||||
|
||||
## Feature available for
|
||||
|
||||
### Databases
|
||||
|
||||
{% connectorsListContainer %}
|
||||
|
||||
{% connectorInfoCard name="BigQuery" stage="BETA" href="/connectors/ingestion/workflows/metadata/incremental-extraction/bigquery" platform="OpenMetadata" / %}
|
||||
{% connectorInfoCard name="Redshift" stage="BETA" href="/connectors/ingestion/workflows/metadata/incremental-extraction/redshift" platform="OpenMetadata" / %}
|
||||
{% connectorInfoCard name="Snowflake" stage="BETA" href="/connectors/ingestion/workflows/metadata/incremental-extraction/snowflake" platform="OpenMetadata" / %}
|
||||
|
||||
{% /connectorsListContainer %}
|
||||
|
||||
<!-- [**BigQuery**](/connectors/ingestion/workflows/metadata/incremental-extraction/bigquery) -->
|
||||
<!-- [**Redshift**](/connectors/ingestion/workflows/metadata/incremental-extraction/redshift) -->
|
||||
<!-- [**Snowflake**](/connectors/ingestion/workflows/metadata/incremental-extraction/snowflake) -->
|
||||
@ -0,0 +1,86 @@
|
||||
---
|
||||
title: Metadata Ingestion - Incremental Extraction - Redshift
|
||||
slug: /connectors/ingestion/workflows/metadata/incremental-extraction/redshift
|
||||
---
|
||||
|
||||
# Metadata Ingestion - Incremental Extraction - Redshift
|
||||
|
||||
## Approach
|
||||
|
||||
In order to implement the Incremental Extraction for Redshift we rely on the [`SYS_QUERY_HISTORY`](https://docs.aws.amazon.com/redshift/latest/dg/SYS_QUERY_HISTORY.html) to get the latest DDL changes.
|
||||
|
||||
We then match the `query_text` with different regex to get the Schema and Table name and understand if it was a deletion.
|
||||
|
||||
If no schema is present in the query we infer that it is the default for the database.
|
||||
|
||||
## Requisites
|
||||
|
||||
The user should be a Superuser to be able to see all the records.
|
||||
|
||||
## Used Regex
|
||||
|
||||
### Table Name
|
||||
|
||||
```python
|
||||
r"(\w+\.){0,2}\w+"
|
||||
```
|
||||
|
||||
### Create Table
|
||||
|
||||
```python
|
||||
rf"^.*CREATE\s+(LOCAL\s+|EXTERNAL\s+)?(TEMPORARY\s+|TEMP\s+)?TABLE\s+(IF\s+NOT\s+EXISTS\s+)?(?P<table>{TABLE_NAME_RE}).*$"
|
||||
```
|
||||
|
||||
### Alter Table
|
||||
|
||||
```python
|
||||
rf"^.*(ALTER\s+TABLE)\s+(?P<table>{TABLE_NAME_RE}).*$"
|
||||
```
|
||||
|
||||
### Drop Table
|
||||
|
||||
```python
|
||||
rf"^.*DROP\s+TABLE\s+(IF\s+EXISTS\s+)?(?P<table>{TABLE_NAME_RE}).*$"
|
||||
```
|
||||
|
||||
### Create View
|
||||
|
||||
```python
|
||||
rf"^.*CREATE\s+(OR\s+REPLACE\s+)?(EXTERNAL\s+|MATERIALIZED\s+)?VIEW\s+(?P<table>{TABLE_NAME_RE}).*$"
|
||||
```
|
||||
|
||||
### Alter View
|
||||
|
||||
```python
|
||||
rf"^.*ALTER\s+(EXTERNAL\s+)?VIEW\s+(?P<table>{TABLE_NAME_RE}).*$"
|
||||
```
|
||||
|
||||
### Drop View
|
||||
|
||||
```python
|
||||
rf"^.*DROP\s+(EXTERNAL\s+|MATERIALIZED\s+)?VIEW\s+(IF\s+EXISTS\s+)?(?P<table>{TABLE_NAME_RE}).*$"
|
||||
```
|
||||
|
||||
### Comment
|
||||
|
||||
```python
|
||||
# Not supporting Comment changes on Constraint
|
||||
rf"^.*COMMENT\s+ON\s+(TABLE|COLUMN|VIEW)\s+(?P<table>{TABLE_NAME_RE}).*$"
|
||||
```
|
||||
|
||||
## Used Query
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
query_text
|
||||
FROM SYS_QUERY_HISTORY
|
||||
WHERE status = 'success'
|
||||
AND (
|
||||
query_type = 'DDL' OR
|
||||
(query_type = 'UTILITY' AND query_text ilike '%%COMMENT ON%%') OR
|
||||
(query_type = 'CTAS' AND query_text ilike '%%CREATE TABLE%%')
|
||||
)
|
||||
and database_name = '{database}'
|
||||
and end_time >= '{start_date}'
|
||||
ORDER BY end_time DESC
|
||||
```
|
||||
@ -0,0 +1,114 @@
|
||||
---
|
||||
title: Metadata Ingestion - Incremental Extraction - Snowflake
|
||||
slug: /connectors/ingestion/workflows/metadata/incremental-extraction/snowflake
|
||||
---
|
||||
|
||||
# Metadata Ingestion - Incremental Extraction - Snowflake
|
||||
|
||||
## Approach
|
||||
|
||||
In order to implement the Incremental Extraction for Snowflake we rely on the [`SNOWFLAKE.ACCOUNT_USAGE.TABLES`](https://docs.snowflake.com/en/sql-reference/account-usage/tables) view to get the latest DDL changes.
|
||||
|
||||
## Used Queries
|
||||
|
||||
### External Tables
|
||||
|
||||
```sql
|
||||
select TABLE_NAME, DELETED
|
||||
from (
|
||||
select
|
||||
TABLE_NAME,
|
||||
DELETED,
|
||||
ROW_NUMBER() over (
|
||||
partition by TABLE_NAME order by LAST_DDL desc
|
||||
) as ROW_NUMBER
|
||||
from snowflake.account_usage.tables
|
||||
where TABLE_CATALOG = '{database}'
|
||||
and TABLE_SCHEMA = '{schema}'
|
||||
and TABLE_TYPE = 'EXTERNAL TABLE'
|
||||
and DATE_PART(epoch_millisecond, LAST_DDL) >= '{date}'
|
||||
)
|
||||
where ROW_NUMBER = 1
|
||||
```
|
||||
|
||||
### Base, Not Transient Tables
|
||||
|
||||
```sql
|
||||
select TABLE_NAME, DELETED
|
||||
from (
|
||||
select
|
||||
TABLE_NAME,
|
||||
DELETED,
|
||||
ROW_NUMBER() over (
|
||||
partition by TABLE_NAME order by LAST_DDL desc
|
||||
) as ROW_NUMBER
|
||||
from snowflake.account_usage.tables
|
||||
where TABLE_CATALOG = '{database}'
|
||||
and TABLE_SCHEMA = '{schema}'
|
||||
and TABLE_TYPE = 'BASE TABLE'
|
||||
and IS_TRANSIENT != 'YES'
|
||||
and DATE_PART(epoch_millisecond, LAST_DDL) >= '{date}'
|
||||
)
|
||||
where ROW_NUMBER = 1
|
||||
```
|
||||
|
||||
### Base, Transient Tables
|
||||
|
||||
```sql
|
||||
select TABLE_NAME, DELETED
|
||||
from (
|
||||
select
|
||||
TABLE_NAME,
|
||||
DELETED,
|
||||
ROW_NUMBER() over (
|
||||
partition by TABLE_NAME order by LAST_DDL desc
|
||||
) as ROW_NUMBER
|
||||
from snowflake.account_usage.tables
|
||||
where TABLE_CATALOG = '{database}'
|
||||
and TABLE_SCHEMA = '{schema}'
|
||||
and TABLE_TYPE = 'BASE TABLE'
|
||||
and IS_TRANSIENT = 'YES'
|
||||
and DATE_PART(epoch_millisecond, LAST_DDL) >= '{date}'
|
||||
)
|
||||
where ROW_NUMBER = 1
|
||||
```
|
||||
|
||||
### Views
|
||||
|
||||
```sql
|
||||
select TABLE_NAME, DELETED
|
||||
from (
|
||||
select
|
||||
TABLE_NAME,
|
||||
DELETED,
|
||||
ROW_NUMBER() over (
|
||||
partition by TABLE_NAME order by LAST_DDL desc
|
||||
) as ROW_NUMBER
|
||||
from snowflake.account_usage.tables
|
||||
where TABLE_CATALOG = '{database}'
|
||||
and TABLE_SCHEMA = '{schema}'
|
||||
and TABLE_TYPE = 'VIEW'
|
||||
and DATE_PART(epoch_millisecond, LAST_DDL) >= '{date}'
|
||||
)
|
||||
where ROW_NUMBER = 1
|
||||
```
|
||||
|
||||
### Materialized Views
|
||||
|
||||
```sql
|
||||
select TABLE_NAME, DELETED
|
||||
from (
|
||||
select
|
||||
TABLE_NAME,
|
||||
DELETED,
|
||||
ROW_NUMBER() over (
|
||||
partition by TABLE_NAME order by LAST_DDL desc
|
||||
) as ROW_NUMBER
|
||||
from snowflake.account_usage.tables
|
||||
where TABLE_CATALOG = '{database}'
|
||||
and TABLE_SCHEMA = '{schema}'
|
||||
and TABLE_TYPE = 'MATERIALIZED VIEW'
|
||||
and DATE_PART(epoch_millisecond, LAST_DDL) >= '{date}'
|
||||
)
|
||||
where ROW_NUMBER = 1
|
||||
```
|
||||
@ -27,7 +27,7 @@ Navigate to Observability > Alerts.
|
||||
|
||||
Once on the Observability alerts page, click on button **Add Alert** in the top right corner.
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/admin-guide/alert1.png"
|
||||
src="/images/v1.4/how-to-guides/admin-guide/alert1.webp"
|
||||
alt="Create Observability Alerts"
|
||||
caption="Create Observability Alerts"
|
||||
/%}
|
||||
@ -39,7 +39,7 @@ As with most configuration in OpenMetadata you will first need to add a **Name**
|
||||
1. **Source**: Specify the data asset for which the alert must be set up. Alerts can be set up for Container, Ingestion Pipeline, Pipeline, Table, Test Case, Test Suite, and Topic.
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/admin-guide/source.png"
|
||||
src="/images/v1.4/how-to-guides/admin-guide/source.webp"
|
||||
alt="Specify the Data Asset"
|
||||
caption="Specify the Data Asset"
|
||||
/%}
|
||||
@ -47,13 +47,13 @@ caption="Specify the Data Asset"
|
||||
2. **Filter**: Specify the change events to narrow the scope of your alerts. Filters are based on source selected. For example, if the source is a Table, then you can filter by Table Name, Domain, or Owner Name. Use the toggle button to **Include** or **Exclude** the selected filter option. Selecting **Include** will result in an alert being sent if the condition is met. If the Include toggle is turned off, then the condition will be silenced and an alert will not be sent for the denied event. You can add multiple filters.
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/admin-guide/filter.png"
|
||||
src="/images/v1.4/how-to-guides/admin-guide/filter.webp"
|
||||
alt="Define the Filter"
|
||||
caption="Define the Filter"
|
||||
/%}
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/admin-guide/filter2.png"
|
||||
src="/images/v1.4/how-to-guides/admin-guide/filter2.webp"
|
||||
alt="Include/Exclude the Filter"
|
||||
caption="Include/Exclude the Filter"
|
||||
/%}
|
||||
@ -69,7 +69,7 @@ caption="Include/Exclude the Filter"
|
||||
Multiple triggers can be added.
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/admin-guide/trigger.png"
|
||||
src="/images/v1.4/how-to-guides/admin-guide/trigger.webp"
|
||||
alt="Select Important Trigger Events"
|
||||
caption="Select Important Trigger Events"
|
||||
/%}
|
||||
@ -77,7 +77,7 @@ caption="Select Important Trigger Events"
|
||||
4. **Destination**: Specify the destination for the alerts. OpenMetadata supports sending alerts to the users within OpenMetadata, i.e., **Internal** such as Admins, Followers, Owners, Teams and Users.
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/admin-guide/internal.png"
|
||||
src="/images/v1.4/how-to-guides/admin-guide/internal.webp"
|
||||
alt="Send Alerts to the Users and Teams within OpenMetadata"
|
||||
caption="Send Alerts to the Users and Teams within OpenMetadata"
|
||||
/%}
|
||||
@ -85,7 +85,7 @@ caption="Send Alerts to the Users and Teams within OpenMetadata"
|
||||
Alerts can be sent to **External** sources like **Email, G Chat, Generic Webhooks, MS Teams, and Slack**. Add the endpoint URL for the destinations. Alerts can be sent to multiple destinations.
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/admin-guide/external.png"
|
||||
src="/images/v1.4/how-to-guides/admin-guide/external.webp"
|
||||
alt="Send Alerts to Email, Slack, Teams, G Chat, & Webhooks"
|
||||
caption="Send Alerts to Email, Slack, Teams, G Chat, & Webhooks"
|
||||
/%}
|
||||
@ -97,13 +97,13 @@ In addition to setting up observability alerts, you can also set up notification
|
||||
Navigate to Settings > Notifications. Once on the Notifications page, click on button **Add Alert** in the top right corner.
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/admin-guide/notify.png"
|
||||
src="/images/v1.4/how-to-guides/admin-guide/notify.webp"
|
||||
alt="Add Notifications for Change Events"
|
||||
caption="Add Notifications for Change Events"
|
||||
/%}
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/admin-guide/alert2.png"
|
||||
src="/images/v1.4/how-to-guides/admin-guide/alert2.webp"
|
||||
alt="Add an Alert"
|
||||
caption="Add an Alert"
|
||||
/%}
|
||||
@ -115,7 +115,7 @@ Add a **Name** and **Description** for your notification. Configure the notifica
|
||||
1. **Source**: Specify the data asset for which the notification must be set up. Notifications can be set up for Announcements, Chart, Conversation, Dashboard, Dashboard Data Model, Dashboard Service, Database, Database Schema, Database Service, Glossary, Glossary Term, Ingestion Pipeline, Location, Messaging Service, Metadata Service, ML Model, ML Model Service, Pipeline, Pipeline Service, Storage Service, Table, Tag, Tag Category, Task, Topic, or for all the data assets.
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/admin-guide/source2.png"
|
||||
src="/images/v1.4/how-to-guides/admin-guide/source2.webp"
|
||||
alt="Specify the Data Asset"
|
||||
caption="Specify the Data Asset"
|
||||
/%}
|
||||
@ -125,7 +125,7 @@ caption="Specify the Data Asset"
|
||||
Use the toggle button to **Include** or **Exclude** the selected filter option. Selecting **Include** will result in a notification being sent if the condition is met. If the Include toggle is turned off, then the condition will be silenced and a notification will not be sent for the denied event. You can add multiple filters.
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/admin-guide/filter3.png"
|
||||
src="/images/v1.4/how-to-guides/admin-guide/filter3.webp"
|
||||
alt="Define the Filter"
|
||||
caption="Define the Filter"
|
||||
/%}
|
||||
@ -140,7 +140,7 @@ To enable email alerts you will need to ensure that you have an SMTP server avai
|
||||
To update the details from the UI, navigate to Settings > Customize OpenMetadata > Email
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/admin-guide/email.png"
|
||||
src="/images/v1.4/how-to-guides/admin-guide/email.webp"
|
||||
alt="Email Configuration"
|
||||
caption="Email Configuration"
|
||||
/%}
|
||||
|
||||
@ -54,7 +54,7 @@ caption="Landing Page Announcement Display (Top Right)"
|
||||
Furthermore, users can react with emojis and reply to the announcements from both the Activity Feed in the homepage and from the data asset page.
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/user-guide-for-data-stewards/react.png"
|
||||
src="/images/v1.4/how-to-guides/user-guide-for-data-stewards/react.webp"
|
||||
alt="Reply or React to an Announcement"
|
||||
caption="Reply or React to an Announcement"
|
||||
/%}
|
||||
|
||||
@ -0,0 +1,32 @@
|
||||
---
|
||||
title: Conversation Threads
|
||||
slug: /how-to-guides/data-collaboration/conversation
|
||||
---
|
||||
|
||||
# Conversation Threads
|
||||
|
||||
As part of Data Collaboration feature of OpenMetadata, Conversation Threads were one of the earliest features introduced to enable users to easily ask any questions that they might have around a data asset. In the 0.9.0 release of OpenMetadata, we introduced the ability to reply and create entire conversation
|
||||
threads around all the various activities across any data asset. In release 0.11.0, we took it to the next level by extending conversation threads to Tasks and adding support for reactions with emojis.
|
||||
|
||||
Across OpenMetadata, users can start conversations around description, column description, tags, announcements, glossary terms of a data asset by clicking the chat icon as shown in the screen-shot below.
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/collaboration/chat3.webp"
|
||||
alt="Start a Conversation"
|
||||
caption="Start a Conversation"
|
||||
/%}
|
||||
|
||||
Users can also reply or react with emojis for any Conversation by hovering over the conversation.
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/collaboration/chat4.webp"
|
||||
alt="Reply or React to a Conversation"
|
||||
caption="Reply or React to a Conversation"
|
||||
/%}
|
||||
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
bold="Tasks"
|
||||
icon="MdArrowForward"
|
||||
href="/how-to-guides/data-collaboration/tasks"%}
|
||||
Add requests to create or update descriptions and tags.
|
||||
{%/inlineCallout%}
|
||||
@ -0,0 +1,31 @@
|
||||
---
|
||||
title: Create Tasks
|
||||
slug: /how-to-guides/data-collaboration/tasks
|
||||
---
|
||||
|
||||
# Create Tasks
|
||||
|
||||
Tasks are an extension to the Conversation Threads feature where users can create tasks to
|
||||
request to create or update description or tags for a data asset. Tasks are assgined to the owner of the data asset by default. If there are no owners, the task can be assigned to an appropriate user or team.
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/collaboration/task.webp"
|
||||
alt="Start a Conversation"
|
||||
caption="Start a Conversation"
|
||||
/%}
|
||||
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
bold="How to Request for Description"
|
||||
icon="MdArrowForward"
|
||||
href="/how-to-guides/data-collaboration/request-description"%}
|
||||
Request for a description and discuss the same within OpenMetadata
|
||||
{%/inlineCallout%}
|
||||
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
bold="How to Request for Tags"
|
||||
icon="MdArrowForward"
|
||||
href="/how-to-guides/data-collaboration/request-tags"%}
|
||||
Request for tags and discuss about the same, all within OpenMetadata.
|
||||
{%/inlineCallout%}
|
||||
@ -54,7 +54,7 @@ caption="Landing Page Announcement Display (Top Right)"
|
||||
Furthermore, users can react with emojis and reply to the announcements from both the Activity Feed in the homepage and from the data asset page.
|
||||
|
||||
{% image
|
||||
src="/images/v1.4/how-to-guides/user-guide-for-data-stewards/react.png"
|
||||
src="/images/v1.4/how-to-guides/user-guide-for-data-stewards/react.webp"
|
||||
alt="Reply or React to an Announcement"
|
||||
caption="Reply or React to an Announcement"
|
||||
/%}
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 75 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 20 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 91 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 29 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 75 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 20 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 91 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 29 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 75 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 20 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 48 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 87 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 20 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 32 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 91 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 29 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 75 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 20 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 48 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user