doc: added more details regarding freshness metrics (#10098)

This commit is contained in:
Teddy 2023-02-03 15:03:28 +01:00 committed by GitHub
parent 92ad68a7b5
commit 3bd613e51a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 9 additions and 6 deletions

View File

@ -35,7 +35,15 @@ It computes the number of rows in the Table.
Returns the number of columns in the Table.
## System Metrics
System metrics are metrics related to DML operations performed on the table. These metrics are available for BigQuery, Redshift and Snowflake only. Other database engines are currently not supported so the computation of the system metrics will be skipped.
System metrics provide information related to DML operations performed on the table. These metrics present a concise view of your data freshness. In a typical data processing flow tables are updated at a certain frequency. Table freshness will be monitored by confirming a set of operations has been performed against the table. To increase trust in your data assets, OpenMetadata will monitor the `INSERT`, `UPDATE` and `DELETE` operations performed against your table to showcase 2 metrics related to freshness (see below for more details). With this information, you are able to see when a specific operation was last perform and how many rows it affected.
<Image
src={"/images/openmetadata/ingestion/workflows/profiler/profiler-freshness-metrics.png"}
alt="table profile freshness metrics"
caption="table profile freshness metrics"
/>
These metrics are available for **BigQuery**, **Redshift** and **Snowflake**. Other database engines are currently not supported so the computation of the system metrics will be skipped.
### DML Operations
This metrics shows all the DML operations performed (`INSERT`, `UPDATE`, `DELETE`) against the table in a timeseries fashion.
@ -134,11 +142,6 @@ OpenMetadata uses `stl_insert`, `stl_delete`, `svv_table_info`, and `stl_queryte
OpenMetadata will look at the previous day to fetch the operations that were performed against a table.
### Redshift
OpenMetadata uses `stl_insert`, `stl_delete`, `svv_table_info`, and `stl_querytext` to fecth DNL operations as well as the number of rows affected by these operations. You need to make sure the user running the profiler workflow has access to these views and tables.
OpenMetadata will look at the previous day to fetch the operations that were performed against a table.
### BigQuery
Bigquery uses the `JOBS` table of the `INFORMATION_SCHEMA` to fecth DNL operations as well as the number of rows affected by these operations. You will need to make sure your data location is properly set when creating your BigQuery service connection in OpenMetadata.

Binary file not shown.

After

Width:  |  Height:  |  Size: 48 KiB