yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-08-16 04:57:11 +00:00
Code Issues Packages Projects Releases Wiki Activity

Bigquery Docs Min Requirements (#6561)

Browse Source
This commit is contained in:
Mayur Singal 2022-08-04 13:59:45 +05:30 committed by GitHub
parent 951251a444
commit 5754b1424a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 58 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
ingestion/src/metadata/utils/sql_queries.py
View File

@ -400,8 +400,8 @@ WHERE creation_time BETWEEN "{start_time}" AND "{end_time}"
AND job_type = "QUERY"
AND state = "DONE"
AND IFNULL(statement_type, "NO") not in ("NO", "DROP_TABLE", "CREATE_TABLE")
AND text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
AND text NOT LIKE '/* {{"app": "dbt", %%}} */%%'
AND query NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
AND query NOT LIKE '/* {{"app": "dbt", %%}} */%%'
"""

18
openmetadata-docs/content/openmetadata/connectors/database/bigquery/airflow.md
View File

@ -7,6 +7,24 @@ slug: /openmetadata/connectors/database/bigquery/airflow
<Requirements />
<h4>GCP Permissions</h4>
<p> To execute metadata extraction and usage workflow successfully the user or the service account should have enough access to fetch required data. Following table describes the minimum required permissions </p>
| # | GCP Permission | GCP Role | Required For |
| :---------- | :---------- | :---------- | :---------- |
| 1 | bigquery.datasets.get | BigQuery Data Viewer | Metadata Ingestion |
| 2 | bigquery.tables.get | BigQuery Data Viewer | Metadata Ingestion |
| 3 | bigquery.tables.getData | BigQuery Data Viewer | Metadata Ingestion |
| 4 | bigquery.tables.list | BigQuery Data Viewer | Metadata Ingestion |
| 5 | resourcemanager.projects.get | BigQuery Data Viewer | Metadata Ingestion |
| 6 | bigquery.jobs.create | BigQuery Job User | Metadata Ingestion |
| 7 | bigquery.jobs.listAll | BigQuery Job User | Metadata Ingestion |
| 8 | datacatalog.taxonomies.get | BigQuery Policy Admin | Fetch Policy Tags |
| 9 | datacatalog.taxonomies.list | BigQuery Policy Admin | Fetch Policy Tags |
| 10 | bigquery.readsessions.create | BigQuery Admin | Bigquery Usage Workflow |
| 11 | bigquery.readsessions.getData | BigQuery Admin | Bigquery Usage Workflow |
<MetadataIngestionServiceDev service="database" connector="BigQuery" goal="Airflow"/>
<h4>Source Configuration - Service Connection</h4>

19
openmetadata-docs/content/openmetadata/connectors/database/bigquery/cli.md
View File

@ -7,6 +7,25 @@ slug: /openmetadata/connectors/database/bigquery/cli
<Requirements />
<h4>GCP Permissions</h4>
<p> To execute metadata extraction and usage workflow successfully the user or the service account should have enough access to fetch required data. Following table describes the minimum required permissions </p>
| # | GCP Permission | GCP Role | Required For |
| :---------- | :---------- | :---------- | :---------- |
| 1 | bigquery.datasets.get | BigQuery Data Viewer | Metadata Ingestion |
| 2 | bigquery.tables.get | BigQuery Data Viewer | Metadata Ingestion |
| 3 | bigquery.tables.getData | BigQuery Data Viewer | Metadata Ingestion |
| 4 | bigquery.tables.list | BigQuery Data Viewer | Metadata Ingestion |
| 5 | resourcemanager.projects.get | BigQuery Data Viewer | Metadata Ingestion |
| 6 | bigquery.jobs.create | BigQuery Job User | Metadata Ingestion |
| 7 | bigquery.jobs.listAll | BigQuery Job User | Metadata Ingestion |
| 8 | datacatalog.taxonomies.get | BigQuery Policy Admin | Fetch Policy Tags |
| 9 | datacatalog.taxonomies.list | BigQuery Policy Admin | Fetch Policy Tags |
| 10 | bigquery.readsessions.create | BigQuery Admin | Bigquery Usage Workflow |
| 11 | bigquery.readsessions.getData | BigQuery Admin | Bigquery Usage Workflow |
<MetadataIngestionServiceDev service="database" connector="BigQuery" goal="CLI"/>
<h4>Source Configuration - Service Connection</h4>

18
openmetadata-docs/content/openmetadata/connectors/database/bigquery/index.md
View File

@ -7,6 +7,24 @@ slug: /openmetadata/connectors/database/bigquery
<Requirements />
<h4>GCP Permissions</h4>
<p> To execute metadata extraction and usage workflow successfully the user or the service account should have enough access to fetch required data. Following table describes the minimum required permissions </p>
| # | GCP Permission | GCP Role | Required For |
| :---------- | :---------- | :---------- | :---------- |
| 1 | bigquery.datasets.get | BigQuery Data Viewer | Metadata Ingestion |
| 2 | bigquery.tables.get | BigQuery Data Viewer | Metadata Ingestion |
| 3 | bigquery.tables.getData | BigQuery Data Viewer | Metadata Ingestion |
| 4 | bigquery.tables.list | BigQuery Data Viewer | Metadata Ingestion |
| 5 | resourcemanager.projects.get | BigQuery Data Viewer | Metadata Ingestion |
| 6 | bigquery.jobs.create | BigQuery Job User | Metadata Ingestion |
| 7 | bigquery.jobs.listAll | BigQuery Job User | Metadata Ingestion |
| 8 | datacatalog.taxonomies.get | BigQuery Policy Admin | Fetch Policy Tags |
| 9 | datacatalog.taxonomies.list | BigQuery Policy Admin | Fetch Policy Tags |
| 10 | bigquery.readsessions.create | BigQuery Admin | Bigquery Usage Workflow |
| 11 | bigquery.readsessions.getData | BigQuery Admin | Bigquery Usage Workflow |
<MetadataIngestionService connector="BigQuery"/>
<h4>Connection Options</h4>

3
openmetadata-docs/content/openmetadata/ingestion/workflows/usage/usage-workflow-query-logs.md
View File

@ -50,8 +50,7 @@ source:
queryLogFilePath: <path to query log file>
processor:
type: query-parser
config:
filter: ''
config: {}
stage:
type: table-usage
config: