yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-10-18 20:30:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

Bigquery Docs Fix (#10692)

Browse Source 
* Bigquery Docs Fix

* typo fix

* keycloak sso

* GCP Create Custom Role
This commit is contained in:
Mayur Singal 2023-03-21 21:25:57 +05:30 committed by GitHub
parent b8e0ae489a
commit 9f99296ea0
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
12 changed files with 159 additions and 39 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

35
openmetadata-docs/content/connectors/database/bigquery/airflow.md
View File

@ -57,25 +57,34 @@ pip3 install "openmetadata-ingestion[bigquery-usage]"
<h4>GCP Permissions</h4> <h4>GCP Permissions</h4>
<p> To execute metadata extraction and usage workflow successfully the user or the service account should have enough access to fetch required data. Following table describes the minimum required permissions </p> <p> To execute metadata extraction and usage workflow successfully the user or the service account should have enough access to fetch required data. Following table describes the minimum required permissions </p>
<Table> <Table>
| # | GCP Permission | GCP Role | Required For | | # | GCP Permission | Required For |
| :---------- | :---------- | :---------- | :---------- | | :--- | :---------------------------- | :---------------------- |
| 1 | bigquery.datasets.get | BigQuery Data Viewer | Metadata Ingestion | | 1 | bigquery.datasets.get | Metadata Ingestion |
| 2 | bigquery.tables.get | BigQuery Data Viewer | Metadata Ingestion | | 2 | bigquery.tables.get | Metadata Ingestion |
| 3 | bigquery.tables.getData | BigQuery Data Viewer | Metadata Ingestion | | 3 | bigquery.tables.getData | Metadata Ingestion |
| 4 | bigquery.tables.list | BigQuery Data Viewer | Metadata Ingestion | | 4 | bigquery.tables.list | Metadata Ingestion |
| 5 | resourcemanager.projects.get | BigQuery Data Viewer | Metadata Ingestion | | 5 | resourcemanager.projects.get | Metadata Ingestion |
| 6 | bigquery.jobs.create | BigQuery Job User | Metadata Ingestion | | 6 | bigquery.jobs.create | Metadata Ingestion |
| 7 | bigquery.jobs.listAll | BigQuery Job User | Metadata Ingestion | | 7 | bigquery.jobs.listAll | Metadata Ingestion |
| 8 | datacatalog.taxonomies.get | BigQuery Policy Admin | Fetch Policy Tags | | 8 | datacatalog.taxonomies.get | Fetch Policy Tags |
| 9 | datacatalog.taxonomies.list | BigQuery Policy Admin | Fetch Policy Tags | | 9 | datacatalog.taxonomies.list | Fetch Policy Tags |
| 10 | bigquery.readsessions.create | BigQuery Admin | Bigquery Usage Workflow | | 10 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow |
| 11 | bigquery.readsessions.getData | BigQuery Admin | Bigquery Usage Workflow | | 11 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow |
</Table> </Table>
<Tile
icon="manage_accounts"
title="Create Custom GCP Role"
text="Checkout this documentation on how to create a custom role and assign it to the service account."
link="/connectors/database/bigquery/roles"
/>
## Metadata Ingestion ## Metadata Ingestion
All connectors are defined as JSON Schemas. All connectors are defined as JSON Schemas.

34
openmetadata-docs/content/connectors/database/bigquery/cli.md
View File

@ -59,23 +59,31 @@ pip3 install "openmetadata-ingestion[bigquery-usage]"
<Table> <Table>
| # | GCP Permission | GCP Role | Required For | | # | GCP Permission | Required For |
| :---------- | :---------- | :---------- | :---------- | | :--- | :---------------------------- | :---------------------- |
| 1 | bigquery.datasets.get | BigQuery Data Viewer | Metadata Ingestion | | 1 | bigquery.datasets.get | Metadata Ingestion |
| 2 | bigquery.tables.get | BigQuery Data Viewer | Metadata Ingestion | | 2 | bigquery.tables.get | Metadata Ingestion |
| 3 | bigquery.tables.getData | BigQuery Data Viewer | Metadata Ingestion | | 3 | bigquery.tables.getData | Metadata Ingestion |
| 4 | bigquery.tables.list | BigQuery Data Viewer | Metadata Ingestion | | 4 | bigquery.tables.list | Metadata Ingestion |
| 5 | resourcemanager.projects.get | BigQuery Data Viewer | Metadata Ingestion | | 5 | resourcemanager.projects.get | Metadata Ingestion |
| 6 | bigquery.jobs.create | BigQuery Job User | Metadata Ingestion | | 6 | bigquery.jobs.create | Metadata Ingestion |
| 7 | bigquery.jobs.listAll | BigQuery Job User | Metadata Ingestion | | 7 | bigquery.jobs.listAll | Metadata Ingestion |
| 8 | datacatalog.taxonomies.get | BigQuery Policy Admin | Fetch Policy Tags | | 8 | datacatalog.taxonomies.get | Fetch Policy Tags |
| 9 | datacatalog.taxonomies.list | BigQuery Policy Admin | Fetch Policy Tags | | 9 | datacatalog.taxonomies.list | Fetch Policy Tags |
| 10 | bigquery.readsessions.create | BigQuery Admin | Bigquery Usage Workflow | | 10 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow |
| 11 | bigquery.readsessions.getData | BigQuery Admin | Bigquery Usage Workflow | | 11 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow |
</Table> </Table>
<Tile
icon="manage_accounts"
title="Create Custom GCP Role"
text="Checkout this documentation on how to create a custom role and assign it to the service account."
link="/connectors/database/bigquery/roles"
/>
## Metadata Ingestion ## Metadata Ingestion
All connectors are defined as JSON Schemas. All connectors are defined as JSON Schemas.

34
openmetadata-docs/content/connectors/database/bigquery/index.md
View File

@ -74,22 +74,30 @@ custom Airflow plugins to handle the workflow deployment.
<Table> <Table>
| # | GCP Permission | GCP Role | Required For | | # | GCP Permission | Required For |
| :--- | :---------------------------- | :-------------------- | :---------------------- | | :--- | :---------------------------- | :---------------------- |
| 1 | bigquery.datasets.get | BigQuery Data Viewer | Metadata Ingestion | | 1 | bigquery.datasets.get | Metadata Ingestion |
| 2 | bigquery.tables.get | BigQuery Data Viewer | Metadata Ingestion | | 2 | bigquery.tables.get | Metadata Ingestion |
| 3 | bigquery.tables.getData | BigQuery Data Viewer | Metadata Ingestion | | 3 | bigquery.tables.getData | Metadata Ingestion |
| 4 | bigquery.tables.list | BigQuery Data Viewer | Metadata Ingestion | | 4 | bigquery.tables.list | Metadata Ingestion |
| 5 | resourcemanager.projects.get | BigQuery Data Viewer | Metadata Ingestion | | 5 | resourcemanager.projects.get | Metadata Ingestion |
| 6 | bigquery.jobs.create | BigQuery Job User | Metadata Ingestion | | 6 | bigquery.jobs.create | Metadata Ingestion |
| 7 | bigquery.jobs.listAll | BigQuery Job User | Metadata Ingestion | | 7 | bigquery.jobs.listAll | Metadata Ingestion |
| 8 | datacatalog.taxonomies.get | BigQuery Policy Admin | Fetch Policy Tags | | 8 | datacatalog.taxonomies.get | Fetch Policy Tags |
| 9 | datacatalog.taxonomies.list | BigQuery Policy Admin | Fetch Policy Tags | | 9 | datacatalog.taxonomies.list | Fetch Policy Tags |
| 10 | bigquery.readsessions.create | BigQuery Admin | Bigquery Usage Workflow | | 10 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow |
| 11 | bigquery.readsessions.getData | BigQuery Admin | Bigquery Usage Workflow | | 11 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow |
</Table> </Table>
<Tile
icon="manage_accounts"
title="Create Custom GCP Role"
text="Checkout this documentation on how to create a custom role and assign it to the service account."
link="/connectors/database/bigquery/roles"
/>
## Metadata Ingestion ## Metadata Ingestion
### 1. Visit the Services Page ### 1. Visit the Services Page

87
openmetadata-docs/content/connectors/database/bigquery/roles.md Normal file
View File

@ -0,0 +1,87 @@
---
title: BigQuery
slug: /connectors/database/bigquery/roles
---
# Create custom role in GCP
This documentation will guide you on how to create a custom role in GCP with the necessary permissions to ingest BigQuery in OpenMetadata.
## Step 1: Navigate to Roles
Search for `Roles` in your GCP console and select the first result under `IAM & Roles` section.
<Image
src="/images/openmetadata/connectors/bigquery/create-role-1.png"
alt="Navigate to Roles"
caption="Navigate to Roles"
/>
## Step 2: Create Role & Add Permissions
Below the search bar you should see a `Create Role` button click on that & navigate to create role page.
<Image
src="/images/openmetadata/connectors/bigquery/create-role-2.png"
alt="Create Role Button"
caption="Create Role"
/>
Once You are on the create role page, you can edit the description & title of the role and finally you can click on add permissions to grant permissions to role.
<Image
src="/images/openmetadata/connectors/bigquery/create-role-3.png"
alt="Create Role"
caption="Create Role"
/>
You can search for the required permissions in the filter box and add them accordingly. To ingest metadata from BigQuery you need to grant the following permissions to the user.
<Table>
| # | GCP Permission | Required For |
| :--- | :---------------------------- | :---------------------- |
| 1 | bigquery.datasets.get | Metadata Ingestion |
| 2 | bigquery.tables.get | Metadata Ingestion |
| 3 | bigquery.tables.getData | Metadata Ingestion |
| 4 | bigquery.tables.list | Metadata Ingestion |
| 5 | resourcemanager.projects.get | Metadata Ingestion |
| 6 | bigquery.jobs.create | Metadata Ingestion |
| 7 | bigquery.jobs.listAll | Metadata Ingestion |
| 8 | datacatalog.taxonomies.get | Fetch Policy Tags |
| 9 | datacatalog.taxonomies.list | Fetch Policy Tags |
| 10 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow |
| 11 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow |
</Table>
<Image
src="/images/openmetadata/connectors/bigquery/create-role-4.png"
alt="Add Permissions"
caption="Add Permissions"
/>
Once you have added all the required permissions, you can create the role by clicking on the create button.
<Image
src="/images/openmetadata/connectors/bigquery/create-role-5.png"
alt="Add Permissions"
caption="Add Permissions"
/>
## Step 3: Assign Role to Service Account
To assign the created role, you can navigate to `IAM` and click on `Grant Access` and you can search your service account in the `Add Principals` section & Assign the created role to the service account.
<Image
src="/images/openmetadata/connectors/bigquery/create-role-6.png"
alt="Add Permissions"
caption="Add Permissions"
/>

6
openmetadata-docs/content/deployment/security/keycloak/index.md
View File

@ -55,6 +55,12 @@ Security requirements for your **production** environment:
- Click on `Save` button. - Click on `Save` button.
<Note>
Note: Scopes `openid`, `email` & `profile` are required to fetch the user details so you will have to add these scopes in your client.
</Note>
<Note> <Note>

2
openmetadata-docs/content/menu.md
View File

@ -206,6 +206,8 @@ site_menu:
url: /connectors/database/bigquery/airflow url: /connectors/database/bigquery/airflow
- category: Connectors / Database / BigQuery / CLI - category: Connectors / Database / BigQuery / CLI
url: /connectors/database/bigquery/cli url: /connectors/database/bigquery/cli
- category: Connectors / Database / BigQuery / Roles
url: /connectors/database/bigquery/roles
- category: Connectors / Database / Clickhouse - category: Connectors / Database / Clickhouse
url: /connectors/database/clickhouse url: /connectors/database/clickhouse
- category: Connectors / Database / Clickhouse / Airflow - category: Connectors / Database / Clickhouse / Airflow

BIN
openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-1.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

BIN
openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 178 KiB

BIN
openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-3.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 354 KiB

BIN
openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-4.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 596 KiB

BIN
openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-5.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 634 KiB

BIN
openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-6.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 118 KiB