yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-12-12 15:57:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

MINOR: Improve BigQuery Create Creds Docs (#16807)

Browse Source
This commit is contained in:
Mayur Singal 2024-06-26 20:54:19 +05:30 committed by GitHub
parent bd2bf4a044
commit d083bd1365
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
8 changed files with 155 additions and 186 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

67
openmetadata-docs/content/v1.4.x/connectors/database/bigquery/create-credentials.md
View File

@ -3,6 +3,73 @@ title: BigQuery
slug: /connectors/database/bigquery/create-credentials
---
# Create custom role in GCP
This documentation will guide you on how to create a custom role in GCP with the necessary permissions to ingest BigQuery in OpenMetadata.
## Step 1: Navigate to Roles
Search for `Roles` in your GCP console and select the first result under `IAM & Roles` section.
{% image
src="/images/v1.4/connectors/bigquery/create-role-1.png"
alt="Navigate to Roles"
caption="Navigate to Roles" /%}
## Step 2: Create Role & Add Permissions
Below the search bar you should see a `Create Role` button click on that & navigate to create role page.
{% image
src="/images/v1.4/connectors/bigquery/create-role-2.png"
alt="Create Role Button"
caption="Create Role" /%}
Once You are on the create role page, you can edit the description & title of the role and finally you can click on `add permissions` to grant permissions to role.
{% image
src="/images/v1.4/connectors/bigquery/create-role-3.png"
alt="Create Role"
caption="Create Role" /%}
You can search for the required permissions in the filter box and add them accordingly. To ingest metadata from BigQuery you need to grant the following permissions to the user.
| # | GCP Permission | Required For |
| :--- | :---------------------------- | :---------------------- |
| 1 | bigquery.datasets.get | Metadata Ingestion |
| 2 | bigquery.tables.get | Metadata Ingestion |
| 3 | bigquery.tables.getData | Metadata Ingestion |
| 4 | bigquery.tables.list | Metadata Ingestion |
| 5 | resourcemanager.projects.get | Metadata Ingestion |
| 6 | bigquery.jobs.create | Metadata Ingestion |
| 7 | bigquery.jobs.listAll | Metadata Ingestion |
| 8 | bigquery.routines.get | Stored Procedure |
| 9 | bigquery.routines.list | Stored Procedure |
| 10 | datacatalog.taxonomies.get | Fetch Policy Tags |
| 11 | datacatalog.taxonomies.list | Fetch Policy Tags |
| 12 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow |
| 13 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow |
| 14 | logging.operations.list | Incremental Metadata Ingestion |
{% image
src="/images/v1.4/connectors/bigquery/create-role-4.png"
alt="Add Permissions"
caption="Add Permissions" /%}
Once you have added all the required permissions, you can create the role by clicking on the create button.
{% image
src="/images/v1.4/connectors/bigquery/create-role-5.png"
alt="Add Permissions"
caption="Add Permissions" /%}
# Create Service Account & Credential
This documentation will guide you on how to create a service account in GCP and create credentials to access the same.

21
openmetadata-docs/content/v1.4.x/connectors/database/bigquery/index.md
View File

@ -33,6 +33,16 @@ Configure and schedule BigQuery metadata and profiler workflows from the OpenMet
You need to create an service account in order to ingest metadata from bigquery refer [this](/connectors/database/bigquery/create-credentials) guide on how to create service account.
{% tilesContainer %}
{% tile
icon="manage_accounts"
title="Create Credentials"
description="Checkout this documentation on how to create a custom role and assign it to the service account."
link="/connectors/database/bigquery/create-credentials"
/ %}
{% /tilesContainer %}
### Data Catalog API Permissions
- Go to [https://console.cloud.google.com/apis/library/datacatalog.googleapis.com](https://console.cloud.google.com/apis/library/datacatalog.googleapis.com)
@ -60,6 +70,7 @@ To execute metadata extraction and usage workflow successfully the user or the s
| 11 | datacatalog.taxonomies.list | Fetch Policy Tags |
| 12 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow |
| 13 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow |
| 14 | logging.operations.list | Incremental Metadata Ingestion |
{% /multiTablesWrapper %}
@ -67,16 +78,6 @@ To execute metadata extraction and usage workflow successfully the user or the s
If the user has `External Tables`, please attach relevant permissions needed for external tables, alongwith the above list of permissions.
{% /note %}
{% tilesContainer %}
{% tile
icon="manage_accounts"
title="Create Custom GCP Role"
description="Checkout this documentation on how to create a custom role and assign it to the service account."
link="/connectors/database/bigquery/roles"
/ %}
{% /tilesContainer %}
{% note %}
If you are using BigQuery and have sharded tables, you might want to consider using partitioned tables instead. Partitioned tables allow you to efficiently query data by date or other criteria, without having to manage multiple tables. Partitioned tables also have lower storage and query costs than sharded tables.
You can learn more about the benefits of partitioned tables [here](https://cloud.google.com/bigquery/docs/partitioned-tables#dt_partition_shard).

81
openmetadata-docs/content/v1.4.x/connectors/database/bigquery/roles.md
View File

@ -1,81 +0,0 @@
---
title: BigQuery
slug: /connectors/database/bigquery/roles
---
# Create custom role in GCP
This documentation will guide you on how to create a custom role in GCP with the necessary permissions to ingest BigQuery in OpenMetadata.
## Step 1: Navigate to Roles
Search for `Roles` in your GCP console and select the first result under `IAM & Roles` section.
{% image
src="/images/v1.4/connectors/bigquery/create-role-1.png"
alt="Navigate to Roles"
caption="Navigate to Roles" /%}
## Step 2: Create Role & Add Permissions
Below the search bar you should see a `Create Role` button click on that & navigate to create role page.
{% image
src="/images/v1.4/connectors/bigquery/create-role-2.png"
alt="Create Role Button"
caption="Create Role" /%}
Once You are on the create role page, you can edit the description & title of the role and finally you can click on add permissions to grant permissions to role.
{% image
src="/images/v1.4/connectors/bigquery/create-role-3.png"
alt="Create Role"
caption="Create Role" /%}
You can search for the required permissions in the filter box and add them accordingly. To ingest metadata from BigQuery you need to grant the following permissions to the user.
| # | GCP Permission | Required For |
| :--- | :---------------------------- | :---------------------- |
| 1 | bigquery.datasets.get | Metadata Ingestion |
| 2 | bigquery.tables.get | Metadata Ingestion |
| 3 | bigquery.tables.getData | Metadata Ingestion |
| 4 | bigquery.tables.list | Metadata Ingestion |
| 5 | resourcemanager.projects.get | Metadata Ingestion |
| 6 | bigquery.jobs.create | Metadata Ingestion |
| 7 | bigquery.jobs.listAll | Metadata Ingestion |
| 8 | bigquery.routines.get | Stored Procedure |
| 9 | bigquery.routines.list | Stored Procedure |
| 10 | datacatalog.taxonomies.get | Fetch Policy Tags |
| 11 | datacatalog.taxonomies.list | Fetch Policy Tags |
| 12 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow |
| 13 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow |
{% image
src="/images/v1.4/connectors/bigquery/create-role-4.png"
alt="Add Permissions"
caption="Add Permissions" /%}
Once you have added all the required permissions, you can create the role by clicking on the create button.
{% image
src="/images/v1.4/connectors/bigquery/create-role-5.png"
alt="Add Permissions"
caption="Add Permissions" /%}
## Step 3: Assign Role to Service Account
To assign the created role, you can navigate to `IAM` and click on `Grant Access` and you can search your service account in the `Add Principals` section & Assign the created role to the service account.
{% image
src="/images/v1.4/connectors/bigquery/create-role-6.png"
alt="Add Permissions"
caption="Add Permissions" /%}

2
openmetadata-docs/content/v1.4.x/menu.md
View File

@ -506,8 +506,6 @@ site_menu:
url: /connectors/database/bigquery
- category: Connectors / Database / BigQuery / Run Externally
url: /connectors/database/bigquery/yaml
- category: Connectors / Database / BigQuery / Roles
url: /connectors/database/bigquery/roles
- category: Connectors / Database / BigQuery / Create Credentials
url: /connectors/database/bigquery/create-credentials
- category: Connectors / Database / BigTable

67
openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/create-credentials.md
View File

@ -3,6 +3,73 @@ title: BigQuery
slug: /connectors/database/bigquery/create-credentials
---
# Create custom role in GCP
This documentation will guide you on how to create a custom role in GCP with the necessary permissions to ingest BigQuery in OpenMetadata.
## Step 1: Navigate to Roles
Search for `Roles` in your GCP console and select the first result under `IAM & Roles` section.
{% image
src="/images/v1.4/connectors/bigquery/create-role-1.png"
alt="Navigate to Roles"
caption="Navigate to Roles" /%}
## Step 2: Create Role & Add Permissions
Below the search bar you should see a `Create Role` button click on that & navigate to create role page.
{% image
src="/images/v1.4/connectors/bigquery/create-role-2.png"
alt="Create Role Button"
caption="Create Role" /%}
Once You are on the create role page, you can edit the description & title of the role and finally you can click on `add permissions` to grant permissions to role.
{% image
src="/images/v1.4/connectors/bigquery/create-role-3.png"
alt="Create Role"
caption="Create Role" /%}
You can search for the required permissions in the filter box and add them accordingly. To ingest metadata from BigQuery you need to grant the following permissions to the user.
| # | GCP Permission | Required For |
| :--- | :---------------------------- | :---------------------- |
| 1 | bigquery.datasets.get | Metadata Ingestion |
| 2 | bigquery.tables.get | Metadata Ingestion |
| 3 | bigquery.tables.getData | Metadata Ingestion |
| 4 | bigquery.tables.list | Metadata Ingestion |
| 5 | resourcemanager.projects.get | Metadata Ingestion |
| 6 | bigquery.jobs.create | Metadata Ingestion |
| 7 | bigquery.jobs.listAll | Metadata Ingestion |
| 8 | bigquery.routines.get | Stored Procedure |
| 9 | bigquery.routines.list | Stored Procedure |
| 10 | datacatalog.taxonomies.get | Fetch Policy Tags |
| 11 | datacatalog.taxonomies.list | Fetch Policy Tags |
| 12 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow |
| 13 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow |
| 14 | logging.operations.list | Incremental Metadata Ingestion |
{% image
src="/images/v1.4/connectors/bigquery/create-role-4.png"
alt="Add Permissions"
caption="Add Permissions" /%}
Once you have added all the required permissions, you can create the role by clicking on the create button.
{% image
src="/images/v1.4/connectors/bigquery/create-role-5.png"
alt="Add Permissions"
caption="Add Permissions" /%}
# Create Service Account & Credential
This documentation will guide you on how to create a service account in GCP and create credentials to access the same.

20
openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/index.md
View File

@ -33,6 +33,15 @@ Configure and schedule BigQuery metadata and profiler workflows from the OpenMet
You need to create an service account in order to ingest metadata from bigquery refer [this](/connectors/database/bigquery/create-credentials) guide on how to create service account.
{% tilesContainer %}
{% tile
icon="manage_accounts"
title="Create Custom GCP Role"
description="Checkout this documentation on how to create a custom role and assign it to the service account."
link="/connectors/database/bigquery/roles"
/ %}
{% /tilesContainer %}
### Data Catalog API Permissions
- Go to [https://console.cloud.google.com/apis/library/datacatalog.googleapis.com](https://console.cloud.google.com/apis/library/datacatalog.googleapis.com)
@ -60,6 +69,7 @@ To execute metadata extraction and usage workflow successfully the user or the s
| 11 | datacatalog.taxonomies.list | Fetch Policy Tags |
| 12 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow |
| 13 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow |
| 14 | logging.operations.list | Incremental Metadata Ingestion |
{% /multiTablesWrapper %}
@ -67,16 +77,6 @@ To execute metadata extraction and usage workflow successfully the user or the s
If the user has `External Tables`, please attach relevant permissions needed for external tables, alongwith the above list of permissions.
{% /note %}
{% tilesContainer %}
{% tile
icon="manage_accounts"
title="Create Custom GCP Role"
description="Checkout this documentation on how to create a custom role and assign it to the service account."
link="/connectors/database/bigquery/roles"
/ %}
{% /tilesContainer %}
{% note %}
If you are using BigQuery and have sharded tables, you might want to consider using partitioned tables instead. Partitioned tables allow you to efficiently query data by date or other criteria, without having to manage multiple tables. Partitioned tables also have lower storage and query costs than sharded tables.
You can learn more about the benefits of partitioned tables [here](https://cloud.google.com/bigquery/docs/partitioned-tables#dt_partition_shard).

81
openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/roles.md
View File

@ -1,81 +0,0 @@
---
title: BigQuery
slug: /connectors/database/bigquery/roles
---
# Create custom role in GCP
This documentation will guide you on how to create a custom role in GCP with the necessary permissions to ingest BigQuery in OpenMetadata.
## Step 1: Navigate to Roles
Search for `Roles` in your GCP console and select the first result under `IAM & Roles` section.
{% image
src="/images/v1.5/connectors/bigquery/create-role-1.png"
alt="Navigate to Roles"
caption="Navigate to Roles" /%}
## Step 2: Create Role & Add Permissions
Below the search bar you should see a `Create Role` button click on that & navigate to create role page.
{% image
src="/images/v1.5/connectors/bigquery/create-role-2.png"
alt="Create Role Button"
caption="Create Role" /%}
Once You are on the create role page, you can edit the description & title of the role and finally you can click on add permissions to grant permissions to role.
{% image
src="/images/v1.5/connectors/bigquery/create-role-3.png"
alt="Create Role"
caption="Create Role" /%}
You can search for the required permissions in the filter box and add them accordingly. To ingest metadata from BigQuery you need to grant the following permissions to the user.
| # | GCP Permission | Required For |
| :--- | :---------------------------- | :---------------------- |
| 1 | bigquery.datasets.get | Metadata Ingestion |
| 2 | bigquery.tables.get | Metadata Ingestion |
| 3 | bigquery.tables.getData | Metadata Ingestion |
| 4 | bigquery.tables.list | Metadata Ingestion |
| 5 | resourcemanager.projects.get | Metadata Ingestion |
| 6 | bigquery.jobs.create | Metadata Ingestion |
| 7 | bigquery.jobs.listAll | Metadata Ingestion |
| 8 | bigquery.routines.get | Stored Procedure |
| 9 | bigquery.routines.list | Stored Procedure |
| 10 | datacatalog.taxonomies.get | Fetch Policy Tags |
| 11 | datacatalog.taxonomies.list | Fetch Policy Tags |
| 12 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow |
| 13 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow |
{% image
src="/images/v1.5/connectors/bigquery/create-role-4.png"
alt="Add Permissions"
caption="Add Permissions" /%}
Once you have added all the required permissions, you can create the role by clicking on the create button.
{% image
src="/images/v1.5/connectors/bigquery/create-role-5.png"
alt="Add Permissions"
caption="Add Permissions" /%}
## Step 3: Assign Role to Service Account
To assign the created role, you can navigate to `IAM` and click on `Grant Access` and you can search your service account in the `Add Principals` section & Assign the created role to the service account.
{% image
src="/images/v1.5/connectors/bigquery/create-role-6.png"
alt="Add Permissions"
caption="Add Permissions" /%}

2
openmetadata-docs/content/v1.5.x-SNAPSHOT/menu.md
View File

@ -463,8 +463,6 @@ site_menu:
url: /connectors/database/bigquery
- category: Connectors / Database / BigQuery / Run Externally
url: /connectors/database/bigquery/yaml
- category: Connectors / Database / BigQuery / Roles
url: /connectors/database/bigquery/roles
- category: Connectors / Database / BigQuery / Create Credentials
url: /connectors/database/bigquery/create-credentials
- category: Connectors / Database / BigTable