diff --git a/openmetadata-docs/content/v1.4.x/connectors/database/bigquery/create-credentials.md b/openmetadata-docs/content/v1.4.x/connectors/database/bigquery/create-credentials.md index 49eb554bef9..7de86b5d087 100644 --- a/openmetadata-docs/content/v1.4.x/connectors/database/bigquery/create-credentials.md +++ b/openmetadata-docs/content/v1.4.x/connectors/database/bigquery/create-credentials.md @@ -3,6 +3,73 @@ title: BigQuery slug: /connectors/database/bigquery/create-credentials --- +# Create custom role in GCP + +This documentation will guide you on how to create a custom role in GCP with the necessary permissions to ingest BigQuery in OpenMetadata. + +## Step 1: Navigate to Roles + +Search for `Roles` in your GCP console and select the first result under `IAM & Roles` section. + +{% image +src="/images/v1.4/connectors/bigquery/create-role-1.png" +alt="Navigate to Roles" +caption="Navigate to Roles" /%} + + +## Step 2: Create Role & Add Permissions + +Below the search bar you should see a `Create Role` button click on that & navigate to create role page. + + +{% image +src="/images/v1.4/connectors/bigquery/create-role-2.png" +alt="Create Role Button" +caption="Create Role" /%} + + + +Once You are on the create role page, you can edit the description & title of the role and finally you can click on `add permissions` to grant permissions to role. + +{% image +src="/images/v1.4/connectors/bigquery/create-role-3.png" +alt="Create Role" +caption="Create Role" /%} + + +You can search for the required permissions in the filter box and add them accordingly. To ingest metadata from BigQuery you need to grant the following permissions to the user. + + + +| # | GCP Permission | Required For | +| :--- | :---------------------------- | :---------------------- | +| 1 | bigquery.datasets.get | Metadata Ingestion | +| 2 | bigquery.tables.get | Metadata Ingestion | +| 3 | bigquery.tables.getData | Metadata Ingestion | +| 4 | bigquery.tables.list | Metadata Ingestion | +| 5 | resourcemanager.projects.get | Metadata Ingestion | +| 6 | bigquery.jobs.create | Metadata Ingestion | +| 7 | bigquery.jobs.listAll | Metadata Ingestion | +| 8 | bigquery.routines.get | Stored Procedure | +| 9 | bigquery.routines.list | Stored Procedure | +| 10 | datacatalog.taxonomies.get | Fetch Policy Tags | +| 11 | datacatalog.taxonomies.list | Fetch Policy Tags | +| 12 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow | +| 13 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow | +| 14 | logging.operations.list | Incremental Metadata Ingestion | + +{% image +src="/images/v1.4/connectors/bigquery/create-role-4.png" +alt="Add Permissions" +caption="Add Permissions" /%} + +Once you have added all the required permissions, you can create the role by clicking on the create button. + +{% image +src="/images/v1.4/connectors/bigquery/create-role-5.png" +alt="Add Permissions" +caption="Add Permissions" /%} + # Create Service Account & Credential This documentation will guide you on how to create a service account in GCP and create credentials to access the same. diff --git a/openmetadata-docs/content/v1.4.x/connectors/database/bigquery/index.md b/openmetadata-docs/content/v1.4.x/connectors/database/bigquery/index.md index c0d3a707927..dafba4cc6c1 100644 --- a/openmetadata-docs/content/v1.4.x/connectors/database/bigquery/index.md +++ b/openmetadata-docs/content/v1.4.x/connectors/database/bigquery/index.md @@ -33,6 +33,16 @@ Configure and schedule BigQuery metadata and profiler workflows from the OpenMet You need to create an service account in order to ingest metadata from bigquery refer [this](/connectors/database/bigquery/create-credentials) guide on how to create service account. +{% tilesContainer %} +{% tile +icon="manage_accounts" +title="Create Credentials" +description="Checkout this documentation on how to create a custom role and assign it to the service account." +link="/connectors/database/bigquery/create-credentials" + / %} +{% /tilesContainer %} + + ### Data Catalog API Permissions - Go to [https://console.cloud.google.com/apis/library/datacatalog.googleapis.com](https://console.cloud.google.com/apis/library/datacatalog.googleapis.com) @@ -60,6 +70,7 @@ To execute metadata extraction and usage workflow successfully the user or the s | 11 | datacatalog.taxonomies.list | Fetch Policy Tags | | 12 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow | | 13 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow | +| 14 | logging.operations.list | Incremental Metadata Ingestion | {% /multiTablesWrapper %} @@ -67,16 +78,6 @@ To execute metadata extraction and usage workflow successfully the user or the s If the user has `External Tables`, please attach relevant permissions needed for external tables, alongwith the above list of permissions. {% /note %} -{% tilesContainer %} -{% tile -icon="manage_accounts" -title="Create Custom GCP Role" -description="Checkout this documentation on how to create a custom role and assign it to the service account." -link="/connectors/database/bigquery/roles" - / %} -{% /tilesContainer %} - - {% note %} If you are using BigQuery and have sharded tables, you might want to consider using partitioned tables instead. Partitioned tables allow you to efficiently query data by date or other criteria, without having to manage multiple tables. Partitioned tables also have lower storage and query costs than sharded tables. You can learn more about the benefits of partitioned tables [here](https://cloud.google.com/bigquery/docs/partitioned-tables#dt_partition_shard). diff --git a/openmetadata-docs/content/v1.4.x/connectors/database/bigquery/roles.md b/openmetadata-docs/content/v1.4.x/connectors/database/bigquery/roles.md deleted file mode 100644 index 45ec663e032..00000000000 --- a/openmetadata-docs/content/v1.4.x/connectors/database/bigquery/roles.md +++ /dev/null @@ -1,81 +0,0 @@ ---- -title: BigQuery -slug: /connectors/database/bigquery/roles ---- - -# Create custom role in GCP - -This documentation will guide you on how to create a custom role in GCP with the necessary permissions to ingest BigQuery in OpenMetadata. - - -## Step 1: Navigate to Roles - -Search for `Roles` in your GCP console and select the first result under `IAM & Roles` section. - -{% image -src="/images/v1.4/connectors/bigquery/create-role-1.png" -alt="Navigate to Roles" -caption="Navigate to Roles" /%} - - -## Step 2: Create Role & Add Permissions - -Below the search bar you should see a `Create Role` button click on that & navigate to create role page. - - -{% image -src="/images/v1.4/connectors/bigquery/create-role-2.png" -alt="Create Role Button" -caption="Create Role" /%} - - - -Once You are on the create role page, you can edit the description & title of the role and finally you can click on add permissions to grant permissions to role. - -{% image -src="/images/v1.4/connectors/bigquery/create-role-3.png" -alt="Create Role" -caption="Create Role" /%} - - -You can search for the required permissions in the filter box and add them accordingly. To ingest metadata from BigQuery you need to grant the following permissions to the user. - - - -| # | GCP Permission | Required For | -| :--- | :---------------------------- | :---------------------- | -| 1 | bigquery.datasets.get | Metadata Ingestion | -| 2 | bigquery.tables.get | Metadata Ingestion | -| 3 | bigquery.tables.getData | Metadata Ingestion | -| 4 | bigquery.tables.list | Metadata Ingestion | -| 5 | resourcemanager.projects.get | Metadata Ingestion | -| 6 | bigquery.jobs.create | Metadata Ingestion | -| 7 | bigquery.jobs.listAll | Metadata Ingestion | -| 8 | bigquery.routines.get | Stored Procedure | -| 9 | bigquery.routines.list | Stored Procedure | -| 10 | datacatalog.taxonomies.get | Fetch Policy Tags | -| 11 | datacatalog.taxonomies.list | Fetch Policy Tags | -| 12 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow | -| 13 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow | - -{% image -src="/images/v1.4/connectors/bigquery/create-role-4.png" -alt="Add Permissions" -caption="Add Permissions" /%} - -Once you have added all the required permissions, you can create the role by clicking on the create button. - -{% image -src="/images/v1.4/connectors/bigquery/create-role-5.png" -alt="Add Permissions" -caption="Add Permissions" /%} - - -## Step 3: Assign Role to Service Account - -To assign the created role, you can navigate to `IAM` and click on `Grant Access` and you can search your service account in the `Add Principals` section & Assign the created role to the service account. - -{% image -src="/images/v1.4/connectors/bigquery/create-role-6.png" -alt="Add Permissions" -caption="Add Permissions" /%} diff --git a/openmetadata-docs/content/v1.4.x/menu.md b/openmetadata-docs/content/v1.4.x/menu.md index ce9302f54b3..b0167bced05 100644 --- a/openmetadata-docs/content/v1.4.x/menu.md +++ b/openmetadata-docs/content/v1.4.x/menu.md @@ -506,8 +506,6 @@ site_menu: url: /connectors/database/bigquery - category: Connectors / Database / BigQuery / Run Externally url: /connectors/database/bigquery/yaml - - category: Connectors / Database / BigQuery / Roles - url: /connectors/database/bigquery/roles - category: Connectors / Database / BigQuery / Create Credentials url: /connectors/database/bigquery/create-credentials - category: Connectors / Database / BigTable diff --git a/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/create-credentials.md b/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/create-credentials.md index 49eb554bef9..7de86b5d087 100644 --- a/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/create-credentials.md +++ b/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/create-credentials.md @@ -3,6 +3,73 @@ title: BigQuery slug: /connectors/database/bigquery/create-credentials --- +# Create custom role in GCP + +This documentation will guide you on how to create a custom role in GCP with the necessary permissions to ingest BigQuery in OpenMetadata. + +## Step 1: Navigate to Roles + +Search for `Roles` in your GCP console and select the first result under `IAM & Roles` section. + +{% image +src="/images/v1.4/connectors/bigquery/create-role-1.png" +alt="Navigate to Roles" +caption="Navigate to Roles" /%} + + +## Step 2: Create Role & Add Permissions + +Below the search bar you should see a `Create Role` button click on that & navigate to create role page. + + +{% image +src="/images/v1.4/connectors/bigquery/create-role-2.png" +alt="Create Role Button" +caption="Create Role" /%} + + + +Once You are on the create role page, you can edit the description & title of the role and finally you can click on `add permissions` to grant permissions to role. + +{% image +src="/images/v1.4/connectors/bigquery/create-role-3.png" +alt="Create Role" +caption="Create Role" /%} + + +You can search for the required permissions in the filter box and add them accordingly. To ingest metadata from BigQuery you need to grant the following permissions to the user. + + + +| # | GCP Permission | Required For | +| :--- | :---------------------------- | :---------------------- | +| 1 | bigquery.datasets.get | Metadata Ingestion | +| 2 | bigquery.tables.get | Metadata Ingestion | +| 3 | bigquery.tables.getData | Metadata Ingestion | +| 4 | bigquery.tables.list | Metadata Ingestion | +| 5 | resourcemanager.projects.get | Metadata Ingestion | +| 6 | bigquery.jobs.create | Metadata Ingestion | +| 7 | bigquery.jobs.listAll | Metadata Ingestion | +| 8 | bigquery.routines.get | Stored Procedure | +| 9 | bigquery.routines.list | Stored Procedure | +| 10 | datacatalog.taxonomies.get | Fetch Policy Tags | +| 11 | datacatalog.taxonomies.list | Fetch Policy Tags | +| 12 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow | +| 13 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow | +| 14 | logging.operations.list | Incremental Metadata Ingestion | + +{% image +src="/images/v1.4/connectors/bigquery/create-role-4.png" +alt="Add Permissions" +caption="Add Permissions" /%} + +Once you have added all the required permissions, you can create the role by clicking on the create button. + +{% image +src="/images/v1.4/connectors/bigquery/create-role-5.png" +alt="Add Permissions" +caption="Add Permissions" /%} + # Create Service Account & Credential This documentation will guide you on how to create a service account in GCP and create credentials to access the same. diff --git a/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/index.md b/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/index.md index 7e5da73943e..9f9a8c9d4bb 100644 --- a/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/index.md +++ b/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/index.md @@ -33,6 +33,15 @@ Configure and schedule BigQuery metadata and profiler workflows from the OpenMet You need to create an service account in order to ingest metadata from bigquery refer [this](/connectors/database/bigquery/create-credentials) guide on how to create service account. +{% tilesContainer %} +{% tile +icon="manage_accounts" +title="Create Custom GCP Role" +description="Checkout this documentation on how to create a custom role and assign it to the service account." +link="/connectors/database/bigquery/roles" + / %} +{% /tilesContainer %} + ### Data Catalog API Permissions - Go to [https://console.cloud.google.com/apis/library/datacatalog.googleapis.com](https://console.cloud.google.com/apis/library/datacatalog.googleapis.com) @@ -60,6 +69,7 @@ To execute metadata extraction and usage workflow successfully the user or the s | 11 | datacatalog.taxonomies.list | Fetch Policy Tags | | 12 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow | | 13 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow | +| 14 | logging.operations.list | Incremental Metadata Ingestion | {% /multiTablesWrapper %} @@ -67,16 +77,6 @@ To execute metadata extraction and usage workflow successfully the user or the s If the user has `External Tables`, please attach relevant permissions needed for external tables, alongwith the above list of permissions. {% /note %} -{% tilesContainer %} -{% tile -icon="manage_accounts" -title="Create Custom GCP Role" -description="Checkout this documentation on how to create a custom role and assign it to the service account." -link="/connectors/database/bigquery/roles" - / %} -{% /tilesContainer %} - - {% note %} If you are using BigQuery and have sharded tables, you might want to consider using partitioned tables instead. Partitioned tables allow you to efficiently query data by date or other criteria, without having to manage multiple tables. Partitioned tables also have lower storage and query costs than sharded tables. You can learn more about the benefits of partitioned tables [here](https://cloud.google.com/bigquery/docs/partitioned-tables#dt_partition_shard). diff --git a/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/roles.md b/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/roles.md deleted file mode 100644 index 740c21a356a..00000000000 --- a/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/database/bigquery/roles.md +++ /dev/null @@ -1,81 +0,0 @@ ---- -title: BigQuery -slug: /connectors/database/bigquery/roles ---- - -# Create custom role in GCP - -This documentation will guide you on how to create a custom role in GCP with the necessary permissions to ingest BigQuery in OpenMetadata. - - -## Step 1: Navigate to Roles - -Search for `Roles` in your GCP console and select the first result under `IAM & Roles` section. - -{% image -src="/images/v1.5/connectors/bigquery/create-role-1.png" -alt="Navigate to Roles" -caption="Navigate to Roles" /%} - - -## Step 2: Create Role & Add Permissions - -Below the search bar you should see a `Create Role` button click on that & navigate to create role page. - - -{% image -src="/images/v1.5/connectors/bigquery/create-role-2.png" -alt="Create Role Button" -caption="Create Role" /%} - - - -Once You are on the create role page, you can edit the description & title of the role and finally you can click on add permissions to grant permissions to role. - -{% image -src="/images/v1.5/connectors/bigquery/create-role-3.png" -alt="Create Role" -caption="Create Role" /%} - - -You can search for the required permissions in the filter box and add them accordingly. To ingest metadata from BigQuery you need to grant the following permissions to the user. - - - -| # | GCP Permission | Required For | -| :--- | :---------------------------- | :---------------------- | -| 1 | bigquery.datasets.get | Metadata Ingestion | -| 2 | bigquery.tables.get | Metadata Ingestion | -| 3 | bigquery.tables.getData | Metadata Ingestion | -| 4 | bigquery.tables.list | Metadata Ingestion | -| 5 | resourcemanager.projects.get | Metadata Ingestion | -| 6 | bigquery.jobs.create | Metadata Ingestion | -| 7 | bigquery.jobs.listAll | Metadata Ingestion | -| 8 | bigquery.routines.get | Stored Procedure | -| 9 | bigquery.routines.list | Stored Procedure | -| 10 | datacatalog.taxonomies.get | Fetch Policy Tags | -| 11 | datacatalog.taxonomies.list | Fetch Policy Tags | -| 12 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow | -| 13 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow | - -{% image -src="/images/v1.5/connectors/bigquery/create-role-4.png" -alt="Add Permissions" -caption="Add Permissions" /%} - -Once you have added all the required permissions, you can create the role by clicking on the create button. - -{% image -src="/images/v1.5/connectors/bigquery/create-role-5.png" -alt="Add Permissions" -caption="Add Permissions" /%} - - -## Step 3: Assign Role to Service Account - -To assign the created role, you can navigate to `IAM` and click on `Grant Access` and you can search your service account in the `Add Principals` section & Assign the created role to the service account. - -{% image -src="/images/v1.5/connectors/bigquery/create-role-6.png" -alt="Add Permissions" -caption="Add Permissions" /%} diff --git a/openmetadata-docs/content/v1.5.x-SNAPSHOT/menu.md b/openmetadata-docs/content/v1.5.x-SNAPSHOT/menu.md index b5e3f524057..2f69708d85d 100644 --- a/openmetadata-docs/content/v1.5.x-SNAPSHOT/menu.md +++ b/openmetadata-docs/content/v1.5.x-SNAPSHOT/menu.md @@ -463,8 +463,6 @@ site_menu: url: /connectors/database/bigquery - category: Connectors / Database / BigQuery / Run Externally url: /connectors/database/bigquery/yaml - - category: Connectors / Database / BigQuery / Roles - url: /connectors/database/bigquery/roles - category: Connectors / Database / BigQuery / Create Credentials url: /connectors/database/bigquery/create-credentials - category: Connectors / Database / BigTable