diff --git a/openmetadata-docs/content/connectors/database/bigquery/airflow.md b/openmetadata-docs/content/connectors/database/bigquery/airflow.md index 73aff176df4..c35d77d115a 100644 --- a/openmetadata-docs/content/connectors/database/bigquery/airflow.md +++ b/openmetadata-docs/content/connectors/database/bigquery/airflow.md @@ -57,25 +57,34 @@ pip3 install "openmetadata-ingestion[bigquery-usage]"

GCP Permissions

To execute metadata extraction and usage workflow successfully the user or the service account should have enough access to fetch required data. Following table describes the minimum required permissions

+ -| # | GCP Permission | GCP Role | Required For | -| :---------- | :---------- | :---------- | :---------- | -| 1 | bigquery.datasets.get | BigQuery Data Viewer | Metadata Ingestion | -| 2 | bigquery.tables.get | BigQuery Data Viewer | Metadata Ingestion | -| 3 | bigquery.tables.getData | BigQuery Data Viewer | Metadata Ingestion | -| 4 | bigquery.tables.list | BigQuery Data Viewer | Metadata Ingestion | -| 5 | resourcemanager.projects.get | BigQuery Data Viewer | Metadata Ingestion | -| 6 | bigquery.jobs.create | BigQuery Job User | Metadata Ingestion | -| 7 | bigquery.jobs.listAll | BigQuery Job User | Metadata Ingestion | -| 8 | datacatalog.taxonomies.get | BigQuery Policy Admin | Fetch Policy Tags | -| 9 | datacatalog.taxonomies.list | BigQuery Policy Admin | Fetch Policy Tags | -| 10 | bigquery.readsessions.create | BigQuery Admin | Bigquery Usage Workflow | -| 11 | bigquery.readsessions.getData | BigQuery Admin | Bigquery Usage Workflow | +| # | GCP Permission | Required For | +| :--- | :---------------------------- | :---------------------- | +| 1 | bigquery.datasets.get | Metadata Ingestion | +| 2 | bigquery.tables.get | Metadata Ingestion | +| 3 | bigquery.tables.getData | Metadata Ingestion | +| 4 | bigquery.tables.list | Metadata Ingestion | +| 5 | resourcemanager.projects.get | Metadata Ingestion | +| 6 | bigquery.jobs.create | Metadata Ingestion | +| 7 | bigquery.jobs.listAll | Metadata Ingestion | +| 8 | datacatalog.taxonomies.get | Fetch Policy Tags | +| 9 | datacatalog.taxonomies.list | Fetch Policy Tags | +| 10 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow | +| 11 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow |
+ + + ## Metadata Ingestion All connectors are defined as JSON Schemas. diff --git a/openmetadata-docs/content/connectors/database/bigquery/cli.md b/openmetadata-docs/content/connectors/database/bigquery/cli.md index 0f3fd1bb7e9..87dc3964eff 100644 --- a/openmetadata-docs/content/connectors/database/bigquery/cli.md +++ b/openmetadata-docs/content/connectors/database/bigquery/cli.md @@ -59,23 +59,31 @@ pip3 install "openmetadata-ingestion[bigquery-usage]" -| # | GCP Permission | GCP Role | Required For | -| :---------- | :---------- | :---------- | :---------- | -| 1 | bigquery.datasets.get | BigQuery Data Viewer | Metadata Ingestion | -| 2 | bigquery.tables.get | BigQuery Data Viewer | Metadata Ingestion | -| 3 | bigquery.tables.getData | BigQuery Data Viewer | Metadata Ingestion | -| 4 | bigquery.tables.list | BigQuery Data Viewer | Metadata Ingestion | -| 5 | resourcemanager.projects.get | BigQuery Data Viewer | Metadata Ingestion | -| 6 | bigquery.jobs.create | BigQuery Job User | Metadata Ingestion | -| 7 | bigquery.jobs.listAll | BigQuery Job User | Metadata Ingestion | -| 8 | datacatalog.taxonomies.get | BigQuery Policy Admin | Fetch Policy Tags | -| 9 | datacatalog.taxonomies.list | BigQuery Policy Admin | Fetch Policy Tags | -| 10 | bigquery.readsessions.create | BigQuery Admin | Bigquery Usage Workflow | -| 11 | bigquery.readsessions.getData | BigQuery Admin | Bigquery Usage Workflow | +| # | GCP Permission | Required For | +| :--- | :---------------------------- | :---------------------- | +| 1 | bigquery.datasets.get | Metadata Ingestion | +| 2 | bigquery.tables.get | Metadata Ingestion | +| 3 | bigquery.tables.getData | Metadata Ingestion | +| 4 | bigquery.tables.list | Metadata Ingestion | +| 5 | resourcemanager.projects.get | Metadata Ingestion | +| 6 | bigquery.jobs.create | Metadata Ingestion | +| 7 | bigquery.jobs.listAll | Metadata Ingestion | +| 8 | datacatalog.taxonomies.get | Fetch Policy Tags | +| 9 | datacatalog.taxonomies.list | Fetch Policy Tags | +| 10 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow | +| 11 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow |
+ + + ## Metadata Ingestion All connectors are defined as JSON Schemas. diff --git a/openmetadata-docs/content/connectors/database/bigquery/index.md b/openmetadata-docs/content/connectors/database/bigquery/index.md index 417951d36a1..7fc70a4924a 100644 --- a/openmetadata-docs/content/connectors/database/bigquery/index.md +++ b/openmetadata-docs/content/connectors/database/bigquery/index.md @@ -74,22 +74,30 @@ custom Airflow plugins to handle the workflow deployment. -| # | GCP Permission | GCP Role | Required For | -| :--- | :---------------------------- | :-------------------- | :---------------------- | -| 1 | bigquery.datasets.get | BigQuery Data Viewer | Metadata Ingestion | -| 2 | bigquery.tables.get | BigQuery Data Viewer | Metadata Ingestion | -| 3 | bigquery.tables.getData | BigQuery Data Viewer | Metadata Ingestion | -| 4 | bigquery.tables.list | BigQuery Data Viewer | Metadata Ingestion | -| 5 | resourcemanager.projects.get | BigQuery Data Viewer | Metadata Ingestion | -| 6 | bigquery.jobs.create | BigQuery Job User | Metadata Ingestion | -| 7 | bigquery.jobs.listAll | BigQuery Job User | Metadata Ingestion | -| 8 | datacatalog.taxonomies.get | BigQuery Policy Admin | Fetch Policy Tags | -| 9 | datacatalog.taxonomies.list | BigQuery Policy Admin | Fetch Policy Tags | -| 10 | bigquery.readsessions.create | BigQuery Admin | Bigquery Usage Workflow | -| 11 | bigquery.readsessions.getData | BigQuery Admin | Bigquery Usage Workflow | +| # | GCP Permission | Required For | +| :--- | :---------------------------- | :---------------------- | +| 1 | bigquery.datasets.get | Metadata Ingestion | +| 2 | bigquery.tables.get | Metadata Ingestion | +| 3 | bigquery.tables.getData | Metadata Ingestion | +| 4 | bigquery.tables.list | Metadata Ingestion | +| 5 | resourcemanager.projects.get | Metadata Ingestion | +| 6 | bigquery.jobs.create | Metadata Ingestion | +| 7 | bigquery.jobs.listAll | Metadata Ingestion | +| 8 | datacatalog.taxonomies.get | Fetch Policy Tags | +| 9 | datacatalog.taxonomies.list | Fetch Policy Tags | +| 10 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow | +| 11 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow |
+ + + ## Metadata Ingestion ### 1. Visit the Services Page diff --git a/openmetadata-docs/content/connectors/database/bigquery/roles.md b/openmetadata-docs/content/connectors/database/bigquery/roles.md new file mode 100644 index 00000000000..f836949cfe8 --- /dev/null +++ b/openmetadata-docs/content/connectors/database/bigquery/roles.md @@ -0,0 +1,87 @@ +--- +title: BigQuery +slug: /connectors/database/bigquery/roles +--- + +# Create custom role in GCP + +This documentation will guide you on how to create a custom role in GCP with the necessary permissions to ingest BigQuery in OpenMetadata. + + +## Step 1: Navigate to Roles + +Search for `Roles` in your GCP console and select the first result under `IAM & Roles` section. + + + + + +## Step 2: Create Role & Add Permissions + +Below the search bar you should see a `Create Role` button click on that & navigate to create role page. + + + + + +Once You are on the create role page, you can edit the description & title of the role and finally you can click on add permissions to grant permissions to role. + + + +You can search for the required permissions in the filter box and add them accordingly. To ingest metadata from BigQuery you need to grant the following permissions to the user. + + + + +| # | GCP Permission | Required For | +| :--- | :---------------------------- | :---------------------- | +| 1 | bigquery.datasets.get | Metadata Ingestion | +| 2 | bigquery.tables.get | Metadata Ingestion | +| 3 | bigquery.tables.getData | Metadata Ingestion | +| 4 | bigquery.tables.list | Metadata Ingestion | +| 5 | resourcemanager.projects.get | Metadata Ingestion | +| 6 | bigquery.jobs.create | Metadata Ingestion | +| 7 | bigquery.jobs.listAll | Metadata Ingestion | +| 8 | datacatalog.taxonomies.get | Fetch Policy Tags | +| 9 | datacatalog.taxonomies.list | Fetch Policy Tags | +| 10 | bigquery.readsessions.create | Bigquery Usage & Lineage Workflow | +| 11 | bigquery.readsessions.getData | Bigquery Usage & Lineage Workflow | + +
+ + + +Once you have added all the required permissions, you can create the role by clicking on the create button. + + + +## Step 3: Assign Role to Service Account + +To assign the created role, you can navigate to `IAM` and click on `Grant Access` and you can search your service account in the `Add Principals` section & Assign the created role to the service account. + + + \ No newline at end of file diff --git a/openmetadata-docs/content/deployment/security/keycloak/index.md b/openmetadata-docs/content/deployment/security/keycloak/index.md index e72a2011098..13b976d4a75 100644 --- a/openmetadata-docs/content/deployment/security/keycloak/index.md +++ b/openmetadata-docs/content/deployment/security/keycloak/index.md @@ -55,6 +55,12 @@ Security requirements for your **production** environment: - Click on `Save` button. + + +Note: Scopes `openid`, `email` & `profile` are required to fetch the user details so you will have to add these scopes in your client. + + + diff --git a/openmetadata-docs/content/menu.md b/openmetadata-docs/content/menu.md index d6be5dd50a7..9071963153b 100644 --- a/openmetadata-docs/content/menu.md +++ b/openmetadata-docs/content/menu.md @@ -206,6 +206,8 @@ site_menu: url: /connectors/database/bigquery/airflow - category: Connectors / Database / BigQuery / CLI url: /connectors/database/bigquery/cli + - category: Connectors / Database / BigQuery / Roles + url: /connectors/database/bigquery/roles - category: Connectors / Database / Clickhouse url: /connectors/database/clickhouse - category: Connectors / Database / Clickhouse / Airflow diff --git a/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-1.png b/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-1.png new file mode 100644 index 00000000000..c23021e9f9c Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-1.png differ diff --git a/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-2.png b/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-2.png new file mode 100644 index 00000000000..aea47b9491f Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-2.png differ diff --git a/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-3.png b/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-3.png new file mode 100644 index 00000000000..fb571bedebe Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-3.png differ diff --git a/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-4.png b/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-4.png new file mode 100644 index 00000000000..f5436d06c5e Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-4.png differ diff --git a/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-5.png b/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-5.png new file mode 100644 index 00000000000..c1e7971517e Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-5.png differ diff --git a/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-6.png b/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-6.png new file mode 100644 index 00000000000..2b953fec202 Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/bigquery/create-role-6.png differ