datahub/metadata-ingestion/docs/sources/bigquery/bigquery-beta_pre.md

Prerequisites

Create a datahub profile in GCP

1. Create a custom role for datahub as per BigQuery docs
2. Grant the following permissions to this role:

Basic Requirements (needs for metadata ingestion)

permission	Description
bigquery.datasets.get	This needs to list datasets
bigquery.datasets.getIamPolicy	This needs to list datasets
bigquery.jobs.create	Needs to submit queries.
bigquery.jobs.list	Needs to check submitted queries status.
bigquery.tables.get	Needs to get metadata about Bigquery Tables.
bigquery.tables.list	Needs to list metadata about Bigquery Tables.
bigquery.readsessions.create	Needs to get resultset of queries.
bigquery.readsessions.getData	Needs to get resultset of queries.
resourcemanager.projects.get	Needs to get resultset of queries.

Lineage/usage generation requirements

Additional requirements needed on the top of the basic requirements. If you want to get lineage from multiple projects you have to grant this permission for each of them.

permission	Description
bigquery.jobs.listAll	Needs for lineage generation and usage to see all the queries were run on a project
logging.logEntries.list	Needs for lineage generation via GCP logging
logging.privateLogEntries.list	Needs for lineage generation via GCP logging

Profiling requirements

Additional requirements needed on the top of the basic requirements.

permission	Description
bigquery.tables.getData	profiler needs to access data to do the profiling
bigquery.tables.create	It needs to create temporary tables to profile partitioned/sharded tables. See below for details.
bigquery.tables.delete	It needs to create temporary tables to profile partitioned/sharded tables. See below for details.

Profiler creates temporary tables to profile partitioned/sharded tables and that is why it needs table create/delete privilege. Use profiling.bigquery_temp_table_schema to restrict to one specific dataset the create/delete permission

Create a service account

1. Setup a ServiceAccount as per BigQuery docs and assign the previously created role to this service account.
2. Download a service account JSON keyfile. Example credential file:

{
  "type": "service_account",
  "project_id": "project-id-1234567",
  "private_key_id": "d0121d0000882411234e11166c6aaa23ed5d74e0",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIyourkey\n-----END PRIVATE KEY-----",
  "client_email": "test@suppproject-id-1234567.iam.gserviceaccount.com",
  "client_id": "113545814931671546333",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/test%suppproject-id-1234567.iam.gserviceaccount.com"
}

3. To provide credentials to the source, you can either: Set an environment variable: $ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"

   or

   Set credential config in your source based on the credential json file. For example:

     credential:
       project_id: project-id-1234567
       private_key_id: "d0121d0000882411234e11166c6aaa23ed5d74e0"
       private_key: "-----BEGIN PRIVATE KEY-----\nMIIyourkey\n-----END PRIVATE KEY-----\n"
       client_email: "test@suppproject-id-1234567.iam.gserviceaccount.com"
       client_id: "123456678890"

Lineage Computation Details

When use_exported_bigquery_audit_metadata is set to true, lineage information will be computed using exported bigquery logs. On how to setup exported bigquery audit logs, refer to the following docs on BigQuery audit logs. Note that only protoPayloads with "type.googleapis.com/google.cloud.audit.BigQueryAuditMetadata" are supported by the current ingestion version. The bigquery_audit_metadata_datasets parameter will be used only if use_exported_bigquery_audit_metadat is set to true.

Note: the bigquery_audit_metadata_datasets parameter receives a list of datasets, in the format $PROJECT.$DATASET. This way queries from a multiple number of projects can be used to compute lineage information.

Note: Since bigquery source also supports dataset level lineage, the auth client will require additional permissions to be able to access the google audit logs. Refer the permissions section in bigquery-usage section below which also accesses the audit logs.

Profiling Details

Profiling can profile normal/partitioned and sharded tables as well but due to performance reasons, we only profile the latest partition for Partitioned tables and the latest shard for sharded tables.

If limit/offset parameter is set or partitioning partitioned or sharded table Great Expectation (the profiling framework we use) needs to create temporary views. By default, these views are created in the schema where the profiled table is but you can control to create all these tables into a predefined schema by setting profiling.bigquery_temp_table_schema property. Temporary tables are removed after profiling.

     profiling:
       enabled: true
       bigquery_temp_table_schema: my-project-id.my-schema-where-views-can-be-created

:::note

Due to performance reasons, we only profile the latest partition for Partitioned tables and the latest shard for sharded tables. You can set partition explicitly with partition.partition_datetime property if you want. (partition will be applied to all partitioned tables) :::

Working with multi-project GCP setups

Sometimes you may have multiple GCP project with one only giving you view access rights and other project where you have view/modify rights.

The GCP roles with which this setup has been tested are as follows

• Storage Project
  • BigQuery Data Viewer
  • BigQuery Metadata Viewer
  • Logs Viewer
  • Private Logs Viewer
• Compute Project
  • BigQuery Admin
  • BigQuery Data Editor
  • BigQuery Job User

If you are using use_exported_bigquery_audit_metadata = True then make sure you prefix the datasets in bigquery_audit_metadata_datasets with storage project id.

Caveats

• For Materialized views lineage is dependent on logs being retained. If your GCP logging is retained for 30 days (default) and 30 days have passed since the creation of the materialized view we won't be able to get lineage for them.