yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2026-01-05 03:56:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Docs: GCP Partial Creation and Addition (#20349)

Browse Source
This commit is contained in:
Rounak Dhillon 2025-03-19 19:43:34 +05:30 committed by GitHub
parent 5da8a967cc
commit 8bf62e29d0
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
14 changed files with 146 additions and 233 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
openmetadata-docs/content/partials/v1.6/connectors/yaml/common/gcp-config-def.md Normal file
View File

@ -0,0 +1,16 @@
**gcpConfig:**
{% codeInfo srNumber=153 %}
- **type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file.
- **projectId**: A project ID is a unique string used to differentiate your project from all others in Google Cloud. To fetch this key, look for the value associated with the `project_id` key in the service account key file. You can also pass multiple project id to ingest metadata from different BigQuery projects into one service.
- **privateKeyId**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file.
- **privateKey**: This is the private key associated with the service account that is used to authenticate and authorize access to BigQuery. To fetch this key, look for the value associated with the `private_key` key in the service account file.
- **clientEmail**: This is the email address associated with the service account. To fetch this key, look for the value associated with the `client_email` key in the service account key file.
- **clientId**: This is a unique identifier for the service account. To fetch this key, look for the value associated with the `client_id` key in the service account key file.
- **authUri**: This is the URI for the authorization server. To fetch this key, look for the value associated with the `auth_uri` key in the service account key file. The default value to Auth URI is https://accounts.google.com/o/oauth2/auth.
- **tokenUri**: The Google Cloud Token URI is a specific endpoint used to obtain an OAuth 2.0 access token from the Google Cloud IAM service. This token allows you to authenticate and access various Google Cloud resources and APIs that require authorization. To fetch this key, look for the value associated with the `token_uri` key in the service account credentials file. Default Value to Token URI is https://oauth2.googleapis.com/token.
- **authProviderX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs
- **clientX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file.
{% /codeInfo %}

15
openmetadata-docs/content/partials/v1.6/connectors/yaml/common/gcp-config.md Normal file
View File

@ -0,0 +1,15 @@
```yaml {% srNumber=153 %}
type: service_account
projectId: project-id # ["project-id-1", "project-id-2"]
privateKeyId: abc123
privateKey: |
-----BEGIN PRIVATE KEY-----
Super secret key
-----END PRIVATE KEY-----
clientEmail: role@project.iam.gserviceaccount.com
clientId: "1234"
# authUri: https://accounts.google.com/o/oauth2/auth (default)
# tokenUri: https://oauth2.googleapis.com/token (default)
# authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs (default)
clientX509CertUrl: https://www.googleapis.com/robot/v1/metadata/x509/role%40project.iam.gserviceaccount.com
```

16
openmetadata-docs/content/partials/v1.7/connectors/yaml/common/gcp-config-def.md Normal file
View File

@ -0,0 +1,16 @@
**gcpConfig:**
{% codeInfo srNumber=153 %}
- **type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file.
- **projectId**: A project ID is a unique string used to differentiate your project from all others in Google Cloud. To fetch this key, look for the value associated with the `project_id` key in the service account key file. You can also pass multiple project id to ingest metadata from different BigQuery projects into one service.
- **privateKeyId**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file.
- **privateKey**: This is the private key associated with the service account that is used to authenticate and authorize access to BigQuery. To fetch this key, look for the value associated with the `private_key` key in the service account file.
- **clientEmail**: This is the email address associated with the service account. To fetch this key, look for the value associated with the `client_email` key in the service account key file.
- **clientId**: This is a unique identifier for the service account. To fetch this key, look for the value associated with the `client_id` key in the service account key file.
- **authUri**: This is the URI for the authorization server. To fetch this key, look for the value associated with the `auth_uri` key in the service account key file. The default value to Auth URI is https://accounts.google.com/o/oauth2/auth.
- **tokenUri**: The Google Cloud Token URI is a specific endpoint used to obtain an OAuth 2.0 access token from the Google Cloud IAM service. This token allows you to authenticate and access various Google Cloud resources and APIs that require authorization. To fetch this key, look for the value associated with the `token_uri` key in the service account credentials file. Default Value to Token URI is https://oauth2.googleapis.com/token.
- **authProviderX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs
- **clientX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file.
{% /codeInfo %}

15
openmetadata-docs/content/partials/v1.7/connectors/yaml/common/gcp-config.md Normal file
View File

@ -0,0 +1,15 @@
```yaml {% srNumber=153 %}
type: service_account
projectId: project-id # ["project-id-1", "project-id-2"]
privateKeyId: abc123
privateKey: |
-----BEGIN PRIVATE KEY-----
Super secret key
-----END PRIVATE KEY-----
clientEmail: role@project.iam.gserviceaccount.com
clientId: "1234"
# authUri: https://accounts.google.com/o/oauth2/auth (default)
# tokenUri: https://oauth2.googleapis.com/token (default)
# authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs (default)
clientX509CertUrl: https://www.googleapis.com/robot/v1/metadata/x509/role%40project.iam.gserviceaccount.com
```

32
openmetadata-docs/content/v1.6.x/connectors/database/bigquery/yaml.md
View File

@ -102,16 +102,11 @@ You can checkout [this](https://cloud.google.com/iam/docs/keys-create-delete#iam
**1.** Passing the raw credential values provided by BigQuery. This requires us to provide the following information, all provided by BigQuery:
- **type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file.
- **projectId**: A project ID is a unique string used to differentiate your project from all others in Google Cloud. To fetch this key, look for the value associated with the `project_id` key in the service account key file. You can also pass multiple project id to ingest metadata from different BigQuery projects into one service.
- **privateKeyId**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file.
- **privateKey**: This is the private key associated with the service account that is used to authenticate and authorize access to BigQuery. To fetch this key, look for the value associated with the `private_key` key in the service account file.
- **clientEmail**: This is the email address associated with the service account. To fetch this key, look for the value associated with the `client_email` key in the service account key file.
- **clientId**: This is a unique identifier for the service account. To fetch this key, look for the value associated with the `client_id` key in the service account key file.
- **authUri**: This is the URI for the authorization server. To fetch this key, look for the value associated with the `auth_uri` key in the service account key file. The default value to Auth URI is https://accounts.google.com/o/oauth2/auth.
- **tokenUri**: The Google Cloud Token URI is a specific endpoint used to obtain an OAuth 2.0 access token from the Google Cloud IAM service. This token allows you to authenticate and access various Google Cloud resources and APIs that require authorization. To fetch this key, look for the value associated with the `token_uri` key in the service account credentials file. Default Value to Token URI is https://oauth2.googleapis.com/token.
- **authProviderX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs
- **clientX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file.
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/gcp-config-def.md" /%}
{% codeInfo srNumber=4 %}
**2.** Passing a local file path that contains the credentials:
- **gcpCredentialsPath**
@ -191,19 +186,10 @@ source:
```yaml {% srNumber=1 %}
credentials:
gcpConfig:
type: service_account
projectId: project-id # ["project-id-1", "project-id-2"]
privateKeyId: abc123
privateKey: |
-----BEGIN PRIVATE KEY-----
Super secret key
-----END PRIVATE KEY-----
clientEmail: role@project.iam.gserviceaccount.com
clientId: "1234"
# authUri: https://accounts.google.com/o/oauth2/auth (default)
# tokenUri: https://oauth2.googleapis.com/token (default)
# authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs (default)
clientX509CertUrl: https://www.googleapis.com/robot/v1/metadata/x509/role%40project.iam.gserviceaccount.com
```
{% partial file="/v1.6/connectors/yaml/common/gcp-config.md" /%}
```yaml {% srNumber=4 %}
# taxonomyLocation: us
# taxonomyProjectID: ["project-id-1", "project-id-2"]
# usageLocation: us

29
openmetadata-docs/content/v1.6.x/connectors/database/bigtable/yaml.md
View File

@ -80,22 +80,13 @@ You can checkout [this](https://cloud.google.com/iam/docs/keys-create-delete#iam
**1.** Passing the raw credential values provided by BigTable. This requires us to provide the following information, all provided by BigTable:
- **type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file.
- **projectId**: A project ID is a unique string used to differentiate your project from all others in Google Cloud. To fetch this key, look for the value associated with the `project_id` key in the service account key file. You can also pass multiple project id to ingest metadata from different BigTable projects into one service.
- **privateKeyId**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file.
- **privateKey**: This is the private key associated with the service account that is used to authenticate and authorize access to BigTable. To fetch this key, look for the value associated with the `private_key` key in the service account file.
- **clientEmail**: This is the email address associated with the service account. To fetch this key, look for the value associated with the `client_email` key in the service account key file.
- **clientId**: This is a unique identifier for the service account. To fetch this key, look for the value associated with the `client_id` key in the service account key file.
- **authUri**: This is the URI for the authorization server. To fetch this key, look for the value associated with the `auth_uri` key in the service account key file. The default value to Auth URI is https://accounts.google.com/o/oauth2/auth.
- **tokenUri**: The Google Cloud Token URI is a specific endpoint used to obtain an OAuth 2.0 access token from the Google Cloud IAM service. This token allows you to authenticate and access various Google Cloud resources and APIs that require authorization. To fetch this key, look for the value associated with the `token_uri` key in the service account credentials file. Default Value to Token URI is https://oauth2.googleapis.com/token.
- **authProviderX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs
- **clientX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file.
**2.** Passing a local file path that contains the credentials:
- **gcpCredentialsPath**
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/gcp-config-def.md" /%}
#### Advanced Configuration
{% codeInfo srNumber=2 %}
@ -127,20 +118,10 @@ source:
```yaml {% srNumber=1 %}
credentials:
gcpConfig:
type: My Type
projectId: project ID # ["project-id-1", "project-id-2"]
privateKeyId: us-east-2
privateKey: |
-----BEGIN PRIVATE KEY-----
Super secret key
-----END PRIVATE KEY-----
clientEmail: client@mail.com
clientId: 1234
# authUri: https://accounts.google.com/o/oauth2/auth (default)
# tokenUri: https://oauth2.googleapis.com/token (default)
# authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs (default)
clientX509CertUrl: https://cert.url
```
{% partial file="/v1.6/connectors/yaml/common/gcp-config.md" /%}
```yaml {% srNumber=2 %}
# connectionOptions:
# key: value

28
openmetadata-docs/content/v1.6.x/connectors/database/gcs-datalake/yaml.md
View File

@ -62,18 +62,10 @@ The workflow is modeled around the following JSON Schema.
#### Source Configuration - Service Connection
{% partial file="/v1.6/connectors/yaml/common/gcp-config-def.md" /%}
{% codeInfo srNumber=5 %}
* **type**: Credentials type, e.g. `service_account`.
* **projectId**
* **privateKey**
* **privateKeyId**
* **clientEmail**
* **clientId**
* **authUri**: [https://accounts.google.com/o/oauth2/auth](https://accounts.google.com/o/oauth2/auth) by default
* **tokenUri**: [https://oauth2.googleapis.com/token](https://oauth2.googleapis.com/token) by default
* **authProviderX509CertUrl**: [https://www.googleapis.com/oauth2/v1/certs](https://www.googleapis.com/oauth2/v1/certs) by default
* **clientX509CertUrl**
* **bucketName**: name of the bucket in GCS
* **Prefix**: prefix in gcp bucket
@ -99,19 +91,11 @@ source:
type: Datalake
configSource:
securityConfig:
```
```yaml {% srNumber=5 %}
gcpConfig:
type: type of account
projectId: project id
privateKeyId: private key id
privateKey: private key
clientEmail: client email
clientId: client id
authUri: https://accounts.google.com/o/oauth2/auth
tokenUri: https://oauth2.googleapis.com/token
authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs
clientX509CertUrl: clientX509 Certificate Url
```
{% partial file="/v1.6/connectors/yaml/common/gcp-config.md" /%}
```yaml {% srNumber=5 %}
bucketName: bucket name
prefix: prefix
```

34
openmetadata-docs/content/v1.6.x/connectors/ml-model/vertexai/yaml.md
View File

@ -71,16 +71,11 @@ You can checkout [this](https://cloud.google.com/iam/docs/keys-create-delete#iam
**1.** Passing the raw credential values provided by VertexAI. This requires us to provide the following information, all provided by VertexAI:
- **type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file.
- **projectId**: A project ID is a unique string used to differentiate your project from all others in Google Cloud. To fetch this key, look for the value associated with the `project_id` key in the service account key file. You can also pass multiple project id to ingest metadata from different VertexAI projects into one service.
- **privateKeyId**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file.
- **privateKey**: This is the private key associated with the service account that is used to authenticate and authorize access to VertexAI. To fetch this key, look for the value associated with the `private_key` key in the service account file.
- **clientEmail**: This is the email address associated with the service account. To fetch this key, look for the value associated with the `client_email` key in the service account key file.
- **clientId**: This is a unique identifier for the service account. To fetch this key, look for the value associated with the `client_id` key in the service account key file.
- **authUri**: This is the URI for the authorization server. To fetch this key, look for the value associated with the `auth_uri` key in the service account key file. The default value to Auth URI is https://accounts.google.com/o/oauth2/auth.
- **tokenUri**: The Google Cloud Token URI is a specific endpoint used to obtain an OAuth 2.0 access token from the Google Cloud IAM service. This token allows you to authenticate and access various Google Cloud resources and APIs that require authorization. To fetch this key, look for the value associated with the `token_uri` key in the service account credentials file. Default Value to Token URI is https://oauth2.googleapis.com/token.
- **authProviderX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs
- **clientX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file.
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/gcp-config.md" /%}
{% codeInfo srNumber=4 %}
**2.** Passing a local file path that contains the credentials:
- **gcpCredentialsPath**
@ -105,21 +100,12 @@ source:
```yaml {% srNumber=1 %}
credentials:
gcpConfig:
type: My Type
projectId: project ID # ["project-id-1", "project-id-2"]
privateKeyId: us-east-2
privateKey: |
-----BEGIN PRIVATE KEY-----
Super secret key
-----END PRIVATE KEY-----
clientEmail: client@mail.com
clientId: 1234
# authUri: https://accounts.google.com/o/oauth2/auth (default)
# tokenUri: https://oauth2.googleapis.com/token (default)
# authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs (default)
clientX509CertUrl: https://cert.url
location: PROJECT LOCATION/REGION (us-central1)
```
{% partial file="/v1.6/connectors/yaml/common/gcp-config.md" /%}
```yaml {% srNumber=4 %}
location: PROJECT LOCATION/REGION (us-central1)
```
```yaml {% srNumber=2 %}
# connectionOptions:

34
openmetadata-docs/content/v1.6.x/connectors/storage/gcs/yaml.md
View File

@ -77,20 +77,17 @@ This is a sample config for Athena:
#### Source Configuration - Service Connection
{% codeInfo srNumber=2 %}
**gcpConfig:**
**1.** Passing the raw credential values provided by GCP. This requires us to provide the following information, all provided by GCP:
- **type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file.
- **projectId**: A project ID is a unique string used to differentiate your project from all others in Google Cloud. To fetch this key, look for the value associated with the `project_id` key in the service account key file. You can also pass multiple project id to ingest metadata from different GCP projects into one service.
- **privateKeyId**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file.
- **privateKey**: This is the private key associated with the service account that is used to authenticate and authorize access to GCP. To fetch this key, look for the value associated with the `private_key` key in the service account file.
- **clientEmail**: This is the email address associated with the service account. To fetch this key, look for the value associated with the `client_email` key in the service account key file.
- **clientId**: This is a unique identifier for the service account. To fetch this key, look for the value associated with the `client_id` key in the service account key file.
- **authUri**: This is the URI for the authorization server. To fetch this key, look for the value associated with the `auth_uri` key in the service account key file. The default value to Auth URI is https://accounts.google.com/o/oauth2/auth.
- **tokenUri**: The Google Cloud Token URI is a specific endpoint used to obtain an OAuth 2.0 access token from the Google Cloud IAM service. This token allows you to authenticate and access various Google Cloud resources and APIs that require authorization. To fetch this key, look for the value associated with the `token_uri` key in the service account credentials file. Default Value to Token URI is https://oauth2.googleapis.com/token.
- **authProviderX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs
- **clientX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file.
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/gcp-config-def.md" /%}
{% codeInfo srNumber=4 %}
**2.** Passing a local file path that contains the credentials:
- **gcpCredentialsPath**
@ -161,19 +158,10 @@ source:
```yaml {% srNumber=1 %}
credentials:
gcpConfig:
type: My Type
projectId: project ID # ["project-id-1", "project-id-2"]
privateKeyId: us-east-2
privateKey: |
-----BEGIN PRIVATE KEY-----
Super secret key
-----END PRIVATE KEY-----
clientEmail: client@mail.com
clientId: "1234"
# authUri: https://accounts.google.com/o/oauth2/auth (default)
# tokenUri: https://oauth2.googleapis.com/token (default)
# authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs (default)
clientX509CertUrl: https://cert.url
```
{% partial file="/v1.6/connectors/yaml/common/gcp-config.md" /%}
```yaml {% srNumber=4 %}
# taxonomyLocation: us
# taxonomyProjectID: ["project-id-1", "project-id-2"]
# usageLocation: us

34
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/bigquery/yaml.md
View File

@ -102,21 +102,15 @@ You can checkout [this](https://cloud.google.com/iam/docs/keys-create-delete#iam
**1.** Passing the raw credential values provided by BigQuery. This requires us to provide the following information, all provided by BigQuery:
- **type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file.
- **projectId**: A project ID is a unique string used to differentiate your project from all others in Google Cloud. To fetch this key, look for the value associated with the `project_id` key in the service account key file. You can also pass multiple project id to ingest metadata from different BigQuery projects into one service.
- **privateKeyId**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file.
- **privateKey**: This is the private key associated with the service account that is used to authenticate and authorize access to BigQuery. To fetch this key, look for the value associated with the `private_key` key in the service account file.
- **clientEmail**: This is the email address associated with the service account. To fetch this key, look for the value associated with the `client_email` key in the service account key file.
- **clientId**: This is a unique identifier for the service account. To fetch this key, look for the value associated with the `client_id` key in the service account key file.
- **authUri**: This is the URI for the authorization server. To fetch this key, look for the value associated with the `auth_uri` key in the service account key file. The default value to Auth URI is https://accounts.google.com/o/oauth2/auth.
- **tokenUri**: The Google Cloud Token URI is a specific endpoint used to obtain an OAuth 2.0 access token from the Google Cloud IAM service. This token allows you to authenticate and access various Google Cloud resources and APIs that require authorization. To fetch this key, look for the value associated with the `token_uri` key in the service account credentials file. Default Value to Token URI is https://oauth2.googleapis.com/token.
- **authProviderX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs
- **clientX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file.
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/gcp-config-def.md" /%}
{% codeInfo srNumber=4 %}
**2.** Passing a local file path that contains the credentials:
- **gcpCredentialsPath**
**Taxonomy Project ID (Optional)**: Bigquery uses taxonomies to create hierarchical groups of policy tags. To apply access controls to BigQuery columns, tag the columns with policy tags. Learn more about how yo can create policy tags and set up column-level access control [here](https://cloud.google.com/bigquery/docs/column-level-security)
If you have attached policy tags to the columns of table available in Bigquery, then OpenMetadata will fetch those tags and attach it to the respective columns.
@ -191,19 +185,11 @@ source:
```yaml {% srNumber=1 %}
credentials:
gcpConfig:
type: service_account
projectId: project-id # ["project-id-1", "project-id-2"]
privateKeyId: abc123
privateKey: |
-----BEGIN PRIVATE KEY-----
Super secret key
-----END PRIVATE KEY-----
clientEmail: role@project.iam.gserviceaccount.com
clientId: "1234"
# authUri: https://accounts.google.com/o/oauth2/auth (default)
# tokenUri: https://oauth2.googleapis.com/token (default)
# authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs (default)
clientX509CertUrl: https://www.googleapis.com/robot/v1/metadata/x509/role%40project.iam.gserviceaccount.com
```
{% partial file="/v1.7/connectors/yaml/common/gcp-config.md" /%}
```yaml {% srNumber=4 %}
# taxonomyLocation: us
# taxonomyProjectID: ["project-id-1", "project-id-2"]
# usageLocation: us

29
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/bigtable/yaml.md
View File

@ -80,22 +80,13 @@ You can checkout [this](https://cloud.google.com/iam/docs/keys-create-delete#iam
**1.** Passing the raw credential values provided by BigTable. This requires us to provide the following information, all provided by BigTable:
- **type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file.
- **projectId**: A project ID is a unique string used to differentiate your project from all others in Google Cloud. To fetch this key, look for the value associated with the `project_id` key in the service account key file. You can also pass multiple project id to ingest metadata from different BigTable projects into one service.
- **privateKeyId**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file.
- **privateKey**: This is the private key associated with the service account that is used to authenticate and authorize access to BigTable. To fetch this key, look for the value associated with the `private_key` key in the service account file.
- **clientEmail**: This is the email address associated with the service account. To fetch this key, look for the value associated with the `client_email` key in the service account key file.
- **clientId**: This is a unique identifier for the service account. To fetch this key, look for the value associated with the `client_id` key in the service account key file.
- **authUri**: This is the URI for the authorization server. To fetch this key, look for the value associated with the `auth_uri` key in the service account key file. The default value to Auth URI is https://accounts.google.com/o/oauth2/auth.
- **tokenUri**: The Google Cloud Token URI is a specific endpoint used to obtain an OAuth 2.0 access token from the Google Cloud IAM service. This token allows you to authenticate and access various Google Cloud resources and APIs that require authorization. To fetch this key, look for the value associated with the `token_uri` key in the service account credentials file. Default Value to Token URI is https://oauth2.googleapis.com/token.
- **authProviderX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs
- **clientX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file.
**2.** Passing a local file path that contains the credentials:
- **gcpCredentialsPath**
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/gcp-config-def.md" /%}
#### Advanced Configuration
{% codeInfo srNumber=2 %}
@ -127,20 +118,10 @@ source:
```yaml {% srNumber=1 %}
credentials:
gcpConfig:
type: My Type
projectId: project ID # ["project-id-1", "project-id-2"]
privateKeyId: us-east-2
privateKey: |
-----BEGIN PRIVATE KEY-----
Super secret key
-----END PRIVATE KEY-----
clientEmail: client@mail.com
clientId: 1234
# authUri: https://accounts.google.com/o/oauth2/auth (default)
# tokenUri: https://oauth2.googleapis.com/token (default)
# authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs (default)
clientX509CertUrl: https://cert.url
```
{% partial file="/v1.7/connectors/yaml/common/gcp-config.md" /%}
```yaml {% srNumber=2 %}
# connectionOptions:
# key: value

28
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/gcs-datalake/yaml.md
View File

@ -62,18 +62,10 @@ The workflow is modeled around the following JSON Schema.
#### Source Configuration - Service Connection
{% partial file="/v1.7/connectors/yaml/common/gcp-config-def.md" /%}
{% codeInfo srNumber=5 %}
* **type**: Credentials type, e.g. `service_account`.
* **projectId**
* **privateKey**
* **privateKeyId**
* **clientEmail**
* **clientId**
* **authUri**: [https://accounts.google.com/o/oauth2/auth](https://accounts.google.com/o/oauth2/auth) by default
* **tokenUri**: [https://oauth2.googleapis.com/token](https://oauth2.googleapis.com/token) by default
* **authProviderX509CertUrl**: [https://www.googleapis.com/oauth2/v1/certs](https://www.googleapis.com/oauth2/v1/certs) by default
* **clientX509CertUrl**
* **bucketName**: name of the bucket in GCS
* **Prefix**: prefix in gcp bucket
@ -99,19 +91,11 @@ source:
type: Datalake
configSource:
securityConfig:
```
```yaml {% srNumber=5 %}
gcpConfig:
type: type of account
projectId: project id
privateKeyId: private key id
privateKey: private key
clientEmail: client email
clientId: client id
authUri: https://accounts.google.com/o/oauth2/auth
tokenUri: https://oauth2.googleapis.com/token
authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs
clientX509CertUrl: clientX509 Certificate Url
```
{% partial file="/v1.7/connectors/yaml/common/gcp-config.md" /%}
```yaml {% srNumber=5 %}
bucketName: bucket name
prefix: prefix
```

34
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/ml-model/vertexai/yaml.md
View File

@ -71,16 +71,11 @@ You can checkout [this](https://cloud.google.com/iam/docs/keys-create-delete#iam
**1.** Passing the raw credential values provided by VertexAI. This requires us to provide the following information, all provided by VertexAI:
- **type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file.
- **projectId**: A project ID is a unique string used to differentiate your project from all others in Google Cloud. To fetch this key, look for the value associated with the `project_id` key in the service account key file. You can also pass multiple project id to ingest metadata from different VertexAI projects into one service.
- **privateKeyId**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file.
- **privateKey**: This is the private key associated with the service account that is used to authenticate and authorize access to VertexAI. To fetch this key, look for the value associated with the `private_key` key in the service account file.
- **clientEmail**: This is the email address associated with the service account. To fetch this key, look for the value associated with the `client_email` key in the service account key file.
- **clientId**: This is a unique identifier for the service account. To fetch this key, look for the value associated with the `client_id` key in the service account key file.
- **authUri**: This is the URI for the authorization server. To fetch this key, look for the value associated with the `auth_uri` key in the service account key file. The default value to Auth URI is https://accounts.google.com/o/oauth2/auth.
- **tokenUri**: The Google Cloud Token URI is a specific endpoint used to obtain an OAuth 2.0 access token from the Google Cloud IAM service. This token allows you to authenticate and access various Google Cloud resources and APIs that require authorization. To fetch this key, look for the value associated with the `token_uri` key in the service account credentials file. Default Value to Token URI is https://oauth2.googleapis.com/token.
- **authProviderX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs
- **clientX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file.
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/gcp-config.md" /%}
{% codeInfo srNumber=4 %}
**2.** Passing a local file path that contains the credentials:
- **gcpCredentialsPath**
@ -105,21 +100,12 @@ source:
```yaml {% srNumber=1 %}
credentials:
gcpConfig:
type: My Type
projectId: project ID # ["project-id-1", "project-id-2"]
privateKeyId: us-east-2
privateKey: |
-----BEGIN PRIVATE KEY-----
Super secret key
-----END PRIVATE KEY-----
clientEmail: client@mail.com
clientId: 1234
# authUri: https://accounts.google.com/o/oauth2/auth (default)
# tokenUri: https://oauth2.googleapis.com/token (default)
# authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs (default)
clientX509CertUrl: https://cert.url
location: PROJECT LOCATION/REGION (us-central1)
```
{% partial file="/v1.7/connectors/yaml/common/gcp-config.md" /%}
```yaml {% srNumber=4 %}
location: PROJECT LOCATION/REGION (us-central1)
```
```yaml {% srNumber=2 %}
# connectionOptions:

35
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/storage/gcs/yaml.md
View File

@ -77,20 +77,17 @@ This is a sample config for Athena:
#### Source Configuration - Service Connection
{% codeInfo srNumber=2 %}
**gcpConfig:**
**1.** Passing the raw credential values provided by GCP. This requires us to provide the following information, all provided by GCP:
- **type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file.
- **projectId**: A project ID is a unique string used to differentiate your project from all others in Google Cloud. To fetch this key, look for the value associated with the `project_id` key in the service account key file. You can also pass multiple project id to ingest metadata from different GCP projects into one service.
- **privateKeyId**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file.
- **privateKey**: This is the private key associated with the service account that is used to authenticate and authorize access to GCP. To fetch this key, look for the value associated with the `private_key` key in the service account file.
- **clientEmail**: This is the email address associated with the service account. To fetch this key, look for the value associated with the `client_email` key in the service account key file.
- **clientId**: This is a unique identifier for the service account. To fetch this key, look for the value associated with the `client_id` key in the service account key file.
- **authUri**: This is the URI for the authorization server. To fetch this key, look for the value associated with the `auth_uri` key in the service account key file. The default value to Auth URI is https://accounts.google.com/o/oauth2/auth.
- **tokenUri**: The Google Cloud Token URI is a specific endpoint used to obtain an OAuth 2.0 access token from the Google Cloud IAM service. This token allows you to authenticate and access various Google Cloud resources and APIs that require authorization. To fetch this key, look for the value associated with the `token_uri` key in the service account credentials file. Default Value to Token URI is https://oauth2.googleapis.com/token.
- **authProviderX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs
- **clientX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file.
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/gcp-config-def.md" /%}
{% codeInfo srNumber=4 %}
**2.** Passing a local file path that contains the credentials:
- **gcpCredentialsPath**
@ -161,19 +158,11 @@ source:
```yaml {% srNumber=1 %}
credentials:
gcpConfig:
type: My Type
projectId: project ID # ["project-id-1", "project-id-2"]
privateKeyId: us-east-2
privateKey: |
-----BEGIN PRIVATE KEY-----
Super secret key
-----END PRIVATE KEY-----
clientEmail: client@mail.com
clientId: "1234"
# authUri: https://accounts.google.com/o/oauth2/auth (default)
# tokenUri: https://oauth2.googleapis.com/token (default)
# authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs (default)
clientX509CertUrl: https://cert.url
```
{% partial file="/v1.7/connectors/yaml/common/gcp-config.md" /%}
```yaml {% srNumber=4 %}
# taxonomyLocation: us
# taxonomyProjectID: ["project-id-1", "project-id-2"]
# usageLocation: us