yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-04 15:50:14 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/metadata-ingestion/docs/sources/vertexai/vertexai_pre.md

112 lines
20 KiB
Markdown
Raw Permalink Normal View History

feat(ingestion) Adding vertexAI ingestion source (v1 - model group and model) (#12632)
2025-03-13 11:02:15 -07:00
Ingesting metadata from VertexAI requires using the **Vertex AI** module.
#### Prerequisites
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
feat(ingestion) Adding vertexAI ingestion source (v1 - model group and model) (#12632)
2025-03-13 11:02:15 -07:00
Please refer to the [Vertex AI documentation](https://cloud.google.com/vertex-ai/docs) for basic information on Vertex AI.
#### Credentials to access to GCP
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
fix(metadata-ingestion) update vertexAI source doc with permissions detail (#13219)
2025-04-15 13:37:26 -07:00
Please read the section to understand how to set up application default Credentials to [GCP docs](https://cloud.google.com/docs/authentication/provide-credentials-adc#how-to).
##### Permissions
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
- Grant the following permissions to the Service Account on every project where you would like to extract metadata from
fix(metadata-ingestion) update vertexAI source doc with permissions detail (#13219)
2025-04-15 13:37:26 -07:00
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
Default GCP Role which contains these permissions [roles/aiplatform.viewer](https://cloud.google.com/vertex-ai/docs/general/access-control#aiplatform.viewer)
| Permission | Description |
| ----------------------------------- | -------------------------------------------------------------------- |
| `aiplatform.models.list` | Allows a user to view and list all ML models in a project |
| `aiplatform.models.get` | Allows a user to view details of a specific ML model |
| `aiplatform.endpoints.list` | Allows a user to view and list all prediction endpoints in a project |
| `aiplatform.endpoints.get` | Allows a user to view details of a specific prediction endpoint |
| `aiplatform.trainingPipelines.list` | Allows a user to view and list all training pipelines in a project |
| `aiplatform.trainingPipelines.get` | Allows a user to view details of a specific training pipeline |
| `aiplatform.customJobs.list` | Allows a user to view and list all custom jobs in a project |
| `aiplatform.customJobs.get` | Allows a user to view details of a specific custom job |
| `aiplatform.experiments.list` | Allows a user to view and list all experiments in a project |
| `laiplatform.experiments.get` | Allows a user to view details of a specific experiment in a project |
| `aiplatform.metadataStores.list` | allows a user to view and list all metadata store in a project |
| `aiplatform.metadataStores.get` | allows a user to view details of a specific metadata store |
| `aiplatform.executions.list` | allows a user to view and list all executions in a project |
| `aiplatform.executions.get` | allows a user to view details of a specific execution |
| `aiplatform.datasets.list` | allows a user to view and list all datasets in a project |
| `aiplatform.datasets.get` | allows a user to view details of a specific dataset |
| `aiplatform.pipelineJobs.get` | allows a user to view and list all pipeline jobs in a project |
| `aiplatform.pipelineJobs.list` | allows a user to view details of a specific pipeline job |
feat(ingestion) Adding vertexAI ingestion source (v1 - model group and model) (#12632)
2025-03-13 11:02:15 -07:00
#### Create a service account and assign roles
1. Setup a ServiceAccount as per [GCP docs](https://cloud.google.com/iam/docs/creating-managing-service-accounts#iam-service-accounts-create-console) and assign the previously created role to this service account.
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
2. Download a service account JSON keyfile.
feat(ingestion) Adding vertexAI ingestion source (v1 - model group and model) (#12632)
2025-03-13 11:02:15 -07:00
- Example credential file:
```json
{
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
"type": "service_account",
"project_id": "project-id-1234567",
"private_key_id": "d0121d0000882411234e11166c6aaa23ed5d74e0",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIyourkey\n-----END PRIVATE KEY-----",
"client_email": "test@suppproject-id-1234567.iam.gserviceaccount.com",
"client_id": "113545814931671546333",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/test%suppproject-id-1234567.iam.gserviceaccount.com"
feat(ingestion) Adding vertexAI ingestion source (v1 - model group and model) (#12632)
2025-03-13 11:02:15 -07:00
}
```
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
feat(ingestion) Adding vertexAI ingestion source (v1 - model group and model) (#12632)
2025-03-13 11:02:15 -07:00
3. To provide credentials to the source, you can either:
- Set an environment variable:
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
```sh
$ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"
```
feat(ingestion) Adding vertexAI ingestion source (v1 - model group and model) (#12632)
2025-03-13 11:02:15 -07:00
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
_or_
feat(ingestion) Adding vertexAI ingestion source (v1 - model group and model) (#12632)
2025-03-13 11:02:15 -07:00
- Set credential config in your source based on the credential json file. For example:
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
```yml
credential:
private_key_id: "d0121d0000882411234e11166c6aaa23ed5d74e0"
private_key: "-----BEGIN PRIVATE KEY-----\nMIIyourkey\n-----END PRIVATE KEY-----\n"
client_email: "test@suppproject-id-1234567.iam.gserviceaccount.com"
client_id: "123456678890"
```
fix(ingestion) Adding Vertex AI Connector documentation (#12967)
2025-03-26 16:37:34 -07:00
### Integration Details
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
Ingestion Job extract Models, Datasets, Training Jobs, Endpoints, Experiment and Experiment Runs in a given project and region on Vertex AI.
fix(ingestion) Adding Vertex AI Connector documentation (#12967)
2025-03-26 16:37:34 -07:00
#### Concept Mapping
This ingestion source maps the following Vertex AI Concepts to DataHub Concepts:
doc: Acryl to DataHub, datahubproject.io to datahub.com (#13252) Co-authored-by: Jay <159848059+jayacryl@users.noreply.github.com>
2025-04-28 23:34:33 +09:00
| Source Concept | DataHub Concept | Notes |
| :------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [`Model`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.Model) | [`MlModelGroup`](https://docs.datahub.com/docs/generated/metamodel/entities/mlmodelgroup/) | The name of a Model Group is the same as Model's name. Model serve as containers for multiple versions of the same model in Vertex AI. |
| [`Model Version`](https://cloud.google.com/vertex-ai/docs/model-registry/versioning) | [`MlModel`](https://docs.datahub.com/docs/generated/metamodel/entities/mlmodel/) | The name of a Model is `{model_name}_{model_version}` (e.g. my_vertexai_model_1 for model registered to Model Registry or Deployed to Endpoint. Each Model Version represents a specific iteration of a model with its own metadata. |
| Dataset <br/><br/> | [`Dataset`](https://docs.datahub.com/docs/generated/metamodel/entities/dataset) | A Managed Dataset resource in Vertex AI is mapped to Dataset in DataHub. <br></br> Supported types of datasets include ([`Text`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.TextDataset), [`Tabular`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.TabularDataset), [`Image Dataset`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.ImageDataset), [`Video`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.VideoDataset), [`TimeSeries`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.TimeSeriesDataset)) |
| [`Training Job`](https://cloud.google.com/vertex-ai/docs/beginner/beginners-guide) | [`DataProcessInstance`](https://docs.datahub.com/docs/generated/metamodel/entities/dataprocessinstance/) | A Training Job is mapped as DataProcessInstance in DataHub. <br></br> Supported types of training jobs include ([`AutoMLTextTrainingJob`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.AutoMLTextTrainingJob), [`AutoMLTabularTrainingJob`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.AutoMLTabularTrainingJob), [`AutoMLImageTrainingJob`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.AutoMLImageTrainingJob), [`AutoMLVideoTrainingJob`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.AutoMLVideoTrainingJob), [`AutoMLForecastingTrainingJob`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.AutoMLForecastingTrainingJob), [`Custom Job`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.CustomJob), [`Custom TrainingJob`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.CustomTrainingJob), [`Custom Container TrainingJob`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.CustomContainerTrainingJob), [`Custom Python Packaging Job`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.CustomPythonPackageTrainingJob) ) |
| [`Experiment`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.Experiment) | [`Container`](https://docs.datahub.com/docs/generated/metamodel/entities/container/) | Experiments organize related runs and serve as logical groupings for model development iterations. Each Experiment is mapped to a Container in DataHub. |
| [`Experiment Run`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.ExperimentRun) | [`DataProcessInstance`](https://docs.datahub.com/docs/generated/metamodel/entities/dataprocessinstance/) | An Experiment Run represents a single execution of a ML workflow. An Experiment Run tracks ML parameters, metricis, artifacts and metadata |
| [`Execution`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.Execution) | [`DataProcessInstance`](https://docs.datahub.com/docs/generated/metamodel/entities/dataprocessinstance/) | Metadata Execution resource for Vertex AI. Metadata Execution is started in a experiment run and captures input and output artifacts. |
fix(ingestion) Adding Vertex AI Connector documentation (#12967)
2025-03-26 16:37:34 -07:00
Vertex AI Concept Diagram:
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
fix(ingestion) Adding Vertex AI Connector documentation (#12967)
2025-03-26 16:37:34 -07:00
<p align="center">
<img width="70%" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/metadata-ingestion/vertexai/concept-mapping.png"/>
</p>
#### Lineage
Lineage is emitted using Vertex AI API to capture the following relationships:
- A training job and a model (which training job produce a model)
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
- A dataset and a training job (which dataset was consumed by a training job to train a model)
fix(ingestion) Adding Vertex AI Connector documentation (#12967)
2025-03-26 16:37:34 -07:00
- Experiment runs and an experiment
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
- Metadata execution and an experiment run
Reference in New Issue Copy Permalink