| 
									
										
										
										
											2022-08-26 12:30:34 +05:30
										 |  |  | --- | 
					
						
							|  |  |  | title: Mlflow | 
					
						
							| 
									
										
										
										
											2022-10-05 21:54:02 -07:00
										 |  |  | slug: /connectors/ml-model/mlflow | 
					
						
							| 
									
										
										
										
											2022-08-26 12:30:34 +05:30
										 |  |  | --- | 
					
						
							|  |  |  | 
 | 
					
						
							| 
									
										
										
										
											2022-08-27 02:57:09 +02:00
										 |  |  | # Mlflow
 | 
					
						
							| 
									
										
										
										
											2022-08-26 12:30:34 +05:30
										 |  |  | 
 | 
					
						
							| 
									
										
										
										
											2022-08-27 02:57:09 +02:00
										 |  |  | In this section, we provide guides and references to use the Mlflow connector. | 
					
						
							| 
									
										
										
										
											2022-08-26 12:30:34 +05:30
										 |  |  | 
 | 
					
						
							| 
									
										
										
										
											2022-08-27 02:57:09 +02:00
										 |  |  | Configure and schedule Mlflow metadata and profiler workflows from the OpenMetadata UI: | 
					
						
							|  |  |  | - [Requirements](#requirements) | 
					
						
							|  |  |  | - [Metadata Ingestion](#metadata-ingestion) | 
					
						
							| 
									
										
										
										
											2022-08-26 12:30:34 +05:30
										 |  |  | 
 | 
					
						
							| 
									
										
										
										
											2022-08-27 02:57:09 +02:00
										 |  |  | If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check | 
					
						
							|  |  |  | the following docs to connect using Airflow SDK or with the CLI. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <TileContainer> | 
					
						
							|  |  |  |   <Tile | 
					
						
							|  |  |  |     icon="air" | 
					
						
							|  |  |  |     title="Ingest with Airflow" | 
					
						
							|  |  |  |     text="Configure the ingestion using Airflow SDK" | 
					
						
							| 
									
										
										
										
											2022-10-05 21:54:02 -07:00
										 |  |  |     link="/connectors/ml-model/mlflow/airflow" | 
					
						
							| 
									
										
										
										
											2022-08-27 02:57:09 +02:00
										 |  |  |     size="half" | 
					
						
							|  |  |  |   /> | 
					
						
							|  |  |  |   <Tile | 
					
						
							|  |  |  |     icon="account_tree" | 
					
						
							|  |  |  |     title="Ingest with the CLI" | 
					
						
							|  |  |  |     text="Run a one-time ingestion using the metadata CLI" | 
					
						
							| 
									
										
										
										
											2022-10-05 21:54:02 -07:00
										 |  |  |     link="/connectors/ml-model/mlflow/cli" | 
					
						
							| 
									
										
										
										
											2022-08-27 02:57:09 +02:00
										 |  |  |     size="half" | 
					
						
							|  |  |  |   /> | 
					
						
							|  |  |  | </TileContainer> | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | ## Requirements
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <InlineCallout color="violet-70" icon="description" bold="OpenMetadata 0.12 or later" href="/deployment"> | 
					
						
							|  |  |  | To deploy OpenMetadata, check the <a href="/deployment">Deployment</a> guides. | 
					
						
							|  |  |  | </InlineCallout> | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with | 
					
						
							|  |  |  | custom Airflow plugins to handle the workflow deployment. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | ## Metadata Ingestion
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | ### 1. Visit the Services Page
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | The first step is ingesting the metadata from your sources. Under | 
					
						
							|  |  |  | Settings, you will find a Services link an external source system to | 
					
						
							|  |  |  | OpenMetadata. Once a service is created, it can be used to configure | 
					
						
							|  |  |  | metadata, usage, and profiler workflows. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | To visit the Services page, select Services from the Settings menu. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <Image | 
					
						
							|  |  |  | src="/images/openmetadata/connectors/visit-services.png" | 
					
						
							|  |  |  | alt="Visit Services Page" | 
					
						
							|  |  |  | caption="Find Services under the Settings menu" | 
					
						
							|  |  |  | /> | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | ### 2. Create a New Service
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | Click on the Add New Service button to start the Service creation. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <Image | 
					
						
							|  |  |  | src="/images/openmetadata/connectors/create-service.png" | 
					
						
							|  |  |  | alt="Create a new service" | 
					
						
							|  |  |  | caption="Add a new Service from the Services page" | 
					
						
							|  |  |  | /> | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | ### 3. Select the Service Type
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | Select Mlflow as the service type and click Next. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <div className="w-100 flex justify-center"> | 
					
						
							|  |  |  | <Image | 
					
						
							|  |  |  |   src="/images/openmetadata/connectors/mlflow/select-service.png" | 
					
						
							|  |  |  |   alt="Select Service" | 
					
						
							|  |  |  |   caption="Select your service from the list" | 
					
						
							|  |  |  | /> | 
					
						
							|  |  |  | </div> | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | ### 4. Name and Describe your Service
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | Provide a name and description for your service as illustrated below. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | #### Service Name
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | OpenMetadata uniquely identifies services by their Service Name. Provide | 
					
						
							|  |  |  | a name that distinguishes your deployment from other services, including | 
					
						
							|  |  |  | the other {connector} services that you might be ingesting metadata | 
					
						
							|  |  |  | from. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <div className="w-100 flex justify-center"> | 
					
						
							|  |  |  | <Image | 
					
						
							|  |  |  |   src="/images/openmetadata/connectors/mlflow/add-new-service.png" | 
					
						
							|  |  |  |   alt="Add New Service" | 
					
						
							|  |  |  |   caption="Provide a Name and description for your Service" | 
					
						
							|  |  |  | /> | 
					
						
							|  |  |  | </div> | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | ### 5. Configure the Service Connection
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | In this step, we will configure the connection settings required for | 
					
						
							|  |  |  | this connector. Please follow the instructions below to ensure that | 
					
						
							|  |  |  | you've configured the connector to read from your mlflow service as | 
					
						
							|  |  |  | desired. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <div className="w-100 flex justify-center"> | 
					
						
							|  |  |  | <Image | 
					
						
							|  |  |  |   src="/images/openmetadata/connectors/mlflow/service-connection.png" | 
					
						
							|  |  |  |   alt="Configure service connection" | 
					
						
							|  |  |  |   caption="Configure the service connection by filling the form" | 
					
						
							|  |  |  | /> | 
					
						
							|  |  |  | </div> | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | Once the credentials have been added, click on `Test Connection` and Save | 
					
						
							|  |  |  | the changes. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <div className="w-100 flex justify-center"> | 
					
						
							|  |  |  | <Image | 
					
						
							|  |  |  |   src="/images/openmetadata/connectors/test-connection.png" | 
					
						
							|  |  |  |   alt="Test Connection" | 
					
						
							|  |  |  |   caption="Test the connection and save the Service" | 
					
						
							|  |  |  | /> | 
					
						
							|  |  |  | </div> | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | #### Connection Options
 | 
					
						
							| 
									
										
										
										
											2022-08-26 12:30:34 +05:30
										 |  |  | 
 | 
					
						
							|  |  |  | - **trackingUri**: Mlflow Experiment tracking URI. E.g., http://localhost:5000 | 
					
						
							|  |  |  | - **registryUri**: Mlflow Model registry backend. E.g., mysql+pymysql://mlflow:password@localhost:3307/experiments | 
					
						
							|  |  |  | 
 | 
					
						
							| 
									
										
										
										
											2022-08-27 02:57:09 +02:00
										 |  |  | ### 6. Configure Metadata Ingestion
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | In this step we will configure the metadata ingestion pipeline, | 
					
						
							|  |  |  | Please follow the instructions below | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <Image | 
					
						
							| 
									
										
										
										
											2022-08-29 08:59:41 +02:00
										 |  |  | src="/images/openmetadata/connectors/configure-metadata-ingestion-ml-model.png" | 
					
						
							| 
									
										
										
										
											2022-08-27 02:57:09 +02:00
										 |  |  | alt="Configure Metadata Ingestion" | 
					
						
							|  |  |  | caption="Configure Metadata Ingestion Page" | 
					
						
							|  |  |  | /> | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | #### Metadata Ingestion Options
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | - **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | ### 7. Schedule the Ingestion and Deploy
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | Scheduling can be set up at an hourly, daily, or weekly cadence. The | 
					
						
							|  |  |  | timezone is in UTC. Select a Start Date to schedule for ingestion. It is | 
					
						
							|  |  |  | optional to add an End Date. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | Review your configuration settings. If they match what you intended, | 
					
						
							|  |  |  | click Deploy to create the service and schedule metadata ingestion. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | If something doesn't look right, click the Back button to return to the | 
					
						
							|  |  |  | appropriate step and change the settings as needed. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <Image | 
					
						
							|  |  |  | src="/images/openmetadata/connectors/schedule.png" | 
					
						
							|  |  |  | alt="Schedule the Workflow" | 
					
						
							|  |  |  | caption="Schedule the Ingestion Pipeline and Deploy" | 
					
						
							|  |  |  | /> | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | After configuring the workflow, you can click on Deploy to create the | 
					
						
							|  |  |  | pipeline. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | ### 8. View the Ingestion Pipeline
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | Once the workflow has been successfully deployed, you can view the | 
					
						
							|  |  |  | Ingestion Pipeline running from the Service Page. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <Image | 
					
						
							|  |  |  | src="/images/openmetadata/connectors/view-ingestion-pipeline.png" | 
					
						
							|  |  |  | alt="View Ingestion Pipeline" | 
					
						
							|  |  |  | caption="View the Ingestion Pipeline from the Service Page" | 
					
						
							|  |  |  | /> | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | ### 9. Workflow Deployment Error
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | If there were any errors during the workflow deployment process, the | 
					
						
							|  |  |  | Ingestion Pipeline Entity will still be created, but no workflow will be | 
					
						
							|  |  |  | present in the Ingestion container. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | You can then edit the Ingestion Pipeline and Deploy it again. | 
					
						
							| 
									
										
										
										
											2022-08-26 12:30:34 +05:30
										 |  |  | 
 | 
					
						
							| 
									
										
										
										
											2022-08-27 02:57:09 +02:00
										 |  |  | <Image | 
					
						
							|  |  |  | src="/images/openmetadata/connectors/workflow-deployment-error.png" | 
					
						
							|  |  |  | alt="Workflow Deployment Error" | 
					
						
							|  |  |  | caption="Edit and Deploy the Ingestion Pipeline" | 
					
						
							|  |  |  | /> | 
					
						
							| 
									
										
										
										
											2022-08-26 12:30:34 +05:30
										 |  |  | 
 | 
					
						
							| 
									
										
										
										
											2022-08-27 02:57:09 +02:00
										 |  |  | From the Connection tab, you can also Edit the Service if needed. |