OpenMetadata/docs/install/metadata-ingestion/scheduler.md

description
This guide will help install Sdscheduler and schedule connectors for ingestion.

Scheduler

{% hint style="info" %} Prerequisites

OpenMetadata is built using Java, DropWizard, Jetty and MySQL.

1. Python 3.7 or above

2. Create an env

   python3 -m venv env
   

3. Activating the enviroment

   source env/bin/activate
   

{% endhint %}

Install dependencies

cd ingestion
python3 -m venv env
source env/bin/activate
pip install '.[scheduler]'
python ingestion_scheduler/scheduler.py

{% hint style="warning" %} Note:

Different Connectors require different dependencies, please go through Connectors Documentation install dependencies as needed. {% endhint %}

Scheduler UI

Main Page

Loads all the Json connectors inside the pipeline directory as cron jobs.

Custom run a job

• Click on Custom Run button under Action column of the jobs.
• Click on Run.
• The Job will start running the ingestion.

Status of an executed job

• Click on Executions tab under Navigation Bar.

Scheduler API

• /api/v1/jobs
  • Fetch data related to all the jobs.
• /api/v1/jobs/{job_id}
  • Fetch data related to specific job.
• /api/v1/executions
  • Fetch results of all executions.
• /api/v1/executions/{execution_id}
  • Fetch results of particular execution of the job.
• /api/v1/status
  • Fetch results of the current running ingestion job.
• /api/v1/status/{job_id}
  • Fetch recent execution results of the job id.