OpenMetadata/docs/openmetadata-apis/schemas/entities/pipeline.md

Pipeline

Pipeline

This schema defines the Pipeline entity. A pipeline enables the flow of data from source to destination through a series of processing steps. ETL is a type of pipeline where the series of steps Extract, Transform and Load the data.

$id:https://open-metadata.org/schema/entity/data/pipeline.json

Type: object

This schema does not accept additional properties.

Properties

<<<<<<< HEAD

• id required
  • Unique identifier that identifies a pipeline instance.
  • $ref: ../../type/basic.json#/definitions/uuid
• name required
  • Name that identifies this pipeline instance uniquely.
  • Type: string
  • Length: between 1 and 128
• displayName
  • Display Name that identifies this Pipeline. It could be title or label from the source services.
  • Type: string
• fullyQualifiedName
  • A unique name that identifies a pipeline in the format 'ServiceName.PipelineName'.
  • Type: string
• description
  • Description of this Pipeline.
  • Type: string
• version
  • Metadata version of the entity.
  • $ref: ../../type/entityHistory.json#/definitions/entityVersion
• updatedAt
  • Last update time corresponding to the new version of the entity in Unix epoch time milliseconds.
  • $ref: ../../type/basic.json#/definitions/timestamp
• updatedBy
  • User who made the update.
  • Type: string
• pipelineUrl
  • Pipeline URL to visit/manage. This URL points to respective pipeline service UI.
  • Type: string
  • String format must be a "uri"
• concurrency
  • Concurrency of the Pipeline.
  • Type: integer
• pipelineLocation
  • Pipeline Code Location.
  • Type: string
• startDate
  • Start date of the workflow.
  • $ref: ../../type/basic.json#/definitions/dateTime
• tasks
  • All the tasks that are part of pipeline.
  • Type: array
    • Items
    • $ref: #/definitions/task
• pipelineStatus
  • Series of pipeline executions and its status.
  • Type: array
    • Items
    • $ref: #/definitions/pipelineStatus
• followers
  • Followers of this Pipeline.
  • $ref: ../../type/entityReference.json#/definitions/entityReferenceList
• tags
  • Tags for this Pipeline.
  • Type: array
    • Items
    • $ref: ../../type/tagLabel.json
• href
  • Link to the resource corresponding to this entity.
  • $ref: ../../type/basic.json#/definitions/href
• owner
  • Owner of this pipeline.
  • $ref: ../../type/entityReference.json
• service required
  • Link to service where this pipeline is hosted in.
  • $ref: ../../type/entityReference.json
• serviceType
  • Service type where this pipeline is hosted in.
  • $ref: ../services/pipelineService.json#/definitions/pipelineServiceType
• changeDescription
  • Change that lead to this version of the entity.
  • $ref: ../../type/entityHistory.json#/definitions/changeDescription
• deleted
  • When true indicates the entity has been soft deleted.
  • Type: boolean
  • Default: false

a07bc411 (updated json schema and schema docs (#3219))

• id required
  • Unique identifier that identifies a pipeline instance.
  • $ref: ../../type/basic.json#/definitions/uuid
• name required
  • Name that identifies this pipeline instance uniquely.
  • Type: string
  • Length: between 1 and 128
• displayName
  • Display Name that identifies this Pipeline. It could be title or label from the source services.
  • Type: string
• fullyQualifiedName
  • A unique name that identifies a pipeline in the format 'ServiceName.PipelineName'.
  • Type: string
• description
  • Description of this Pipeline.
  • Type: string
• version
  • Metadata version of the entity.
  • $ref: ../../type/entityHistory.json#/definitions/entityVersion
• updatedAt
  • Last update time corresponding to the new version of the entity in Unix epoch time milliseconds.
  • $ref: ../../type/basic.json#/definitions/timestamp
• updatedBy
  • User who made the update.
  • Type: string
• pipelineUrl
  • Pipeline URL to visit/manage. This URL points to respective pipeline service UI.
  • Type: string
  • String format must be a "uri"
• concurrency
  • Concurrency of the Pipeline.
  • Type: integer
• pipelineLocation
  • Pipeline Code Location.
  • Type: string
• startDate
  • Start date of the workflow.
  • $ref: ../../type/basic.json#/definitions/dateTime
• tasks
  • All the tasks that are part of pipeline.
  • Type: array
    • Items
    • $ref: #/definitions/task
• followers
  • Followers of this Pipeline.
  • $ref: ../../type/entityReference.json#/definitions/entityReferenceList
• tags
  • Tags for this Pipeline.
  • Type: array
    • Items
    • $ref: ../../type/tagLabel.json
• href
  • Link to the resource corresponding to this entity.
  • $ref: ../../type/basic.json#/definitions/href
• owner
  • Owner of this pipeline.
  • $ref: ../../type/entityReference.json
• service required
  • Link to service where this pipeline is hosted in.
  • $ref: ../../type/entityReference.json
• serviceType
  • Service type where this pipeline is hosted in.
  • $ref: ../services/pipelineService.json#/definitions/pipelineServiceType
• changeDescription
  • Change that lead to this version of the entity.
  • $ref: ../../type/entityHistory.json#/definitions/changeDescription
• deleted
  • When true indicates the entity has been soft deleted.
  • Type: boolean
  • Default: false

Type definitions in this schema

<<<<<<< HEAD

task

• Type: object
• Properties
  • name required
    • Name that identifies this task instance uniquely.
    • Type: string
  • displayName
    • Display Name that identifies this Task. It could be title or label from the pipeline services.
    • Type: string
  • fullyQualifiedName
    • A unique name that identifies a pipeline in the format 'ServiceName.PipelineName.TaskName'.
    • Type: string
  • description
    • Description of this Task.
    • Type: string
  • taskUrl
    • Task URL to visit/manage. This URL points to respective pipeline service UI.
    • Type: string
    • String format must be a "uri"
  • downstreamTasks
    • All the tasks that are downstream of this task.
    • Type: array
      • Items
      • Type: string
  • taskType
    • Type of the Task. Usually refers to the class it implements.
    • Type: string
  • taskSQL
    • SQL used in the task. Can be used to determine the lineage.
    • $ref: ../../type/basic.json#/definitions/sqlQuery
  • tags
    • Tags for this task.
    • Type: array
      • Items
      • $ref: ../../type/tagLabel.json

This document was updated on: Tuesday, January 25, 2022

statusType

• Enum defining the possible Status.
• Type: string
• The value is restricted to the following:
  1. "Successful"
  2. "Failed"
  3. "Pending"

taskStatus

• This schema defines a time series of the status of a Pipeline or Task.
• Type: object
• This schema does not accept additional properties.
• Properties
  • name
    • Name of the Task.
    • Type: string
  • executionStatus
    • Status at a specific execution date.
    • $ref: #/definitions/statusType

task

• Type: object
• This schema does not accept additional properties.
• Properties
  • name required
    • Name that identifies this task instance uniquely.
    • Type: string
  • displayName
    • Display Name that identifies this Task. It could be title or label from the pipeline services.
    • Type: string
  • fullyQualifiedName
    • A unique name that identifies a pipeline in the format 'ServiceName.PipelineName.TaskName'.
    • Type: string
  • description
    • Description of this Task.
    • Type: string
  • taskUrl
    • Task URL to visit/manage. This URL points to respective pipeline service UI.
    • Type: string
    • String format must be a "uri"
  • downstreamTasks
    • All the tasks that are downstream of this task.
    • Type: array
      • Items
      • Type: string
  • taskType
    • Type of the Task. Usually refers to the class it implements.
    • Type: string
  • taskSQL
    • SQL used in the task. Can be used to determine the lineage.
    • $ref: ../../type/basic.json#/definitions/sqlQuery
  • tags
    • Tags for this task.
    • Type: array
      • Items
      • $ref: ../../type/tagLabel.json

pipelineStatus

• Series of pipeline executions, its status and task status.
• Type: object
• This schema does not accept additional properties.
• Properties
  • executionDate
    • Date where the job was executed.
    • $ref: ../../type/basic.json#/definitions/timestamp
  • executionStatus
    • Status at a specific execution date.
    • $ref: #/definitions/statusType
  • taskStatus
    • Series of task executions and its status.
    • Type: array
      • Items
      • $ref: #/definitions/taskStatus

This document was updated on: Monday, March 7, 2022

a07bc411 (updated json schema and schema docs (#3219))