OpenMetadata/docs/openmetadata-apis/schemas/entities/pipeline.md

Pipeline

This schema defines the Pipeline entity. A pipeline enables the flow of data from source to destination through a series of processing steps. ETL is a type of pipeline where the series of steps Extract, Transform and Load the data.

$id:https://open-metadata.org/schema/entity/data/pipeline.json

Type: object

Properties

• id required
  • Unique identifier that identifies a pipeline instance.
  • $ref: ../../type/basic.json#/definitions/uuid
• name required
  • Name that identifies this pipeline instance uniquely.
  • Type: string
  • Length: between 1 and 64
• displayName
  • Display Name that identifies this Pipeline. It could be title or label from the source services.
  • Type: string
• fullyQualifiedName
  • A unique name that identifies a pipeline in the format 'ServiceName.PipelineName'.
  • Type: string
  • Length: between 1 and 64
• description
  • Description of this Pipeline.
  • Type: string
• version
  • Metadata version of the entity.
  • $ref: ../../type/entityHistory.json#/definitions/entityVersion
• updatedAt
  • Last update time corresponding to the new version of the entity.
  • $ref: ../../type/basic.json#/definitions/dateTime
• updatedBy
  • User who made the update.
  • Type: string
• pipelineUrl
  • Pipeline URL to visit/manage. This URL points to respective pipeline service UI.
  • Type: string
  • String format must be a "uri"
• concurrency
  • Concurrency of the Pipeline.
  • Type: integer
• pipelineLocation
  • Pipeline Code Location.
  • Type: string
• startDate
  • Start date of the workflow.
  • $ref: ../../type/basic.json#/definitions/dateTime
• tasks
  • All the tasks that are part of pipeline.
  • Type: array
    • Items
    • $ref: #/definitions/task
• followers
  • Followers of this Pipeline.
  • $ref: ../../type/entityReference.json#/definitions/entityReferenceList
• tags
  • Tags for this Pipeline.
  • Type: array
    • Items
    • $ref: ../../type/tagLabel.json
• href
  • Link to the resource corresponding to this entity.
  • $ref: ../../type/basic.json#/definitions/href
• owner
  • Owner of this pipeline.
  • $ref: ../../type/entityReference.json
• service required
  • Link to service where this pipeline is hosted in.
  • $ref: ../../type/entityReference.json
• changeDescription
  • Change that lead to this version of the entity.
  • $ref: ../../type/entityHistory.json#/definitions/changeDescription

Type definitions in this schema

task

• Type: object
• Properties
  • name required
    • Name that identifies this task instance uniquely.
    • Type: string
    • Length: between 1 and 64
  • displayName
    • Display Name that identifies this Task. It could be title or label from the pipeline services.
    • Type: string
  • fullyQualifiedName
    • A unique name that identifies a pipeline in the format 'ServiceName.PipelineName.TaskName'.
    • Type: string
    • Length: between 1 and 64
  • description
    • Description of this Task.
    • Type: string
  • taskUrl
    • Task URL to visit/manage. This URL points to respective pipeline service UI.
    • Type: string
    • String format must be a "uri"
  • downstreamTasks
    • All the tasks that are downstream of this task.
    • Type: array
      • Items
      • Type: string
      • Length: between 1 and 64
  • taskType
    • Type of the Task. Usually refers to the class it implements.
    • Type: string
  • taskSQL
    • SQL used in the task. Can be used to determine the lineage.
    • $ref: ../../type/basic.json#/definitions/sqlQuery
  • tags
    • Tags for this task.
    • Type: array
      • Items
      • $ref: ../../type/tagLabel.json

This document was updated on: Monday, November 15, 2021