Updated doc with latest json schema (#743)

Co-authored-by: Parth Panchal <parthpanchal@Parths-MacBook-Air.local>
This commit is contained in:
parthp2107 2021-10-12 14:28:29 +05:30 committed by GitHub
parent f0e59f2fea
commit e4ac8499a7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
34 changed files with 382 additions and 99 deletions

View File

@ -13,6 +13,7 @@
* [Basic Types](openmetadata-apis/schemas/types/basic.md)
* [Audit Log](openmetadata-apis/schemas/types/auditlog.md)
* [Daily Count](openmetadata-apis/schemas/types/dailycount.md)
* [Entity Lineage](openmetadata-apis/schemas/types/entitylineage.md)
* [Entity Reference](openmetadata-apis/schemas/types/entityreference.md)
* [Entity Usage](openmetadata-apis/schemas/types/entityusage.md)
* [Jdbc Connection](openmetadata-apis/schemas/types/jdbcconnection.md)
@ -31,12 +32,15 @@
* [Messaging Service](openmetadata-apis/schemas/entities/messagingservice.md)
* [Feed](openmetadata-apis/schemas/entities/thread.md)
* [Metrics](openmetadata-apis/schemas/entities/metrics.md)
* [Model](openmetadata-apis/schemas/entities/model.md)
* [Pipeline](openmetadata-apis/schemas/entities/pipeline.md)
* [Pipeline Service](openmetadata-apis/schemas/entities/pipelineservice.md)
* [Report](openmetadata-apis/schemas/entities/report.md)
* [Table](openmetadata-apis/schemas/entities/table.md)
* [Tag Category](openmetadata-apis/schemas/entities/tagcategory.md)
* [Task](openmetadata-apis/schemas/entities/task.md)
* [Team](openmetadata-apis/schemas/entities/team.md)
* [Topic](openmetadata-apis/schemas/entities/topic.md)
* [Tag Category](openmetadata-apis/schemas/entities/tagcategory.md)
* [User](openmetadata-apis/schemas/entities/user.md)
* [APIs](openmetadata-apis/apis/README.md)
* [Overview](openmetadata-apis/apis/overview.md)

View File

@ -29,17 +29,22 @@ OpenMetadata has the following high-level core entities and relationships:
{% page-ref page="metrics.md" %}
{% page-ref page="model.md" %}
{% page-ref page="pipeline.md" %}
{% page-ref page="pipelineservice.md" %}
{% page-ref page="report.md" %}
{% page-ref page="table.md" %}
{% page-ref page="tagcategory.md" %}
{% page-ref page="task.md" %}
{% page-ref page="team.md" %}
{% page-ref page="topic.md" %}
{% page-ref page="user.md" %}
{% page-ref page="tagcategory.md" %}

View File

@ -24,4 +24,4 @@ Type: `object`
- Link to the resource corresponding to this bot.
- $ref: [../type/basic.json#/definitions/href](../types/basic.md#href)
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -74,4 +74,4 @@ Type: `object`
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -54,4 +54,4 @@ Type: `object`
- Latest usage information for this database.
- $ref: [../../type/usageDetails.json](../types/usagedetails.md)
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -51,4 +51,4 @@ Type: `object`
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -46,4 +46,4 @@ Type: `object`
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -47,7 +47,7 @@ Type: `object`
8. _"Oracle"_
9. _"Athena"_
10. _"Presto"_
11. _"Vertica"_
12. _"Trino"_
11. _"Trino"_
12. _"Vertica"_
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -54,4 +54,4 @@ Type: `object`
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -34,4 +34,4 @@ Type: `object`
- Latest usage information for this database.
- $ref: [../../type/usageDetails.json](../types/usagedetails.md)
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -0,0 +1,52 @@
# Model
This schema defines the Model entity. Models are algorithms trained on data to find patterns or make predictions.
**$id: https://open-metadata.org/schema/entity/data/model.json**
Type: `object`
## Properties
- **id** `required`
- Unique identifier of a model instance.
- $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
- **name** `required`
- Name that identifies this model.
- Type: `string`
- Length: between 1 and 64
- **fullyQualifiedName**
- A unique name that identifies a model.
- Type: `string`
- Length: between 1 and 64
- **displayName**
- Display Name that identifies this model.
- Type: `string`
- **description**
- Description of the model, what it is, and how to use it.
- Type: `string`
- **algorithm** `required`
- Algorithm used to train the model.
- Type: `string`
- **dashboard**
- Performance Dashboard URL to track metric evolution.
- $ref: [../../type/entityReference.json](../types/entityreference.md)
- **href**
- Link to the resource corresponding to this entity.
- $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
- **owner**
- Owner of this model.
- $ref: [../../type/entityReference.json](../types/entityreference.md)
- **followers**
- Followers of this model.
- $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
- **tags**
- Tags for this model.
- Type: `array`
- **Items**
- $ref: [../../type/tagLabel.json](../types/taglabel.md)
- **usageSummary**
- Latest usage information for this model.
- $ref: [../../type/usageDetails.json](../types/usagedetails.md)
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -14,13 +14,42 @@ Type: `object`
- Name that identifies this pipeline instance uniquely.
- Type: `string`
- Length: between 1 and 64
- **displayName**
- Display Name that identifies this Pipeline. It could be title or label from the source services.
- Type: `string`
- **fullyQualifiedName**
- A unique name that identifies a pipeline in the format 'ServiceName.PipelineName'.
- Type: `string`
- Length: between 1 and 64
- **description**
- Description of this pipeline.
- Description of this Pipeline.
- Type: `string`
- **pipelineUrl**
- Pipeline URL to visit/manage. This URL points to respective pipeline service UI.
- Type: `string`
- String format must be a "uri"
- **concurrency**
- Concurrency of the Pipeline.
- Type: `integer`
- **pipelineLocation**
- Pipeline Code Location.
- Type: `string`
- **startDate**
- Start date of the workflow.
- $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime)
- **tasks**
- All the tasks that are part of pipeline.
- Type: `array`
- **Items**
- $ref: [../../type/entityReference.json](../types/entityreference.md)
- **followers**
- Followers of this Pipeline.
- $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
- **tags**
- Tags for this Pipeline.
- Type: `array`
- **Items**
- $ref: [../../type/tagLabel.json](../types/taglabel.md)
- **href**
- Link to the resource corresponding to this entity.
- $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
@ -31,4 +60,4 @@ Type: `object`
- Link to service where this pipeline is hosted in.
- $ref: [../../type/entityReference.json](../types/entityreference.md)
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -0,0 +1,45 @@
# Pipeline Service
This schema defines the Pipeline Service entity, such as Airflow and Prefect.
**$id: https://open-metadata.org/schema/entity/services/messagingservice.json**
Type: `object`
## Properties
- **id** `required`
- Unique identifier of this pipeline service instance.
- $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
- **name** `required`
- Name that identifies this pipeline service.
- Type: `string`
- Length: between 1 and 64
- **serviceType**
- Type of pipeline service such as Airflow or Prefect...
- $ref: [#/definitions/pipelineServiceType](#pipelineservicetype)
- **description**
- Description of a pipeline service instance.
- Type: `string`
- **pipelineUrl** `required`
- Pipeline Service Management/UI URL.
- Type: `string`
- String format must be a "uri"
- **ingestionSchedule**
- Schedule for running metadata ingestion jobs.
- $ref: [../../type/schedule.json](../types/schedule.md)
- **href**
- Link to the resource corresponding to this pipeline service.
- $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
## Type definitions in this schema
### pipelineServiceType
- Type of pipeline service - Airflow or Prefect.
- Type: `string`
- The value is restricted to the following:
1. _"Airflow"_
2. _"Prefect"_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -34,4 +34,4 @@ Type: `object`
- Latest usage information for this database.
- $ref: [../../type/usageDetails.json](../types/usagedetails.md)
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -80,7 +80,7 @@ Type: `object`
5. _"MaterializedView"_
### columnDataType
### dataType
- This enum defines the type of data stored in a column.
- Type: `string`
@ -90,37 +90,38 @@ Type: `object`
3. _"SMALLINT"_
4. _"INT"_
5. _"BIGINT"_
6. _"FLOAT"_
7. _"DOUBLE"_
8. _"DECIMAL"_
9. _"NUMERIC"_
10. _"TIMESTAMP"_
11. _"TIME"_
12. _"DATE"_
13. _"DATETIME"_
14. _"INTERVAL"_
15. _"STRING"_
16. _"MEDIUMTEXT"_
17. _"TEXT"_
18. _"CHAR"_
19. _"VARCHAR"_
20. _"BOOLEAN"_
21. _"BINARY"_
22. _"VARBINARY"_
23. _"ARRAY"_
24. _"BLOB"_
25. _"LONGBLOB"_
26. _"MEDIUMBLOB"_
27. _"MAP"_
28. _"STRUCT"_
29. _"UNION"_
30. _"SET"_
31. _"GEOGRAPHY"_
32. _"ENUM"_
33. _"JSON"_
6. _"BYTEINT"_
7. _"FLOAT"_
8. _"DOUBLE"_
9. _"DECIMAL"_
10. _"NUMERIC"_
11. _"TIMESTAMP"_
12. _"TIME"_
13. _"DATE"_
14. _"DATETIME"_
15. _"INTERVAL"_
16. _"STRING"_
17. _"MEDIUMTEXT"_
18. _"TEXT"_
19. _"CHAR"_
20. _"VARCHAR"_
21. _"BOOLEAN"_
22. _"BINARY"_
23. _"VARBINARY"_
24. _"ARRAY"_
25. _"BLOB"_
26. _"LONGBLOB"_
27. _"MEDIUMBLOB"_
28. _"MAP"_
29. _"STRUCT"_
30. _"UNION"_
31. _"SET"_
32. _"GEOGRAPHY"_
33. _"ENUM"_
34. _"JSON"_
### columnConstraint
### constraint
- This enum defines the type for column constraint.
- Type: `string`
@ -152,7 +153,7 @@ Type: `object`
### columnName
- Local name (not fully qualified name) of the column.
- Local name (not fully qualified name) of the column. ColumnName is `-` when the column is not named in struct dataType. For example, BigQuery supports struct with unnamed fields.
- Type: `string`
- The value must match this pattern: `^[^.]*$`
- Length: between 1 and 64
@ -168,7 +169,7 @@ Type: `object`
### fullyQualifiedColumnName
- Fully qualified name of the column that includes `serviceName.databaseName.tableName.columnName`.
- Fully qualified name of the column that includes `serviceName.databaseName.tableName.columnName[.nestedColumnName]`. When columnName is null for dataType struct fields, `field_#` where `#` is field index is used. For map dataType, for key the field name `key` is used and for the value field `value` is used.
- Type: `string`
- Length: between 1 and 256
@ -177,12 +178,22 @@ Type: `object`
- This schema defines the type for a column in a table.
- Type: `object`
- This schema <u>does not</u> accept additional properties.
- **Properties**
- **name** `required`
- $ref: [#/definitions/columnName](#columnname)
- **columnDataType** `required`
- **dataType** `required`
- Data type of the column (int, date etc.).
- $ref: [#/definitions/columnDataType](#columndatatype)
- $ref: [#/definitions/dataType](#datatype)
- **arrayDataType**
- Data type used array in dataType. For example, `array<int>` has dataType as `array` and arrayDataType as `int`.
- $ref: [#/definitions/dataType](#datatype)
- **dataLength**
- Length of `char`, `varchar`, `binary`, `varbinary` `dataTypes`, else null. For example, `varchar(20)` has dataType as `varchar` and dataLength as `20`.
- Type: `integer`
- **dataTypeDisplay**
- Display name used for dataType. This is useful for complex types, such as `array<int>, map<int,string>, struct<>, and union types.
- Type: `string`
- **description**
- Description of the column.
- Type: `string`
@ -193,12 +204,20 @@ Type: `object`
- Type: `array`
- **Items**
- $ref: [../../type/tagLabel.json](../types/taglabel.md)
- **columnConstraint**
- **constraint**
- Column level constraint.
- $ref: [#/definitions/columnConstraint](#columnconstraint)
- $ref: [#/definitions/constraint](#constraint)
- **ordinalPosition**
- Ordinal position of the column.
- Type: `integer`
- **jsonSchema**
- Json schema only if the dataType is JSON else null.
- Type: `string`
- **children**
- Child columns if dataType or arrayDataType is `map`, `struct`, or `union` else `null`.
- Type: `array`
- **Items**
- $ref: [#/definitions/column](#column)
### columnJoins
@ -278,21 +297,21 @@ Type: `object`
- **nullProportion**
- No.of null value proportion in columns.
- Type: `number`
- **min**
- Minimum value in a column.
- Type: `string`
- **max**
- Maximum value in a column.
- Type: `string`
- **mean**
- Avg value in a column.
- Type: `string`
- **median**
- Median value in a column.
- Type: `string`
- **stddev**
- Standard deviation of a column.
- Type: `number`
- **min**
- Minimum value in a column.
- Type: `string`
- **max**
- Maximum value in a column.
- Type: `string`
- **mean**
- Avg value in a column.
- Type: `string`
- **median**
- Median value in a column.
- Type: `string`
- **stddev**
- Standard deviation of a column.
- Type: `number`
### tableProfile
@ -318,4 +337,4 @@ Type: `object`
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -49,6 +49,4 @@ This schema <u>does not</u> accept additional properties.
### tag
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -0,0 +1,65 @@
# Task
This schema defines the Task entity. A task is a unit of computation in a Pipeline.
**id: https://open-metadata.org/schema/entity/data/task.json**
Type: `object`
## Properties
- **id** `required`
- Unique identifier that identifies a task instance.
- $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
- **name** `required`
- Name that identifies this task instance uniquely.
- Type: `string`
- Length: between 1 and 64
- **displayName**
- Display Name that identifies this Task. It could be title or label from the pipeline services.
- Type: `string`
- **fullyQualifiedName**
- A unique name that identifies a pipeline in the format 'ServiceName.PipelineName.TaskName'.
- Type: `string`
- Length: between 1 and 64
- **description**
- Description of this Task.
- Type: `string`
- **taskUrl**
- Task URL to visit/manage. This URL points to respective pipeline service UI.
- Type: `string`
- String format must be a "uri"
- **downstreamTasks**
- All the tasks that are downstream of this task.
- Type: `array`
- **Items**
- Type: `string`
- Length: between 1 and 64
- **taskType**
- Type of the Task. Usually refers to the class it implements.
- Type: `string`
- **taskSQL**
- SQL used in the task. Can be used to determine the lineage.
- Type: `string`
- **startDate**
- Start date of the task.
- $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime)
- **endDate**
- End date of the task.
- $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime)
- **tags**
- Tags for this Pipeline.
- Type: `array`
- **Items**
- $ref: [../../type/tagLabel.json](../types/taglabel.md)
- **href**
- Link to the resource corresponding to this entity.
- $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
- **owner**
- Owner of this pipeline.
- $ref: [../../type/entityReference.json](../types/entityreference.md)
- **service** `required`
- Link to service where this pipeline is hosted in.
- $ref: [../../type/entityReference.json](../types/entityreference.md)
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -43,4 +43,4 @@ Type: `object`
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -5,6 +5,7 @@ This schema defines the Thread entity. A Thread is a collection of posts made by
**$id: https://open-metadata.org/schema/entity/feed/thread.json**
Type: `object`
## Properties
- **id** `required`
- Unique identifier that identifies an entity instance.
@ -36,14 +37,14 @@ Type: `object`
- **message** `required`
- Message in markdown format. See markdown support for more details.
- Type: `string`
- **postTs**
- Timestamp of the post.
- Type: `string`
- String format must be a "date-time"
- **from** `required`
- ID of User (regular user or a bot) posting the message.
- $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
- **postTs**
- Timestamp of the post.
- Type: `string`
- String format must be a "date-time"
- **from** `required`
- ID of User (regular user or a bot) posting the message.
- $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -97,4 +97,4 @@ Type: `object`
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -59,4 +59,4 @@ This schema <u>does not</u> accept additional properties.
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -8,6 +8,8 @@ OpenMetadata supports many common types reused across different schemas. Definin
{% page-ref page="dailycount.md" %}
{% page-ref page="entitylineage.md" %}
{% page-ref page="entityreference.md" %}
{% page-ref page="entityusage.md" %}

View File

@ -34,4 +34,4 @@ Type: `object`
- Type of Entity that is modified by the operation.
- Type: `string`
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -84,4 +84,4 @@ This schema defines basic common types that are used by other schemas.
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -16,4 +16,4 @@ This schema <u>does not</u> accept additional properties.
- **date** `required`
- $ref: [basic.json#/definitions/date](basic.md#date)
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -0,0 +1,60 @@
# Entity Lineage
This schema defines the type used for lineage of an entity.
**$id: https://open-metadata.org/schema/type/entitylineage.json**
Type: `object`
This schema <u>does not</u> accept additional properties.
## Properties
- **entity** `required`
- Primary entity for which this lineage graph is created.
- $ref: [entityReference.json](entityreference.md)
- **nodes**
- Type: `array`
- **Items**
- $ref: [entityReference.json](entityreference.md)
- **upstreamEdges**
- Type: `array`
- **Items**
- $ref: [#/definitions/edge](#edge)
- **downstreamEdges**
- Type: `array`
- **Items**
- $ref: [#/definitions/edge](#edge)
## Type definitions in this schema
### edge
- Edge in the lineage graph from one entity to another by entity IDs.
- Type: `object`
- **Properties**
- **fromEntity**
- From entity that is upstream of lineage edge.
- $ref: [basic.json#/definitions/uuid](basic.md#uuid)
- **toEntity**
- To entity that is downstream of lineage edge.
- $ref: [basic.json#/definitions/uuid](basic.md#uuid)
- **description**
- Type: `string`
### entitiesEdge
- Edge in the lineage graph from one entity to another using entity references.
- Type: `object`
- **Properties**
- **fromEntity**
- From entity that is upstream of lineage edge.
- $ref: [entityReference.json](entityreference.md)
- **toEntity**
- To entity that is downstream of lineage edge.
- $ref: [entityReference.json](entityreference.md)
- **description**
- Type: `string`
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -6,6 +6,8 @@ This schema defines the EntityReference type used for referencing an entity. Ent
Type: `object`
This schema <u>does not</u> accept additional properties.
## Properties
- **id** `required`
- Unique identifier that identifies an entity instance.
@ -33,4 +35,4 @@ Type: `object`
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -12,8 +12,8 @@ Type: `object`
- $ref: [entityReference.json](entityreference.md)
- **usage** `required`
- List usage details per day.
- Type: `array`
- **Items**
- $ref: [usageDetails.json](usagedetails.md)
- Type: `array`
- **Items**
- $ref: [usageDetails.json](usagedetails.md)
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -30,9 +30,8 @@ Type: `object`
### connectionUrl
- Type used for JDBC connection URL.
- Type used for JDBC connection URL of format `url_scheme://<username>:<password>@<host>:<port>/<db_name>`.
- Type: `string`
- String format must be a "uri"
### jdbcInfo
@ -47,4 +46,4 @@ Type: `object`
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -17,4 +17,4 @@ Type: `object`
- Total number of entries available to page through.
- Type: `integer`
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -41,4 +41,4 @@ Type: `object`
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -14,4 +14,4 @@ Type: `object`
- Repeat frequency in ISO 8601 duration format. Example - 'P23DT23H'.
- $ref: [basic.json#/definitions/duration](basic.md#duration)
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -6,6 +6,8 @@ This schema defines the type for labeling an entity with a Tag.
Type: `object`
This schema <u>does not</u> accept additional properties.
## Properties
- **tagFQN**
- Type: `string`
@ -30,4 +32,4 @@ Type: `object`
- Link to the tag resource.
- $ref: [basic.json#/definitions/href](basic.md#href)
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_

View File

@ -39,4 +39,4 @@ Type: `object`
_This document was updated on: Thursday, September 16, 2021_
_This document was updated on: Tuesday, October 12, 2021_