mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-10-31 18:48:35 +00:00

* updated schema docs

* addressing ingestion-core version bump

2022-03-10 12:55:35 +05:30

15 KiB

Raw Blame History

Table

This schema defines the Table entity. A Table organizes data in rows and columns and is defined by a Schema. OpenMetadata does not have a separate abstraction for Schema. Both Table and Schema are captured in this entity.

$id:https://open-metadata.org/schema/entity/data/table.json

Type: object

This schema does not accept additional properties.

Properties

id required
- Unique identifier of this table instance.
- $ref: ../../type/basic.json#/definitions/uuid
name required
- Name of a table. Expected to be unique within a database.
- $ref: #/definitions/tableName
displayName
- Display Name that identifies this table. It could be title or label from the source services.
- Type: string
fullyQualifiedName
- Fully qualified name of a table in the form serviceName.databaseName.tableName.
- Type: string
description
- Description of a table.
- Type: string
version
- Metadata version of the entity.
- $ref: ../../type/entityHistory.json#/definitions/entityVersion
updatedAt
- Last update time corresponding to the new version of the entity in Unix epoch time milliseconds.
- $ref: ../../type/basic.json#/definitions/timestamp
updatedBy
- User who made the update.
- Type: string
href
- Link to this table resource.
- $ref: ../../type/basic.json#/definitions/href
tableType
- $ref: #/definitions/tableType
columns required
- Columns in this table.
- Type: array
  - Items
  - $ref: #/definitions/column
tableConstraints
- Table constraints.
- Type: array
  - Items

TableConstraint

	 - $ref: [#/definitions/tableConstraint](#tableconstraint)

owner
- Owner of this table.
- $ref: ../../type/entityReference.json
database
- Reference to Database that contains this table.
- $ref: ../../type/entityReference.json
service
- Link to Database service this table is hosted in.
- $ref: ../../type/entityReference.json
serviceType
- Service type this table is hosted in.
- $ref: ../services/databaseService.json#/definitions/databaseServiceType
location
- Reference to the Location that contains this table.
- $ref: ../../type/entityReference.json
viewDefinition
- View Definition in SQL. Applies to TableType.View only.
- $ref: ../../type/basic.json#/definitions/sqlQuery
tags
- Tags for this table.
- Type: array
  - Items
  - $ref: ../../type/tagLabel.json
usageSummary
- Latest usage information for this table.
- $ref: ../../type/usageDetails.json
followers
- Followers of this table.
- $ref: ../../type/entityReference.json#/definitions/entityReferenceList
joins
- Details of other tables this table is frequently joined with.
- $ref: #/definitions/tableJoins
sampleData
- Sample data for a table.
- $ref: #/definitions/tableData
tableProfile
- Data profile for a table.
- Type: array
  - Items
  - $ref: #/definitions/tableProfile
tableQueries
- List of queries that ran against a table.
- Type: array
  - Items
  - $ref: #/definitions/sqlQuery
tableTests
- List of test cases that ran against a table.
- Type: array
  - Items
  - $ref: ../../tests/tableTest.json
dataModel
- This captures information about how the table is modeled. Currently only DBT model is supported.
- $ref: #/definitions/dataModel
changeDescription
- Change that lead to this version of the entity.
- $ref: ../../type/entityHistory.json#/definitions/changeDescription
deleted
- When true indicates the entity has been soft deleted.
- Type: boolean
- Default: false

Type definitions in this schema

tableType

This schema defines the type used for describing different types of tables.
Type: string
The value is restricted to the following:
1. "Regular"
2. "External"
3. "View"
4. "SecureView"
5. "MaterializedView"
6. "Iceberg"

dataType

This enum defines the type of data stored in a column.
Type: string
The value is restricted to the following:
1. "NUMBER"
2. "TINYINT"
3. "SMALLINT"
4. "INT"
5. "BIGINT"
6. "BYTEINT"
7. "BYTES"
8. "FLOAT"
9. "DOUBLE"
10. "DECIMAL"
11. "NUMERIC"
12. "TIMESTAMP"
13. "TIME"
14. "DATE"
15. "DATETIME"
16. "INTERVAL"
17. "STRING"
18. "MEDIUMTEXT"
19. "TEXT"
20. "CHAR"
21. "VARCHAR"
22. "BOOLEAN"
23. "BINARY"
24. "VARBINARY"
25. "ARRAY"
26. "BLOB"
27. "LONGBLOB"
28. "MEDIUMBLOB"
29. "MAP"
30. "STRUCT"
31. "UNION"
32. "SET"
33. "GEOGRAPHY"
34. "ENUM"
35. "JSON"
36. "UUID"

constraint

This enum defines the type for column constraint.
Type: string
The value is restricted to the following:
1. "NULL"
2. "NOT_NULL"
3. "UNIQUE"
4. "PRIMARY_KEY"

tableConstraint

TableConstraint

This enum defines the type for table constraint.
Type: object
This schema does not accept additional properties.
Properties
- constraintType
  - Type: string
  - The value is restricted to the following:
    1. "UNIQUE"
    2. "PRIMARY_KEY"
    3. "FOREIGN_KEY"
- columns
  - List of column names corresponding to the constraint.
  - Type: array
    - Items
    - Type: string

columnName

Local name (not fully qualified name) of the column. ColumnName is - when the column is not named in struct dataType. For example, BigQuery supports struct with unnamed fields.
Type: string
Length: between 1 and 128

tableName

Local name (not fully qualified name) of a table. Dots will be escaped automatically.
Type: string
Length: between 1 and 128

fullyQualifiedColumnName

Fully qualified name of the column that includes serviceName.databaseName.tableName.columnName[.nestedColumnName]. When columnName is null for dataType struct fields, field_# where # is field index is used. For map dataType, for key the field name key is used and for the value field value is used.
Type: string
Length: between 1 and 256

column

This schema defines the type for a column in a table.
Type: object
This schema does not accept additional properties.
Properties
- name required
  - $ref: #/definitions/columnName
- displayName
  - Display Name that identifies this column name.
  - Type: string
- dataType required
  - Data type of the column (int, date etc.).
  - $ref: #/definitions/dataType
- arrayDataType
  - Data type used array in dataType. For example, array<int> has dataType as array and arrayDataType as int.
  - $ref: #/definitions/dataType
- dataLength
  - Length of char, varchar, binary, varbinary dataTypes, else null. For example, varchar(20) has dataType as varchar and dataLength as 20.
  - Type: integer
- dataTypeDisplay
  - Display name used for dataType. This is useful for complex types, such as `array, map<int,string>, struct<>, and union types.
  - Type: string
- description
  - Description of the column.
  - Type: string
- fullyQualifiedName
  - $ref: #/definitions/fullyQualifiedColumnName
- tags
  - Tags associated with the column.
  - Type: array
    - Items
    - $ref: ../../type/tagLabel.json
- constraint
  - Column level constraint.
  - $ref: #/definitions/constraint
- ordinalPosition
  - Ordinal position of the column.
  - Type: integer
- jsonSchema
  - Json schema only if the dataType is JSON else null.
  - Type: string
- children
  - Child columns if dataType or arrayDataType is map, struct, or union else null.
  - Type: array
    - Items
    - $ref: #/definitions/column
- columnTests
  - List of column test cases that ran against a table column.
  - Type: array
    - Items
    - $ref: ../../tests/columnTest.json

columnJoins

This schema defines the type to capture how frequently a column is joined with columns in the other tables.
Type: object
This schema does not accept additional properties.
Properties
- columnName
  - $ref: #/definitions/columnName
- joinedWith
  - Fully qualified names of the columns that this column is joined with.
  - Type: array
    - Items
    - Type: object
    - This schema does not accept additional properties.
    - Properties
      - fullyQualifiedName
        
        $ref: #/definitions/fullyQualifiedColumnName
      - joinCount
        
        Type: integer

tableJoins

This schema defines the type to capture information about how columns in this table are joined with columns in the other tables.
Type: object
This schema does not accept additional properties.
Properties
- startDate
  - Date can be only from today going back to last 29 days.
  - $ref: ../../type/basic.json#/definitions/date
- dayCount
  - Type: integer
  - Default: 1
- columnJoins
  - Type: array
    - Items
    - $ref: #/definitions/columnJoins

tableData

This schema defines the type to capture rows of sample data for a table.
Type: object
This schema does not accept additional properties.
Properties
- columns
  - List of local column names (not fully qualified column names) of the table.
  - Type: array
    - Items
    - $ref: #/definitions/columnName
- rows
  - Data for multiple rows of the table.
  - Type: array
    - Items
    - Data for a single row of the table within the same order as columns fields.
    - Type: array

columnProfile

This schema defines the type to capture the table's column profile.
Type: object
This schema does not accept additional properties.
Properties
- name
  - Column Name.
  - Type: string
- valuesCount
  - Total count of the values in this column.
  - Type: number
- valuesPercentage
  - Percentage of values in this column with respect to rowcount.
  - Type: number
- validCount
  - Total count of valid values in this column.
  - Type: number
- duplicateCount
  - No.of Rows that contain duplicates in a column.
  - Type: number
- nullCount
  - No.of null values in a column.
  - Type: number
- nullProportion
  - No.of null value proportion in columns.
  - Type: number
- missingPercentage
  - Missing Percentage is calculated by taking percentage of validCount/valuesCount.
  - Type: number
- missingCount
  - Missing count is calculated by subtracting valuesCount - validCount.
  - Type: number
- uniqueCount
  - No. of unique values in the column.
  - Type: number
- uniqueProportion
  - Proportion of number of unique values in a column.
  - Type: number
- distinctCount
  - Number of values that contain distinct values.
  - Type: number
- distinctProportion
  - Proportion of distinct values in a column.
  - Type: number
- min
  - Minimum value in a column.
  - Types: number, integer, string
- max
  - Maximum value in a column.
  - Types: number, integer, string
- minLength
  - Minimum string length in a column.
  - Type: number
- maxLength
  - Maximum string length in a column.
  - Type: number
- mean
  - Avg value in a column.
  - Type: number
- sum
  - Median value in a column.
  - Type: number
- stddev
  - Standard deviation of a column.
  - Type: number
- variance
  - Variance of a column.
  - Type: number
- histogram
  - Histogram of a column.

tableProfile

This schema defines the type to capture the table's data profile.
Type: object
This schema does not accept additional properties.
Properties
- profileDate
  - Data one which profile is taken.
  - $ref: ../../type/basic.json#/definitions/date
- columnCount
  - No.of columns in the table.
  - Type: number
- rowCount
  - No.of rows in the table.
  - Type: number
- columnProfile
  - List of local column profiles of the table.
  - Type: array
    - Items
    - $ref: #/definitions/columnProfile

sqlQuery

This schema defines the type to capture the table's sql queries.
Type: object
This schema does not accept additional properties.
Properties
- query
  - SQL Query text that matches the table name.
  - Type: string
- duration
  - How long did the query took to run in seconds.
  - Type: number
- user
  - User who ran this query.
  - $ref: ../../type/entityReference.json
- vote
  - Users can vote up to rank the popular queries.
  - Type: number
  - Default: 1
- checksum
  - Checksum to avoid registering duplicate queries.
  - Type: string
- queryDate
  - Date on which the query ran.
  - $ref: ../../type/basic.json#/definitions/date

modelType

The value is restricted to the following:
1. "DBT"

dataModel

This captures information about how the table is modeled. Currently only DBT model is supported.
Type: object
This schema does not accept additional properties.
Properties
- modelType required
  - $ref: #/definitions/modelType
- description
  - Description of the Table from the model.
  - Type: string
- path
  - Path to sql definition file.
  - Type: string
- rawSql
  - This corresponds to rws SQL from <model_name>.sql in DBT. This might be null when SQL query need not be compiled as done in DBT.
  - $ref: ../../type/basic.json#/definitions/sqlQuery
- sql required
  - This corresponds to compile SQL from <model_name>.sql in DBT. In cases where compilation is not necessary, this corresponds to SQL that created the table.
  - $ref: ../../type/basic.json#/definitions/sqlQuery
- upstream
  - Fully qualified name of Models/tables used for in sql for creating this table.
  - Type: array
    - Items
    - Type: string
- columns
  - Columns from the schema defined during modeling. In case of DBT, the metadata here comes from schema.yaml.
  - Type: array
    - Items
    - $ref: #/definitions/column
- generatedAt
  - $ref: ../../type/basic.json#/definitions/dateTime

This document was updated on: Wednesday, March 9, 2022

15 KiB Raw Blame History

Table

Properties

TableConstraint

Type definitions in this schema

tableType

dataType

constraint

tableConstraint

TableConstraint

columnName

tableName

fullyQualifiedColumnName

column

columnJoins

tableJoins

tableData

columnProfile

tableProfile

sqlQuery

modelType

dataModel

15 KiB

Raw Blame History