OpenMetadata/docs/openmetadata-apis/schemas/entities/table.md

Table

Table

This schema defines the Table entity. A Table organizes data in rows and columns and is defined by a Schema. OpenMetadata does not have a separate abstraction for Schema. Both Table and Schema are captured in this entity.

$id:https://open-metadata.org/schema/entity/data/table.json

Type: object

This schema does not accept additional properties.

Properties

<<<<<<< HEAD

• id required
  • Unique identifier of this table instance.
  • $ref: ../../type/basic.json#/definitions/uuid
• name required
  • Name of a table. Expected to be unique within a database.
  • $ref: #/definitions/tableName
• displayName
  • Display Name that identifies this table. It could be title or label from the source services.
  • Type: string
• fullyQualifiedName
  • Fully qualified name of a table in the form serviceName.databaseName.tableName.
  • Type: string
• description
  • Description of a table.
  • Type: string
• version
  • Metadata version of the entity.
  • $ref: ../../type/entityHistory.json#/definitions/entityVersion
• updatedAt
  • Last update time corresponding to the new version of the entity in Unix epoch time milliseconds.
  • $ref: ../../type/basic.json#/definitions/timestamp
• updatedBy
  • User who made the update.
  • Type: string
• href
  • Link to this table resource.
  • $ref: ../../type/basic.json#/definitions/href
• tableType
  • $ref: #/definitions/tableType
• columns required
  • Columns in this table.
  • Type: array
    • Items
    • $ref: #/definitions/column
• tableConstraints
  • Table constraints.
  • Type: array
    • Items

TableConstraint

	 - $ref: [#/definitions/tableConstraint](#tableconstraint)

• owner
  • Owner of this table.
  • $ref: ../../type/entityReference.json
• database
  • Reference to Database that contains this table.
  • $ref: ../../type/entityReference.json
• service
  • Link to Database service this table is hosted in.
  • $ref: ../../type/entityReference.json
• serviceType
  • Service type this table is hosted in.
  • $ref: ../services/databaseService.json#/definitions/databaseServiceType
• location
  • Reference to the Location that contains this table.
  • $ref: ../../type/entityReference.json
• viewDefinition
  • View Definition in SQL. Applies to TableType.View only.
  • $ref: ../../type/basic.json#/definitions/sqlQuery
• tags
  • Tags for this table.
  • Type: array
    • Items
    • $ref: ../../type/tagLabel.json
• usageSummary
  • Latest usage information for this table.
  • $ref: ../../type/usageDetails.json
• followers
  • Followers of this table.
  • $ref: ../../type/entityReference.json#/definitions/entityReferenceList
• joins
  • Details of other tables this table is frequently joined with.
  • $ref: #/definitions/tableJoins
• sampleData
  • Sample data for a table.
  • $ref: #/definitions/tableData
• tableProfile
  • Data profile for a table.
  • Type: array
    • Items
    • $ref: #/definitions/tableProfile
• tableQueries
  • List of queries that ran against a table.
  • Type: array
    • Items
    • $ref: #/definitions/sqlQuery
• tableTests
  • List of test cases that ran against a table.
  • Type: array
    • Items
    • $ref: ../../tests/tableTest.json
• dataModel
  • This captures information about how the table is modeled. Currently only DBT model is supported.
  • $ref: #/definitions/dataModel
• changeDescription
  • Change that lead to this version of the entity.
  • $ref: ../../type/entityHistory.json#/definitions/changeDescription
• deleted
  • When true indicates the entity has been soft deleted.
  • Type: boolean
  • Default: false

a07bc411 (updated json schema and schema docs (#3219))

• id required
  • Unique identifier of this table instance.
  • $ref: ../../type/basic.json#/definitions/uuid
• name required
  • Name of a table. Expected to be unique within a database.
  • $ref: #/definitions/tableName
• displayName
  • Display Name that identifies this table. It could be title or label from the source services.
  • Type: string
• fullyQualifiedName
  • Fully qualified name of a table in the form serviceName.databaseName.tableName.
  • Type: string
• description
  • Description of a table.
  • Type: string
• version
  • Metadata version of the entity.
  • $ref: ../../type/entityHistory.json#/definitions/entityVersion
• updatedAt
  • Last update time corresponding to the new version of the entity in Unix epoch time milliseconds.
  • $ref: ../../type/basic.json#/definitions/timestamp
• updatedBy
  • User who made the update.
  • Type: string
• href
  • Link to this table resource.
  • $ref: ../../type/basic.json#/definitions/href
• tableType
  • $ref: #/definitions/tableType
• columns required
  • Columns in this table.
  • Type: array
    • Items
    • $ref: #/definitions/column
• tableConstraints
  • Table constraints.
  • Type: array
    • Items
    • $ref: #/definitions/tableConstraint
• owner
  • Owner of this table.
  • $ref: ../../type/entityReference.json
• database
  • Reference to Database that contains this table.
  • $ref: ../../type/entityReference.json
• service
  • Link to Database service this table is hosted in.
  • $ref: ../../type/entityReference.json
• serviceType
  • Service type this table is hosted in.
  • $ref: ../services/databaseService.json#/definitions/databaseServiceType
• location
  • Reference to the Location that contains this table.
  • $ref: ../../type/entityReference.json
• viewDefinition
  • View Definition in SQL. Applies to TableType.View only.
  • $ref: ../../type/basic.json#/definitions/sqlQuery
• tags
  • Tags for this table.
  • Type: array
    • Items
    • $ref: ../../type/tagLabel.json
• usageSummary
  • Latest usage information for this table.
  • $ref: ../../type/usageDetails.json
• followers
  • Followers of this table.
  • $ref: ../../type/entityReference.json#/definitions/entityReferenceList
• joins
  • Details of other tables this table is frequently joined with.
  • $ref: #/definitions/tableJoins
• sampleData
  • Sample data for a table.
  • $ref: #/definitions/tableData
• tableProfile
  • Data profile for a table.
  • Type: array
    • Items
    • $ref: #/definitions/tableProfile
• tableQueries
  • List of queries that ran against a table.
  • Type: array
    • Items
    • $ref: #/definitions/sqlQuery
• dataModel
  • This captures information about how the table is modeled. Currently only DBT model is supported.
  • $ref: #/definitions/dataModel
• changeDescription
  • Change that lead to this version of the entity.
  • $ref: ../../type/entityHistory.json#/definitions/changeDescription
• deleted
  • When true indicates the entity has been soft deleted.
  • Type: boolean
  • Default: false

Type definitions in this schema

tableType

• This schema defines the type used for describing different types of tables.
• Type: string
• The value is restricted to the following:
  1. "Regular"
  2. "External"
  3. "View"
  4. "SecureView"
  5. "MaterializedView"

dataType

<<<<<<< HEAD

• This enum defines the type of data stored in a column.
• Type: string
• The value is restricted to the following:
  1. "NUMBER"
  2. "TINYINT"
  3. "SMALLINT"
  4. "INT"
  5. "BIGINT"
  6. "BYTEINT"
  7. "BYTES"
  8. "FLOAT"
  9. "DOUBLE"
  10. "DECIMAL"
  11. "NUMERIC"
  12. "TIMESTAMP"
  13. "TIME"
  14. "DATE"
  15. "DATETIME"
  16. "INTERVAL"
  17. "STRING"
  18. "MEDIUMTEXT"
  19. "TEXT"
  20. "CHAR"
  21. "VARCHAR"
  22. "BOOLEAN"
  23. "BINARY"
  24. "VARBINARY"
  25. "ARRAY"
  26. "BLOB"
  27. "LONGBLOB"
  28. "MEDIUMBLOB"
  29. "MAP"
  30. "STRUCT"
  31. "UNION"
  32. "SET"
  33. "GEOGRAPHY"
  34. "ENUM"
  35. "JSON" =======
• This enum defines the type of data stored in a column.
• Type: string
• The value is restricted to the following:
  1. "NUMBER"
  2. "TINYINT"
  3. "SMALLINT"
  4. "INT"
  5. "BIGINT"
  6. "BYTEINT"
  7. "BYTES"
  8. "FLOAT"
  9. "DOUBLE"
  10. "DECIMAL"
  11. "NUMERIC"
  12. "TIMESTAMP"
  13. "TIME"
  14. "DATE"
  15. "DATETIME"
  16. "INTERVAL"
  17. "STRING"
  18. "MEDIUMTEXT"
  19. "TEXT"
  20. "CHAR"
  21. "VARCHAR"
  22. "BOOLEAN"
  23. "BINARY"
  24. "VARBINARY"
  25. "ARRAY"
  26. "BLOB"
  27. "LONGBLOB"
  28. "MEDIUMBLOB"
  29. "MAP"
  30. "STRUCT"
  31. "UNION"
  32. "SET"
  33. "GEOGRAPHY"
  34. "ENUM"
  35. "JSON"
  36. "UUID"

a07bc411 (updated json schema and schema docs (#3219))

• Type: string
• The value is restricted to the following:
  1. "Regular"
  2. "External"
  3. "View"
  4. "SecureView"
  5. "MaterializedView"
  6. "Iceberg"

constraint

• This enum defines the type for column constraint.
• Type: string
• The value is restricted to the following:
  1. "NULL"
  2. "NOT_NULL"
  3. "UNIQUE"
  4. "PRIMARY_KEY"

tableConstraint

<<<<<<< HEAD

• This enum defines the type for table constraint.
• Type: object
• Properties
  • constraintType
    • Type: string
    • The value is restricted to the following:
      1. "UNIQUE"
      2. "PRIMARY_KEY"
      3. "FOREIGN_KEY"
  • columns
    • List of column names corresponding to the constraint.
    • Type: array
      • Items
      • Type: string

columnName

• Local name (not fully qualified name) of the column. ColumnName is - when the column is not named in struct dataType. For example, BigQuery supports struct with unnamed fields.
• Type: string
• The value must match this pattern: ^[^.]*$
• Length: between 1 and 128

tableName

• Local name (not fully qualified name) of a table.
• Type: string
• The value must match this pattern: ^[^.]*$
• Length: between 1 and 128 =======

TableConstraint

• This enum defines the type for table constraint.
• Type: object
• This schema does not accept additional properties.
• Properties
  • constraintType
    • Type: string
    • The value is restricted to the following:
      1. "UNIQUE"
      2. "PRIMARY_KEY"
      3. "FOREIGN_KEY"
  • columns
    • List of column names corresponding to the constraint.
    • Type: array
      • Items
      • Type: string

columnName

• Local name (not fully qualified name) of the column. ColumnName is - when the column is not named in struct dataType. For example, BigQuery supports struct with unnamed fields.
• Type: string
• Length: between 1 and 128

tableName

• Local name (not fully qualified name) of a table. Dots will be escaped automatically.
• Type: string
• Length: between 1 and 128

a07bc411 (updated json schema and schema docs (#3219))

fullyQualifiedColumnName

• Fully qualified name of the column that includes serviceName.databaseName.tableName.columnName[.nestedColumnName]. When columnName is null for dataType struct fields, field_# where # is field index is used. For map dataType, for key the field name key is used and for the value field value is used.
• Type: string
• Length: between 1 and 256

column

<<<<<<< HEAD

• This schema defines the type for a column in a table.
• Type: object
• This schema does not accept additional properties.
• Properties
  • name required
    • $ref: #/definitions/columnName
  • dataType required
    • Data type of the column (int, date etc.).
    • $ref: #/definitions/dataType
  • arrayDataType
    • Data type used array in dataType. For example, array<int> has dataType as array and arrayDataType as int.
    • $ref: #/definitions/dataType
  • dataLength
    • Length of char, varchar, binary, varbinary dataTypes, else null. For example, varchar(20) has dataType as varchar and dataLength as 20.
    • Type: integer
  • dataTypeDisplay
    • Display name used for dataType. This is useful for complex types, such as `array, map<int,string>, struct<>, and union types.
    • Type: string
  • description
    • Description of the column.
    • Type: string
  • fullyQualifiedName
    • $ref: #/definitions/fullyQualifiedColumnName
  • tags
    • Tags associated with the column.
    • Type: array
      • Items
      • $ref: ../../type/tagLabel.json
  • constraint
    • Column level constraint.
    • $ref: #/definitions/constraint
  • ordinalPosition
    • Ordinal position of the column.
    • Type: integer
  • jsonSchema
    • Json schema only if the dataType is JSON else null.
    • Type: string
  • children
    • Child columns if dataType or arrayDataType is map, struct, or union else null.
    • Type: array
      • Items
      • $ref: #/definitions/column

columnJoins

• This schema defines the type to capture how frequently a column is joined with columns in the other tables.
• Type: object
• This schema does not accept additional properties.
• Properties
  • columnName
    • $ref: #/definitions/columnName
  • joinedWith
    • Fully qualified names of the columns that this column is joined with.
    • Type: array
      • Items
      • Type: object
      • Properties
        • fullyQualifiedName
          • $ref: #/definitions/fullyQualifiedColumnName
        • joinCount
          • Type: integer =======
• This schema defines the type for a column in a table.
• Type: object
• This schema does not accept additional properties.
• Properties
  • name required
    • $ref: #/definitions/columnName
  • displayName
    • Display Name that identifies this column name.
    • Type: string
  • dataType required
    • Data type of the column (int, date etc.).
    • $ref: #/definitions/dataType
  • arrayDataType
    • Data type used array in dataType. For example, array<int> has dataType as array and arrayDataType as int.
    • $ref: #/definitions/dataType
  • dataLength
    • Length of char, varchar, binary, varbinary dataTypes, else null. For example, varchar(20) has dataType as varchar and dataLength as 20.
    • Type: integer
  • dataTypeDisplay
    • Display name used for dataType. This is useful for complex types, such as `array, map<int,string>, struct<>, and union types.
    • Type: string
  • description
    • Description of the column.
    • Type: string
  • fullyQualifiedName
    • $ref: #/definitions/fullyQualifiedColumnName
  • tags
    • Tags associated with the column.
    • Type: array
      • Items
      • $ref: ../../type/tagLabel.json
  • constraint
    • Column level constraint.
    • $ref: #/definitions/constraint
  • ordinalPosition
    • Ordinal position of the column.
    • Type: integer
  • jsonSchema
    • Json schema only if the dataType is JSON else null.
    • Type: string
  • children
    • Child columns if dataType or arrayDataType is map, struct, or union else null.
    • Type: array
      • Items
      • $ref: #/definitions/column
  • columnTests
    • List of column test cases that ran against a table column.
    • Type: array
      • Items
      • $ref: ../../tests/columnTest.json

columnJoins

• This schema defines the type to capture how frequently a column is joined with columns in the other tables.
• Type: object
• This schema does not accept additional properties.
• Properties
  • columnName
    • $ref: #/definitions/columnName
  • joinedWith
    • Fully qualified names of the columns that this column is joined with.
    • Type: array
      • Items
      • Type: object
      • This schema does not accept additional properties.
      • Properties
        • fullyQualifiedName
          • $ref: #/definitions/fullyQualifiedColumnName
        • joinCount
          • Type: integer

a07bc411 (updated json schema and schema docs (#3219))

tableJoins

• This schema defines the type to capture information about how columns in this table are joined with columns in the other tables.
• Type: object
• This schema does not accept additional properties.
• Properties
  • startDate
    • Date can be only from today going back to last 29 days.
    • $ref: ../../type/basic.json#/definitions/date
  • dayCount
    • Type: integer
    • Default: 1
  • columnJoins
    • Type: array
      • Items
      • $ref: #/definitions/columnJoins

tableData

• This schema defines the type to capture rows of sample data for a table.
• Type: object
• This schema does not accept additional properties.
• Properties
  • columns
    • List of local column names (not fully qualified column names) of the table.
    • Type: array
      • Items
      • $ref: #/definitions/columnName
  • rows
    • Data for multiple rows of the table.
    • Type: array
      • Items
      • Data for a single row of the table within the same order as columns fields.
      • Type: array

columnProfile

<<<<<<< HEAD

• This schema defines the type to capture the table's column profile.
• Type: object
• This schema does not accept additional properties.
• Properties
  • name
    • Column Name.
    • Type: string
  • valuesCount
    • Total count of the values in this column.
    • Type: number
  • valuesPercentage
    • Percentage of values in this column with respect to rowcount.
    • Type: number
  • validCount
    • Total count of valid values in this column.
    • Type: number
  • duplicateCount
    • No.of Rows that contain duplicates in a column.
    • Type: number
  • nullCount
    • No.of null values in a column.
    • Type: number
  • nullProportion
    • No.of null value proportion in columns.
    • Type: number
  • missingPercentage
    • Missing Percentage is calculated by taking percentage of validCount/valuesCount.
    • Type: number
  • missingCount
    • Missing count is calculated by subtracting valuesCount - validCount.
    • Type: number
  • uniqueCount
    • No. of unique values in the column.
    • Type: number
  • uniqueProportion
    • Proportion of number of unique values in a column.
    • Type: number
  • distinctCount
    • Number of values that contain distinct values.
    • Type: number
  • min
    • Minimum value in a column.
    • Type: number
  • max
    • Maximum value in a column.
    • Type: number
  • mean
    • Avg value in a column.
    • Type: number
  • sum
    • Median value in a column.
    • Type: number
  • stddev
    • Standard deviation of a column.
    • Type: number
  • variance
    • Variance of a column.
    • Type: number
  • histogram
    • Histogram of a column. =======
• This schema defines the type to capture the table's column profile.
• Type: object
• This schema does not accept additional properties.
• Properties
  • name
    • Column Name.
    • Type: string
  • valuesCount
    • Total count of the values in this column.
    • Type: number
  • valuesPercentage
    • Percentage of values in this column with respect to rowcount.
    • Type: number
  • validCount
    • Total count of valid values in this column.
    • Type: number
  • duplicateCount
    • No.of Rows that contain duplicates in a column.
    • Type: number
  • nullCount
    • No.of null values in a column.
    • Type: number
  • nullProportion
    • No.of null value proportion in columns.
    • Type: number
  • missingPercentage
    • Missing Percentage is calculated by taking percentage of validCount/valuesCount.
    • Type: number
  • missingCount
    • Missing count is calculated by subtracting valuesCount - validCount.
    • Type: number
  • uniqueCount
    • No. of unique values in the column.
    • Type: number
  • uniqueProportion
    • Proportion of number of unique values in a column.
    • Type: number
  • distinctCount
    • Number of values that contain distinct values.
    • Type: number
  • min
    • Minimum value in a column.
    • Types: number, integer, string
  • max
    • Maximum value in a column.
    • Types: number, integer, string
  • minLength
    • Minimum string length in a column.
    • Type: number
  • maxLength
    • Maximum string length in a column.
    • Type: number
  • mean
    • Avg value in a column.
    • Type: number
  • sum
    • Median value in a column.
    • Type: number
  • stddev
    • Standard deviation of a column.
    • Type: number
  • variance
    • Variance of a column.
    • Type: number
  • histogram
    • Histogram of a column.

a07bc411 (updated json schema and schema docs (#3219))

tableProfile

• This schema defines the type to capture the table's data profile.
• Type: object
• This schema does not accept additional properties.
• Properties
  • profileDate
    • Data one which profile is taken.
    • $ref: ../../type/basic.json#/definitions/date
  • columnCount
    • No.of columns in the table.
    • Type: number
  • rowCount
    • No.of rows in the table.
    • Type: number
  • columnProfile
    • List of local column profiles of the table.
    • Type: array
      • Items
      • $ref: #/definitions/columnProfile

sqlQuery

<<<<<<< HEAD

• This schema defines the type to capture the table's sql queries.
• Type: object
• Properties
  • query
    • SQL Query text that matches the table name.
    • Type: string
  • duration
    • How long did the query took to run in seconds.
    • Type: number
  • user
    • User who ran this query.
    • $ref: ../../type/entityReference.json
  • vote
    • Users can vote up to rank the popular queries.
    • Type: number
    • Default: 1
  • checksum
    • Checksum to avoid registering duplicate queries.
    • Type: string
  • queryDate
    • Date on which the query ran.
    • $ref: ../../type/basic.json#/definitions/date =======
• This schema defines the type to capture the table's sql queries.
• Type: object
• This schema does not accept additional properties.
• Properties
  • query
    • SQL Query text that matches the table name.
    • Type: string
  • duration
    • How long did the query took to run in seconds.
    • Type: number
  • user
    • User who ran this query.
    • $ref: ../../type/entityReference.json
  • vote
    • Users can vote up to rank the popular queries.
    • Type: number
    • Default: 1
  • checksum
    • Checksum to avoid registering duplicate queries.
    • Type: string
  • queryDate
    • Date on which the query ran.
    • $ref: ../../type/basic.json#/definitions/date

a07bc411 (updated json schema and schema docs (#3219))

modelType

• The value is restricted to the following:
  1. "DBT"

dataModel

<<<<<<< HEAD

• This captures information about how the table is modeled. Currently only DBT model is supported.
• Type: object
• Properties
  • modelType required
    • $ref: #/definitions/modelType
  • description
    • Description of the Table from the model.
    • Type: string
  • path
    • Path to sql definition file.
    • Type: string
  • rawSql
    • This corresponds to rws SQL from <model_name>.sql in DBT. This might be null when SQL query need not be compiled as done in DBT.
    • $ref: ../../type/basic.json#/definitions/sqlQuery
  • sql required
    • This corresponds to compile SQL from <model_name>.sql in DBT. In cases where compilation is not necessary, this corresponds to SQL that created the table.
    • $ref: ../../type/basic.json#/definitions/sqlQuery
  • upstream
    • Fully qualified name of Models/tables used for in sql for creating this table.
    • Type: array
      • Items
      • Type: string
  • columns
    • Columns from the schema defined during modeling. In case of DBT, the metadata here comes from schema.yaml.
    • Type: array
      • Items
      • $ref: #/definitions/column
  • generatedAt
    • $ref: ../../type/basic.json#/definitions/dateTime

This document was updated on: Tuesday, January 25, 2022

• This captures information about how the table is modeled. Currently only DBT model is supported.
• Type: object
• This schema does not accept additional properties.
• Properties
  • modelType required
    • $ref: #/definitions/modelType
  • description
    • Description of the Table from the model.
    • Type: string
  • path
    • Path to sql definition file.
    • Type: string
  • rawSql
    • This corresponds to rws SQL from <model_name>.sql in DBT. This might be null when SQL query need not be compiled as done in DBT.
    • $ref: ../../type/basic.json#/definitions/sqlQuery
  • sql required
    • This corresponds to compile SQL from <model_name>.sql in DBT. In cases where compilation is not necessary, this corresponds to SQL that created the table.
    • $ref: ../../type/basic.json#/definitions/sqlQuery
  • upstream
    • Fully qualified name of Models/tables used for in sql for creating this table.
    • Type: array
      • Items
      • Type: string
  • columns
    • Columns from the schema defined during modeling. In case of DBT, the metadata here comes from schema.yaml.
    • Type: array
      • Items
      • $ref: #/definitions/column
  • generatedAt
    • $ref: ../../type/basic.json#/definitions/dateTime

This document was updated on: Monday, March 7, 2022

This document was updated on: Wednesday, March 9, 2022