diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 79cce8425f3..e29fded754a 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -10,8 +10,8 @@ * [JSON Schema](openmetadata-apis/schemas/schema-language.md) * [Schema Concepts](openmetadata-apis/schemas/overview.md) * [OpenMetadata Types](openmetadata-apis/schemas/types/README.md) - * [Basic Types](openmetadata-apis/schemas/types/basic.md) * [Audit Log](openmetadata-apis/schemas/types/auditlog.md) + * [Basic Types](openmetadata-apis/schemas/types/basic.md) * [Daily Count](openmetadata-apis/schemas/types/dailycount.md) * [Entity Lineage](openmetadata-apis/schemas/types/entitylineage.md) * [Entity Reference](openmetadata-apis/schemas/types/entityreference.md) @@ -86,6 +86,7 @@ * [Oracle](install/metadata-ingestion/connectors/database-services/oracle.md) * [Postgres](install/metadata-ingestion/connectors/database-services/postgres.md) * [Presto](install/metadata-ingestion/connectors/database-services/presto.md) + * [Redshift](install/metadata-ingestion/connectors/database-services/redshift.md) * [Redshift Usage](install/metadata-ingestion/connectors/database-services/redshift-usage.md) * [Salesforce](install/metadata-ingestion/connectors/database-services/salesforce.md) * [Snowflake](install/metadata-ingestion/connectors/database-services/snowflake.md) diff --git a/docs/install/metadata-ingestion/connectors/database-services/README.md b/docs/install/metadata-ingestion/connectors/database-services/README.md index 2fa87229c45..3ca51d5dfbe 100644 --- a/docs/install/metadata-ingestion/connectors/database-services/README.md +++ b/docs/install/metadata-ingestion/connectors/database-services/README.md @@ -34,6 +34,10 @@ OpenMetadata supports connectors to some popular data sources. We will continue [postgres.md](postgres.md) {% endcontent-ref %} +{% content-ref url="presto.md" %} +[presto.md](presto.md) +{% endcontent-ref %} + {% content-ref url="redshift.md" %} [redshift.md](redshift.md) {% endcontent-ref %} @@ -59,8 +63,12 @@ OpenMetadata supports connectors to some popular data sources. We will continue {% endcontent-ref %} {% page-ref page="trino.md" %} +[trino.md](trino.md) +{% endcontent-ref %} {% page-ref page="vertica.md" %} +[vertica.md](vertica.md) +{% endcontent-ref %} diff --git a/docs/install/metadata-ingestion/connectors/database-services/redshift.md b/docs/install/metadata-ingestion/connectors/database-services/redshift.md new file mode 100644 index 00000000000..8e640216f95 --- /dev/null +++ b/docs/install/metadata-ingestion/connectors/database-services/redshift.md @@ -0,0 +1,92 @@ +--- +description: This guide will help install Redshift connector and run manually +--- + +# Redshift + +{% hint style="info" %} +**Prerequisites** + +OpenMetadata is built using Java, DropWizard, Jetty, and MySQL. + +1. Python 3.7 or above +2. OpenMetadata Server up and running + +{% endhint %} + +## Install from PyPI + +```bash +pip install 'openmetadata-ingestion[redshift]' +``` + +## Run Manually + +```bash +metadata ingest -c ./examples/workflows/redshift.json +``` + +## Configuration + +{% code title="redshift.json" %} +```javascript +{ + "source": { + "type": "redshift", + "config": { + "host_port": "cluster.name.region.redshift.amazonaws.com:5439", + "username": "username", + "password": "strong_password", + "database": "warehouse", + "service_name": "aws_redshift", + "filter_pattern": { + "excludes": ["information_schema.*", "[\\w]*event_vw.*"] + } + } + }, +... +``` +{% endcode %} + +1. **username** - pass the Redshift username. +2. **password** - the password for the Redshift username. +3. **service_name** - Service Name for this Redshift cluster. If you added the Redshift cluster through OpenMetadata UI, make sure the service name matches the same. +4. **filter_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata. +5. **database -** Database name from where data is to be fetched.‌ + +## Publish to OpenMetadata + +Below is the configuration to publish Redshift data into the OpenMetadata service.‌ + +Add Optionally `pii` processor and `metadata-rest` sink along with `metadata-server` config + +{% code title="redshift.json" %} +```javascript +{ + "source": { + "type": "redshift", + "config": { + "host_port": "cluster.name.region.redshift.amazonaws.com:5439", + "username": "username", + "password": "strong_password", + "database": "warehouse", + "service_name": "aws_redshift", + "filter_pattern": { + "excludes": ["information_schema.*", "[\\w]*event_vw.*"] + } + } + }, + "sink": { + "type": "metadata-rest", + "config": {} + }, + "metadata_server": { + "type": "metadata-server", + "config": { + "api_endpoint": "http://localhost:8585/api", + "auth_provider_type": "no-auth" + } + } +} +``` +{% endcode %} diff --git a/docs/open-source-community/developer/README.md b/docs/open-source-community/developer/README.md index 4a1d02099cd..f524939570e 100644 --- a/docs/open-source-community/developer/README.md +++ b/docs/open-source-community/developer/README.md @@ -6,6 +6,8 @@ This document summarizes information relevant to OpenMetadata committers and con {% page-ref page="coding-style.md" %} +{% page-ref page="how-to-contribute.md" %} + {% page-ref page="build-code-run-tests.md" %} {% page-ref page="run-integration-tests.md" %} diff --git a/docs/openmetadata-apis/schemas/entities/README.md b/docs/openmetadata-apis/schemas/entities/README.md index 598e6727d13..904a5672379 100644 --- a/docs/openmetadata-apis/schemas/entities/README.md +++ b/docs/openmetadata-apis/schemas/entities/README.md @@ -40,18 +40,22 @@ OpenMetadata has the following high-level core entities and relationships: [messagingservice.md](messagingservice.md) {% endcontent-ref %} -{% content-ref url="thread.md" %} -[thread.md](thread.md) -{% endcontent-ref %} - {% content-ref url="metrics.md" %} [metrics.md](metrics.md) {% endcontent-ref %} +{% content-ref url="model.md" %} +[model.md](model.md) +{% endcontent-ref %} + {% content-ref url="pipeline.md" %} [pipeline.md](pipeline.md) {% endcontent-ref %} +{% content-ref url="pipelineservice.md" %} +[pipelineservice.md](pipelineservice.md) +{% endcontent-ref %} + {% content-ref url="report.md" %} [report.md](report.md) {% endcontent-ref %} @@ -60,10 +64,22 @@ OpenMetadata has the following high-level core entities and relationships: [table.md](table.md) {% endcontent-ref %} +{% content-ref url="tagcategory.md" %} +[tagcategory.md](tagcategory.md) +{% endcontent-ref %} + +{% content-ref url="task.md" %} +[task.md](task.md) +{% endcontent-ref %} + {% content-ref url="team.md" %} [team.md](team.md) {% endcontent-ref %} +{% content-ref url="thread.md" %} +[thread.md](thread.md) +{% endcontent-ref %} + {% content-ref url="topic.md" %} [topic.md](topic.md) {% endcontent-ref %} @@ -71,7 +87,3 @@ OpenMetadata has the following high-level core entities and relationships: {% content-ref url="user.md" %} [user.md](user.md) {% endcontent-ref %} - -{% content-ref url="tagcategory.md" %} -[tagcategory.md](tagcategory.md) -{% endcontent-ref %} diff --git a/docs/openmetadata-apis/schemas/entities/bot.md b/docs/openmetadata-apis/schemas/entities/bot.md index 4b15de021b3..482cf9a10a0 100644 --- a/docs/openmetadata-apis/schemas/entities/bot.md +++ b/docs/openmetadata-apis/schemas/entities/bot.md @@ -2,7 +2,7 @@ This schema defines Bot entity. A bot automates tasks, such as adding description, identifying the importance of data. It runs as a special user in the system. -**$id: **[https://open-metadata.org/schema/entity/bots.json](https://open-metadata.org/schema/entity/bots.json) +**$id:[https://open-metadata.org/schema/entity/bots.json](https://open-metadata.org/schema/entity/bots.json)** Type: `object` @@ -20,25 +20,18 @@ Type: `object` - **description** - Description of the bot. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **href** - Link to the resource corresponding to this bot. - $ref: [../type/basic.json#/definitions/href](../types/basic.md#href) -* **id** - * Unique identifier of a bot instance. - * $ref: [../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** - * Name of the bot. - * Type: `string` - * Length: between 1 and 64 -* **displayName** - * Name used for display purposes. Example 'FirstName LastName'. - * Type: `string` -* **description** - * Description of the bot. - * Type: `string` -* **href** - * Link to the resource corresponding to this bot. - * $ref: [../type/basic.json#/definitions/href](../types/basic.md#href) -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/chart.md b/docs/openmetadata-apis/schemas/entities/chart.md index 39b55925f17..0ed062b71bb 100644 --- a/docs/openmetadata-apis/schemas/entities/chart.md +++ b/docs/openmetadata-apis/schemas/entities/chart.md @@ -2,7 +2,7 @@ This schema defines the Chart entity. Charts are built using tables or sql queries by analyzing the data. Charts can be part of Dashboard. -**$id: **[https://open-metadata.org/schema/entity/data/chart.json](https://open-metadata.org/schema/entity/data/chart.json) +**$id: [https://open-metadata.org/schema/entity/data/chart.json](https://open-metadata.org/schema/entity/data/chart.json)** Type: `object` @@ -24,6 +24,15 @@ Type: `object` - **description** - Description of the dashboard, what it is, and how to use it. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **chartType** - $ref: [#/definitions/chartType](#charttype) - **chartUrl** @@ -54,52 +63,6 @@ Type: `object` - Latest usage information for this database. - $ref: [../../type/usageDetails.json](../types/usagedetails.md) -* **id** `required` - * Unique identifier that identifies a chart instance. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** `required` - * Name that identifies this Chart. - * Type: `string` - * Length: between 1 and 64 -* **displayName** - * Display Name that identifies this Chart. It could be a title or label from the source services. - * Type: `string` -* **fullyQualifiedName** - * A unique name that identifies a dashboard in the format 'ServiceName.ChartName'. - * Type: `string` - * Length: between 1 and 64 -* **description** - * Description of the dashboard, what it is, and how to use it. - * Type: `string` -* **chartType** - * $ref: [#/definitions/chartType](chart.md#charttype) -* **chartUrl** - * Chart URL, pointing to its own Service URL. - * Type: `string` - * String format must be a "uri" -* **href** - * Link to the resource corresponding to this entity. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **owner** - * Owner of this dashboard. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **tables** - * Link to table used in this chart. - * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) -* **followers** - * Followers of this chart. - * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) -* **tags** - * Tags for this chart. - * Type: `array` - * **Items** - * $ref: [../../type/tagLabel.json](../types/taglabel.md) -* **service** `required` - * Link to service where this dashboard is hosted in. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **usageSummary** - * Latest usage information for this database. - * $ref: [../../type/usageDetails.json](../types/usagedetails.md) ## Type definitions in this schema ### chartType @@ -119,4 +82,4 @@ Type: `object` 10. _"Other"_ -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/dashboard.md b/docs/openmetadata-apis/schemas/entities/dashboard.md index 2cb3318393a..fe2af3de9cb 100644 --- a/docs/openmetadata-apis/schemas/entities/dashboard.md +++ b/docs/openmetadata-apis/schemas/entities/dashboard.md @@ -2,7 +2,7 @@ This schema defines the Dashboard entity. Dashboards are computed from data and visually present data, metrics, and KPIs. They are updated in real-time and allow interactive data exploration. -**$id: **[https://open-metadata.org/schema/entity/data/dashboard.json](https://open-metadata.org/schema/entity/data/dashboard.json) +**$id: [https://open-metadata.org/schema/entity/data/dashboard.json](https://open-metadata.org/schema/entity/data/dashboard.json)** Type: `object` @@ -24,6 +24,15 @@ Type: `object` - **description** - Description of the dashboard, what it is, and how to use it. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **dashboardUrl** - Dashboard URL. - Type: `string` @@ -54,51 +63,5 @@ Type: `object` - Latest usage information for this database. - $ref: [../../type/usageDetails.json](../types/usagedetails.md) -* **id** `required` - * Unique identifier that identifies a dashboard instance. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** `required` - * Name that identifies this dashboard. - * Type: `string` - * Length: between 1 and 64 -* **displayName** - * Display Name that identifies this Dashboard. It could be a title or label from the source services. - * Type: `string` -* **fullyQualifiedName** - * A unique name that identifies a dashboard in the format 'ServiceName.DashboardName'. - * Type: `string` - * Length: between 1 and 64 -* **description** - * Description of the dashboard, what it is, and how to use it. - * Type: `string` -* **dashboardUrl** - * Dashboard URL. - * Type: `string` - * String format must be a "uri" -* **charts** - * All the charts are included in this Dashboard. - * Type: `array` - * **Items** - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **href** - * Link to the resource corresponding to this entity. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **owner** - * Owner of this dashboard. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **followers** - * Followers of this dashboard. - * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) -* **tags** - * Tags for this dashboard. - * Type: `array` - * **Items** - * $ref: [../../type/tagLabel.json](../types/taglabel.md) -* **service** `required` - * Link to service where this dashboard is hosted in. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **usageSummary** - * Latest usage information for this database. - * $ref: [../../type/usageDetails.json](../types/usagedetails.md) -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/dashboardservice.md b/docs/openmetadata-apis/schemas/entities/dashboardservice.md index d6e740da1d6..e9d1b040f37 100644 --- a/docs/openmetadata-apis/schemas/entities/dashboardservice.md +++ b/docs/openmetadata-apis/schemas/entities/dashboardservice.md @@ -2,7 +2,7 @@ This schema defines the Dashboard Service entity, such as Looker and Superset. -**$id: **[https://open-metadata.org/schema/entity/services/dashboardService.json](https://open-metadata.org/schema/entity/services/dashboardService.json) +**$id: [https://open-metadata.org/schema/entity/services/dashboardService.json](https://open-metadata.org/schema/entity/services/dashboardService.json)** Type: `object` @@ -23,6 +23,15 @@ Type: `object` - **description** - Description of a dashboard service instance. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **dashboardUrl** `required` - Dashboard Service URL. This will be used to make REST API calls to Dashboard Service. - Type: `string` @@ -40,35 +49,6 @@ Type: `object` - Link to the resource corresponding to this dashboard service. - $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **id** `required` - * Unique identifier of this dashboard service instance. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** `required` - * Name that identifies this dashboard service. - * Type: `string` - * Length: between 1 and 64 -* **serviceType** `required` - * Type of dashboard services such as Looker or Superset... - * $ref: [#/definitions/dashboardServiceType](dashboardservice.md#dashboardservicetype) -* **description** - * Description of a dashboard service instance. - * Type: `string` -* **dashboardUrl** `required` - * Dashboard Service URL. This will be used to make REST API calls to Dashboard Service. - * Type: `string` - * String format must be a "uri" -* **username** - * Username to log in to Dashboard Service. - * Type: `string` -* **password** - * Password to log in to Dashboard Service. - * Type: `string` -* **ingestionSchedule** - * Schedule for running metadata ingestion jobs. - * $ref: [../../type/schedule.json](../types/schedule.md) -* **href** - * Link to the resource corresponding to this messaging service. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) ## Type definitions in this schema ### dashboardServiceType @@ -82,4 +62,5 @@ Type: `object` 4. _"Redash"_ -_This document was updated on: Thursday, September 16, 2021_ + +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/database.md b/docs/openmetadata-apis/schemas/entities/database.md index 20a1ac7f816..7e805c8849a 100644 --- a/docs/openmetadata-apis/schemas/entities/database.md +++ b/docs/openmetadata-apis/schemas/entities/database.md @@ -2,7 +2,7 @@ This schema defines the Database entity. A database also referred to as Database Catalog is a collection of tables. -**$id: **[https://open-metadata.org/schema/entity/data/database.json](https://open-metadata.org/schema/entity/data/database.json) +**$id: [https://open-metadata.org/schema/entity/data/database.json](https://open-metadata.org/schema/entity/data/database.json)** Type: `object` @@ -22,6 +22,15 @@ Type: `object` - **description** - Description of the database instance. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **href** - Link to the resource corresponding to this entity. - $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) @@ -38,33 +47,6 @@ Type: `object` - References to tables in the database. - $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) -* **id** - * Unique identifier that identifies this database instance. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** `required` - * Name that identifies the database. - * $ref: [#/definitions/databaseName](database.md#databasename) -* **fullyQualifiedName** - * Name that uniquely identifies a database in the format 'ServiceName.DatabaseName'. - * Type: `string` -* **description** - * Description of the database instance. - * Type: `string` -* **href** - * Link to the resource corresponding to this entity. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **owner** - * Owner of this database. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **service** `required` - * Link to the database cluster/service where this database is hosted in. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **usageSummary** - * Latest usage information for this database. - * $ref: [../../type/usageDetails.json](../types/usagedetails.md) -* **tables** - * References to tables in the database. - * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) ## Type definitions in this schema ### databaseName @@ -75,4 +57,5 @@ Type: `object` - Length: between 1 and 64 -_This document was updated on: Thursday, September 16, 2021_ + +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/databaseservice.md b/docs/openmetadata-apis/schemas/entities/databaseservice.md index 03d090420a9..1049b3aa9bf 100644 --- a/docs/openmetadata-apis/schemas/entities/databaseservice.md +++ b/docs/openmetadata-apis/schemas/entities/databaseservice.md @@ -2,7 +2,7 @@ This schema defines the Database Service entity, such as MySQL, BigQuery, Redshift, Postgres, or Snowflake. Alternative terms such as Database Cluster, Database Server instance are also used for database service. -**$id: **[https://open-metadata.org/schema/entity/services/databaseService.json](https://open-metadata.org/schema/entity/services/databaseService.json) +**$id: [https://open-metadata.org/schema/entity/services/databaseService.json](https://open-metadata.org/schema/entity/services/databaseService.json)** Type: `object` @@ -23,6 +23,15 @@ Type: `object` - **description** - Description of a database service instance. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **href** `required` - Link to the resource corresponding to this database service. - $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) @@ -33,35 +42,13 @@ Type: `object` - Schedule for running metadata ingestion jobs. - $ref: [../../type/schedule.json](../types/schedule.md) -* **id** `required` - * Unique identifier of this database service instance. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** `required` - * Name that identifies this database service. - * Type: `string` - * Length: between 1 and 64 -* **serviceType** `required` - * Type of database service such as MySQL, BigQuery, Snowflake, Redshift, Postgres... - * $ref: [#/definitions/databaseServiceType](databaseservice.md#databaseservicetype) -* **description** - * Description of a database service instance. - * Type: `string` -* **href** `required` - * Link to the resource corresponding to this database service. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **jdbc** `required` - * JDBC connection information. - * $ref: [../../type/jdbcConnection.json#/definitions/jdbcInfo](../types/jdbcconnection.md#jdbcinfo) -* **ingestionSchedule** - * Schedule for running metadata ingestion jobs. - * $ref: [../../type/schedule.json](../types/schedule.md) ## Type definitions in this schema ### databaseServiceType -* Type of database service such as MySQL, BigQuery, Snowflake, Redshift, Postgres... -* Type: `string` -* The value is restricted to the following: +- Type of database service such as MySQL, BigQuery, Snowflake, Redshift, Postgres... +- Type: `string` +- The value is restricted to the following: 1. _"BigQuery"_ 2. _"MySQL"_ 3. _"Redshift"_ @@ -72,6 +59,7 @@ Type: `object` 8. _"Oracle"_ 9. _"Athena"_ 10. _"Presto"_ - 11. _"Vertica"_ + 11. _"Trino"_ + 12. _"Vertica"_ -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/messagingservice.md b/docs/openmetadata-apis/schemas/entities/messagingservice.md index d06ad9d7bf3..28e39b61779 100644 --- a/docs/openmetadata-apis/schemas/entities/messagingservice.md +++ b/docs/openmetadata-apis/schemas/entities/messagingservice.md @@ -2,7 +2,7 @@ This schema defines the Messaging Service entity, such as Kafka and Pulsar. -**$id: **[https://open-metadata.org/schema/entity/services/messagingService.json](https://open-metadata.org/schema/entity/services/messagingService.json) +**$id: [https://open-metadata.org/schema/entity/services/messagingService.json](https://open-metadata.org/schema/entity/services/messagingService.json)** Type: `object` @@ -21,8 +21,17 @@ Type: `object` - Description of a messaging service instance. - Type: `string` - **displayName** - - Display Name that identifies this messaging service. It could be title or label from the source services. - - Type: `string` + - Display Name that identifies this messaging service. It could be title or label from the source services. + - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **brokers** `required` - Multiple bootstrap addresses for Kafka. Single proxy address for Pulsar. - $ref: [#/definitions/brokers](#brokers) @@ -37,32 +46,6 @@ Type: `object` - Link to the resource corresponding to this messaging service. - $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **id** `required` - * Unique identifier of this messaging service instance. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** `required` - * Name that identifies this messaging service. - * Type: `string` - * Length: between 1 and 64 -* **serviceType** `required` - * Type of messaging services such as Kafka or Pulsar... - * $ref: [#/definitions/messagingServiceType](messagingservice.md#messagingservicetype) -* **description** - * Description of a messaging service instance. - * Type: `string` -* **brokers** `required` - * Multiple bootstrap addresses for Kafka. Single proxy address for Pulsar. - * $ref: [#/definitions/brokers](messagingservice.md#brokers) -* **schemaRegistry** - * Schema registry URL. - * Type: `string` - * String format must be a "uri" -* **ingestionSchedule** - * Schedule for running metadata ingestion jobs. - * $ref: [../../type/schedule.json](../types/schedule.md) -* **href** - * Link to the resource corresponding to this messaging service. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) ## Type definitions in this schema ### messagingServiceType @@ -76,4 +59,9 @@ Type: `object` ### brokers -_This document was updated on: Thursday, September 16, 2021_ + - Multiple bootstrap addresses for Kafka. Single proxy address for Pulsar. + - Type: `array` + - **Items** + - Type: `string` + +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/metrics.md b/docs/openmetadata-apis/schemas/entities/metrics.md index 30e82fd70da..159215cb1a9 100644 --- a/docs/openmetadata-apis/schemas/entities/metrics.md +++ b/docs/openmetadata-apis/schemas/entities/metrics.md @@ -2,7 +2,7 @@ This schema defines the Metrics entity. Metrics are measurements computed from data such as `Monthly Active Users`. Some of the metrics that measures used to determine performance against an objective are called KPIs or Key Performance Indicators, such as `User Retention`. -**$id: **[https://open-metadata.org/schema/entity/data/metrics.json](https://open-metadata.org/schema/entity/data/metrics.json) +**$id: [https://open-metadata.org/schema/entity/data/metrics.json](https://open-metadata.org/schema/entity/data/metrics.json)** Type: `object` @@ -24,6 +24,15 @@ Type: `object` - **description** - Description of metrics instance, what it is, and how to use it. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **href** - Link to the resource corresponding to this entity. - $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) @@ -37,31 +46,5 @@ Type: `object` - Latest usage information for this database. - $ref: [../../type/usageDetails.json](../types/usagedetails.md) -* **id** `required` - * Unique identifier that identifies this metrics instance. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** `required` - * Name that identifies this metrics instance uniquely. - * Type: `string` - * Length: between 1 and 64 -* **fullyQualifiedName** - * A unique name that identifies a metric in the format 'ServiceName.MetricName'. - * Type: `string` - * Length: between 1 and 64 -* **description** - * Description of metrics instance, what it is, and how to use it. - * Type: `string` -* **href** - * Link to the resource corresponding to this entity. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **owner** - * Owner of these metrics. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **service** `required` - * Link to service where this metrics is hosted in. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **usageSummary** - * Latest usage information for this database. - * $ref: [../../type/usageDetails.json](../types/usagedetails.md) -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/model.md b/docs/openmetadata-apis/schemas/entities/model.md index 5af5c84c069..05b77236f52 100644 --- a/docs/openmetadata-apis/schemas/entities/model.md +++ b/docs/openmetadata-apis/schemas/entities/model.md @@ -21,6 +21,15 @@ Type: `object` - **displayName** - Display Name that identifies this model. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **description** - Description of the model, what it is, and how to use it. - Type: `string` @@ -49,4 +58,4 @@ Type: `object` - $ref: [../../type/usageDetails.json](../types/usagedetails.md) -_This document was updated on: Thursday, October 14, 2021_ \ No newline at end of file +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/pipeline.md b/docs/openmetadata-apis/schemas/entities/pipeline.md index d45dbd3d0ae..53b3f150dd7 100644 --- a/docs/openmetadata-apis/schemas/entities/pipeline.md +++ b/docs/openmetadata-apis/schemas/entities/pipeline.md @@ -2,7 +2,7 @@ This schema defines the Pipeline entity. A pipeline enables the flow of data from source to destination through a series of processing steps. ETL is a type of pipeline where the series of steps Extract, Transform and Load the data. -**$id: **[https://open-metadata.org/schema/entity/data/pipeline.json](https://open-metadata.org/schema/entity/data/pipeline.json) +**$id: [https://open-metadata.org/schema/entity/data/pipeline.json](https://open-metadata.org/schema/entity/data/pipeline.json)** Type: `object` @@ -24,6 +24,15 @@ Type: `object` - **description** - Description of this Pipeline. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **pipelineUrl** - Pipeline URL to visit/manage. This URL points to respective pipeline service UI. - Type: `string` @@ -60,28 +69,5 @@ Type: `object` - Link to service where this pipeline is hosted in. - $ref: [../../type/entityReference.json](../types/entityreference.md) -* **id** `required` - * Unique identifier that identifies a pipeline instance. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** `required` - * Name that identifies this pipeline instance uniquely. - * Type: `string` - * Length: between 1 and 64 -* **fullyQualifiedName** - * A unique name that identifies a pipeline in the format 'ServiceName.PipelineName'. - * Type: `string` - * Length: between 1 and 64 -* **description** - * Description of this pipeline. - * Type: `string` -* **href** - * Link to the resource corresponding to this entity. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **owner** - * Owner of this pipeline. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **service** `required` - * Link to service where this pipeline is hosted in. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/pipelineservice.md b/docs/openmetadata-apis/schemas/entities/pipelineservice.md index 19fb2281563..a7e5ea06d0d 100644 --- a/docs/openmetadata-apis/schemas/entities/pipelineservice.md +++ b/docs/openmetadata-apis/schemas/entities/pipelineservice.md @@ -21,8 +21,17 @@ Type: `object` - Description of a pipeline service instance. - Type: `string` - **displayName** - - Display Name that identifies this pipeline service. It could be title or label from the source services. - - Type: `string` + - Display Name that identifies this pipeline service. It could be title or label from the source services. + - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **pipelineUrl** `required` - Pipeline Service Management/UI URL. - Type: `string` @@ -44,5 +53,4 @@ Type: `object` 1. _"Airflow"_ 2. _"Prefect"_ - -_This document was updated on: Thursday, October 14, 2021_ \ No newline at end of file +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/report.md b/docs/openmetadata-apis/schemas/entities/report.md index 41ff9de0098..f73080e7ff9 100644 --- a/docs/openmetadata-apis/schemas/entities/report.md +++ b/docs/openmetadata-apis/schemas/entities/report.md @@ -2,7 +2,7 @@ This schema defines the Report entity. Reports are static information computed from data periodically that includes data in text, table, and visual form. -**$id: **[https://open-metadata.org/schema/entity/data/report.json](https://open-metadata.org/schema/entity/data/report.json) +**$id: [https://open-metadata.org/schema/entity/data/report.json](https://open-metadata.org/schema/entity/data/report.json)** Type: `object` @@ -24,6 +24,15 @@ Type: `object` - **description** - Description of this report instance. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **href** - Link to the resource corresponding to this report. - $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) @@ -37,31 +46,5 @@ Type: `object` - Latest usage information for this database. - $ref: [../../type/usageDetails.json](../types/usagedetails.md) -* **id** `required` - * The Unique identifier that identifies this report. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** `required` - * Name that identifies this report instance uniquely. - * Type: `string` - * Length: between 1 and 64 -* **fullyQualifiedName** - * A unique name that identifies a report in the format 'ServiceName.ReportName'. - * Type: `string` - * Length: between 1 and 64 -* **description** - * Description of this report instance. - * Type: `string` -* **href** - * Link to the resource corresponding to this report. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **owner** - * Owner of this pipeline. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **service** `required` - * Link to service where this report is hosted in. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **usageSummary** - * Latest usage information for this database. - * $ref: [../../type/usageDetails.json](../types/usagedetails.md) -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/table.md b/docs/openmetadata-apis/schemas/entities/table.md index 91a1431dc70..62c6c36e2fe 100644 --- a/docs/openmetadata-apis/schemas/entities/table.md +++ b/docs/openmetadata-apis/schemas/entities/table.md @@ -2,7 +2,7 @@ This schema defines the Table entity. A Table organizes data in rows and columns and is defined by a Schema. OpenMetadata does not have a separate abstraction for Schema. Both Table and Schema are captured in this entity. -**$id: **[https://open-metadata.org/schema/entity/data/table.json](https://open-metadata.org/schema/entity/data/table.json) +**$id: [https://open-metadata.org/schema/entity/data/table.json](https://open-metadata.org/schema/entity/data/table.json)** Type: `object` @@ -14,19 +14,28 @@ Type: `object` - Name of a table. Expected to be unique within a database. - $ref: [#/definitions/tableName](#tablename) - **displayName** - - Display Name that identifies this table. It could be title or label from the source services. + - Display Name that identifies this table. It could be title or label from the source services. + - Type: `string` + - **fullyQualifiedName** + - Fully qualified name of a table in the form `serviceName.databaseName.tableName`. - Type: `string` - **description** - Description of a table. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **href** - Link to this table resource. - $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) - **tableType** - $ref: [#/definitions/tableType](#tabletype) - - **fullyQualifiedName** - - Fully qualified name of a table in the form `serviceName.databaseName.tableName`. - - Type: `string` - **columns** `required` - Columns in this table. - Type: `array` @@ -37,15 +46,9 @@ Type: `object` - Type: `array` - **Items** - $ref: [#/definitions/tableConstraint](#tableconstraint) - - **usageSummary** - - Latest usage information for this table. - - $ref: [../../type/usageDetails.json](../types/usagedetails.md) - **owner** - Owner of this table. - $ref: [../../type/entityReference.json](../types/entityreference.md) - - **followers** - - Followers of this table. - - $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) - **database** - Reference to Database that contains this table. - $ref: [../../type/entityReference.json](../types/entityreference.md) @@ -57,6 +60,12 @@ Type: `object` - Type: `array` - **Items** - $ref: [../../type/tagLabel.json](../types/taglabel.md) + - **usageSummary** + - Latest usage information for this table. + - $ref: [../../type/usageDetails.json](../types/usagedetails.md) + - **followers** + - Followers of this table. + - $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) - **joins** - Details of other tables this table is frequently joined with. - $ref: [#/definitions/tableJoins](#tablejoins) @@ -69,157 +78,105 @@ Type: `object` - **Items** - $ref: [#/definitions/tableProfile](#tableprofile) -* **id** `required` - * Unique identifier of this table instance. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** `required` - * Name of a table. Expected to be unique within a database. - * $ref: [#/definitions/tableName](table.md#tablename) -* **description** - * Description of a table. - * Type: `string` -* **href** - * Link to this table resource. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **tableType** - * $ref: [#/definitions/tableType](table.md#tabletype) -* **fullyQualifiedName** - * Fully qualified name of a table in the form `serviceName.databaseName.tableName`. - * Type: `string` -* **columns** `required` - * Columns in this table. - * Type: `array` - * **Items** - * $ref: [#/definitions/column](table.md#column) -* **tableConstraints** - * Table constraints. - * Type: `array` - * **Items** - * $ref: [#/definitions/tableConstraint](table.md#tableconstraint) -* **usageSummary** - * Latest usage information for this table. - * $ref: [../../type/usageDetails.json](../types/usagedetails.md) -* **owner** - * Owner of this table. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **followers** - * Followers of this table. - * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) -* **database** - * Reference to Database that contains this table. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **viewDefinition** - * View Definition in SQL. Applies to TableType.View only. - * $ref: [../../type/basic.json#/definitions/sqlQuery](../types/basic.md#sqlquery) -* **tags** - * Tags for this table. - * Type: `array` - * **Items** - * $ref: [../../type/tagLabel.json](../types/taglabel.md) -* **joins** - * Details of other tables this table is frequently joined with. - * $ref: [#/definitions/tableJoins](table.md#tablejoins) -* **sampleData** - * Sample data for a table. - * $ref: [#/definitions/tableData](table.md#tabledata) -* **tableProfile** - * Data profile for a table. - * Type: `array` - * **Items** - * $ref: [#/definitions/tableProfile](table.md#tableprofile) ## Type definitions in this schema ### tableType -* This schema defines the type used for describing different types of tables. -* Type: `string` -* The value is restricted to the following: - 1. _"Regular"_ - 2. _"External"_ - 3. _"View"_ - 4. _"SecureView"_ - 5. _"MaterializedView"_ + - This schema defines the type used for describing different types of tables. + - Type: `string` + - The value is restricted to the following: + 1. _"Regular"_ + 2. _"External"_ + 3. _"View"_ + 4. _"SecureView"_ + 5. _"MaterializedView"_ -### columnDataType -* This enum defines the type of data stored in a column. -* Type: `string` -* The value is restricted to the following: - 1. _"NUMBER"_ - 2. _"TINYINT"_ - 3. _"SMALLINT"_ - 4. _"INT"_ - 5. _"BIGINT"_ - 6. _"FLOAT"_ - 7. _"DOUBLE"_ - 8. _"DECIMAL"_ - 9. _"NUMERIC"_ - 10. _"TIMESTAMP"_ - 11. _"TIME"_ - 12. _"DATE"_ - 13. _"DATETIME"_ - 14. _"INTERVAL"_ - 15. _"STRING"_ - 16. _"MEDIUMTEXT"_ - 17. _"TEXT"_ - 18. _"CHAR"_ - 19. _"VARCHAR"_ - 20. _"BOOLEAN"_ - 21. _"BINARY"_ - 22. _"VARBINARY"_ - 23. _"ARRAY"_ - 24. _"BLOB"_ - 25. _"LONGBLOB"_ - 26. _"MEDIUMBLOB"_ - 27. _"MAP"_ - 28. _"STRUCT"_ - 29. _"UNION"_ - 30. _"SET"_ - 31. _"GEOGRAPHY"_ - 32. _"ENUM"_ - 33. _"JSON"_ +### dataType -### columnConstraint + - This enum defines the type of data stored in a column. + - Type: `string` + - The value is restricted to the following: + 1. _"NUMBER"_ + 2. _"TINYINT"_ + 3. _"SMALLINT"_ + 4. _"INT"_ + 5. _"BIGINT"_ + 6. _"BYTEINT"_ + 7. _"FLOAT"_ + 8. _"DOUBLE"_ + 9. _"DECIMAL"_ + 10. _"NUMERIC"_ + 11. _"TIMESTAMP"_ + 12. _"TIME"_ + 13. _"DATE"_ + 14. _"DATETIME"_ + 15. _"INTERVAL"_ + 16. _"STRING"_ + 17. _"MEDIUMTEXT"_ + 18. _"TEXT"_ + 19. _"CHAR"_ + 20. _"VARCHAR"_ + 21. _"BOOLEAN"_ + 22. _"BINARY"_ + 23. _"VARBINARY"_ + 24. _"ARRAY"_ + 25. _"BLOB"_ + 26. _"LONGBLOB"_ + 27. _"MEDIUMBLOB"_ + 28. _"MAP"_ + 29. _"STRUCT"_ + 30. _"UNION"_ + 31. _"SET"_ + 32. _"GEOGRAPHY"_ + 33. _"ENUM"_ + 34. _"JSON"_ + + +### constraint + + - This enum defines the type for column constraint. + - Type: `string` + - The value is restricted to the following: + 1. _"NULL"_ + 2. _"NOT_NULL"_ + 3. _"UNIQUE"_ + 4. _"PRIMARY_KEY"_ -* This enum defines the type for column constraint. -* Type: `string` -* The value is restricted to the following: - 1. _"NULL"_ - 2. _"NOT_NULL"_ - 3. _"UNIQUE"_ - 4. _"PRIMARY_KEY"_ -* Default: _"NULL"_ ### tableConstraint -* This enum defines the type for table constraint. -* Type: `object` -* **Properties** - * **constraintType** - * Type: `string` - * The value is restricted to the following: - 1. _"UNIQUE"_ - 2. _"PRIMARY_KEY"_ - 3. _"FOREIGN_KEY"_ - * **columns** - * List of column names corresponding to the constraint. - * Type: `array` - * **Items** - * Type: `string` + - This enum defines the type for table constraint. + - Type: `object` + - **Properties** + - **constraintType** + - Type: `string` + - The value is restricted to the following: + 1. _"UNIQUE"_ + 2. _"PRIMARY_KEY"_ + 3. _"FOREIGN_KEY"_ + - **columns** + - List of column names corresponding to the constraint. + - Type: `array` + - **Items** + - Type: `string` + ### columnName -* Local name (not fully qualified name) of the column. -* Type: `string` -* The value must match this pattern: `^[^.]*$` -* Length: between 1 and 64 + - Local name (not fully qualified name) of the column. ColumnName is `-` when the column is not named in struct dataType. For example, BigQuery supports struct with unnamed fields. + - Type: `string` + - The value must match this pattern: `^[^.]*$` + - Length: between 1 and 64 + ### tableName -* Local name (not fully qualified name) of a table. -* Type: `string` -* The value must match this pattern: `^[^.]*$` -* Length: between 1 and 64 + - Local name (not fully qualified name) of a table. + - Type: `string` + - The value must match this pattern: `^[^.]*$` + - Length: between 1 and 64 + ### fullyQualifiedColumnName @@ -230,84 +187,106 @@ Type: `object` ### column -* This schema defines the type for a column in a table. -* Type: `object` -* **Properties** - * **name** `required` - * $ref: [#/definitions/columnName](table.md#columnname) - * **columnDataType** `required` - * Data type of the column (int, date etc.). - * $ref: [#/definitions/columnDataType](table.md#columndatatype) - * **description** - * Description of the column. - * Type: `string` - * **fullyQualifiedName** - * $ref: [#/definitions/fullyQualifiedColumnName](table.md#fullyqualifiedcolumnname) - * **tags** - * Tags associated with the column. - * Type: `array` - * **Items** - * $ref: [../../type/tagLabel.json](../types/taglabel.md) - * **columnConstraint** - * Column level constraint. - * $ref: [#/definitions/columnConstraint](table.md#columnconstraint) - * **ordinalPosition** - * Ordinal position of the column. - * Type: `integer` + - This schema defines the type for a column in a table. + - Type: `object` + - This schema does not accept additional properties. + - **Properties** + - **name** `required` + - $ref: [#/definitions/columnName](#columnname) + - **dataType** `required` + - Data type of the column (int, date etc.). + - $ref: [#/definitions/dataType](#datatype) + - **arrayDataType** + - Data type used array in dataType. For example, `array` has dataType as `array` and arrayDataType as `int`. + - $ref: [#/definitions/dataType](#datatype) + - **dataLength** + - Length of `char`, `varchar`, `binary`, `varbinary` `dataTypes`, else null. For example, `varchar(20)` has dataType as `varchar` and dataLength as `20`. + - Type: `integer` + - **dataTypeDisplay** + - Display name used for dataType. This is useful for complex types, such as `array, map, struct<>, and union types. + - Type: `string` + - **description** + - Description of the column. + - Type: `string` + - **fullyQualifiedName** + - $ref: [#/definitions/fullyQualifiedColumnName](#fullyqualifiedcolumnname) + - **tags** + - Tags associated with the column. + - Type: `array` + - **Items** + - $ref: [../../type/tagLabel.json](../types/taglabel.md) + - **constraint** + - Column level constraint. + - $ref: [#/definitions/constraint](#constraint) + - **ordinalPosition** + - Ordinal position of the column. + - Type: `integer` + - **jsonSchema** + - Json schema only if the dataType is JSON else null. + - Type: `string` + - **children** + - Child columns if dataType or arrayDataType is `map`, `struct`, or `union` else `null`. + - Type: `array` + - **Items** + - $ref: [#/definitions/column](#column) + ### columnJoins -* This schema defines the type to capture how frequently a column is joined with columns in the other tables. -* Type: `object` -* This schema does not accept additional properties. -* **Properties** - * **columnName** - * $ref: [#/definitions/columnName](table.md#columnname) - * **joinedWith** - * Fully qualified names of the columns that this column is joined with. - * Type: `array` - * **Items** - * Type: `object` -* **Properties** - * **fullyQualifiedName** - * $ref: [#/definitions/fullyQualifiedColumnName](table.md#fullyqualifiedcolumnname) - * **joinCount** - * Type: `integer` + - This schema defines the type to capture how frequently a column is joined with columns in the other tables. + - Type: `object` + - This schema does not accept additional properties. + - **Properties** + - **columnName** + - $ref: [#/definitions/columnName](#columnname) + - **joinedWith** + - Fully qualified names of the columns that this column is joined with. + - Type: `array` + - **Items** + - Type: `object` + - **Properties** + - **fullyQualifiedName** + - $ref: [#/definitions/fullyQualifiedColumnName](#fullyqualifiedcolumnname) + - **joinCount** + - Type: `integer` + ### tableJoins -* This schema defines the type to capture information about how columns in this table are joined with columns in the other tables. -* Type: `object` -* This schema does not accept additional properties. -* **Properties** - * **startDate** - * Date can be only from today going back to last 29 days. - * $ref: [../../type/basic.json#/definitions/date](../types/basic.md#date) - * **dayCount** - * Type: `integer` - * Default: `1` - * **columnJoins** - * Type: `array` - * **Items** - * $ref: [#/definitions/columnJoins](table.md#columnjoins) + - This schema defines the type to capture information about how columns in this table are joined with columns in the other tables. + - Type: `object` + - This schema does not accept additional properties. + - **Properties** + - **startDate** + - Date can be only from today going back to last 29 days. + - $ref: [../../type/basic.json#/definitions/date](../types/basic.md#date) + - **dayCount** + - Type: `integer` + - Default: `1` + - **columnJoins** + - Type: `array` + - **Items** + - $ref: [#/definitions/columnJoins](#columnjoins) + ### tableData -* This schema defines the type to capture rows of sample data for a table. -* Type: `object` -* This schema does not accept additional properties. -* **Properties** -* **columns** - * List of local column names (not fully qualified column names) of the table. - * Type: `array` - * **Items** - * $ref: [#/definitions/columnName](table.md#columnname) -* **rows** - * Data for multiple rows of the table. - * Type: `array` - * **Items** - * Data for a single row of the table within the same order as columns fields. - * Type: `array` + - This schema defines the type to capture rows of sample data for a table. + - Type: `object` + - This schema does not accept additional properties. + - **Properties** + - **columns** + - List of local column names (not fully qualified column names) of the table. + - Type: `array` + - **Items** + - $ref: [#/definitions/columnName](#columnname) + - **rows** + - Data for multiple rows of the table. + - Type: `array` + - **Items** + - Data for a single row of the table within the same order as columns fields. + - Type: `array` + ### columnProfile @@ -346,23 +325,25 @@ Type: `object` - Type: `number` -* This schema defines the type to capture the table's data profile. -* Type: `object` -* This schema does not accept additional properties. - * **Properties** - * **profileDate** - * Data one which profile is taken. - * $ref: [../../type/basic.json#/definitions/date](../types/basic.md#date) - * **columnCount** - * No.of columns in the table. - * Type: `number` - * **rowCount** - * No.of rows in the table. - * Type: `number` - * **columnProfile** - * List of local column profiles of the table. - * Type: `array` - * **Items** - * $ref: [#/definitions/columnProfile](table.md#columnprofile) +### tableProfile -_This document was updated on: Thursday, September 16, 2021_ + - This schema defines the type to capture the table's data profile. + - Type: `object` + - This schema does not accept additional properties. + - **Properties** + - **profileDate** + - Data one which profile is taken. + - $ref: [../../type/basic.json#/definitions/date](../types/basic.md#date) + - **columnCount** + - No.of columns in the table. + - Type: `number` + - **rowCount** + - No.of rows in the table. + - Type: `number` + - **columnProfile** + - List of local column profiles of the table. + - Type: `array` + - **Items** + - $ref: [#/definitions/columnProfile](#columnprofile) + +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/tagcategory.md b/docs/openmetadata-apis/schemas/entities/tagcategory.md index fc43159a913..55f6e31e6bf 100644 --- a/docs/openmetadata-apis/schemas/entities/tagcategory.md +++ b/docs/openmetadata-apis/schemas/entities/tagcategory.md @@ -2,7 +2,7 @@ This schema defines the Tag Category entity. A Tag Category contains tags called Primary Tags. Primary Tags can further have children Tags called Secondary Tags. Only two levels of tags are supported currently. -**$id: **[https://open-metadata.org/schema/entity/tags/tagCategory.json](https://open-metadata.org/schema/entity/tags/tagCategory.json) +**$id: [https://open-metadata.org/schema/entity/tags/tagCategory.json](https://open-metadata.org/schema/entity/tags/tagCategory.json)** Type: `object` @@ -17,6 +17,15 @@ This schema does not accept additional properties. - **description** `required` - Description of the tag category. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **categoryType** `required` - $ref: [#/definitions/tagCategoryType](#tagcategorytype) - **href** @@ -31,24 +40,6 @@ This schema does not accept additional properties. - **Items** - $ref: [#/definitions/tag](#tag) -* **name** `required` - * $ref: [#/definitions/tagName](tagcategory.md#tagname) -* **description** `required` - * Description of the tag category. - * Type: `string` -* **categoryType** `required` - * $ref: [#/definitions/tagCategoryType](tagcategory.md#tagcategorytype) -* **href** - * Link to the resource corresponding to the tag category. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **usageCount** - * Count how many times the tags from this tag category are used. - * Type: `integer` -* **children** - * Tags under this category. - * Type: `array` - * **Items** - * $ref: [#/definitions/tag](tagcategory.md#tag) ## Type definitions in this schema ### tagName @@ -69,4 +60,4 @@ This schema does not accept additional properties. ### tag -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/task.md b/docs/openmetadata-apis/schemas/entities/task.md index 5f0a8521b68..c0ab6b4ae2d 100644 --- a/docs/openmetadata-apis/schemas/entities/task.md +++ b/docs/openmetadata-apis/schemas/entities/task.md @@ -24,6 +24,15 @@ Type: `object` - **description** - Description of this Task. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **taskUrl** - Task URL to visit/manage. This URL points to respective pipeline service UI. - Type: `string` @@ -62,4 +71,4 @@ Type: `object` - $ref: [../../type/entityReference.json](../types/entityreference.md) -_This document was updated on: Thursday, October 14, 2021_ \ No newline at end of file +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/team.md b/docs/openmetadata-apis/schemas/entities/team.md index 8b030b2aa10..3c446e68ee8 100644 --- a/docs/openmetadata-apis/schemas/entities/team.md +++ b/docs/openmetadata-apis/schemas/entities/team.md @@ -2,7 +2,7 @@ This schema defines the Team entity. A Team is a group of zero or more users. Teams can own zero or more data assets. -**$id: **[https://open-metadata.org/schema/entity/teams/team.json](https://open-metadata.org/schema/entity/teams/team.json) +**$id: [https://open-metadata.org/schema/entity/teams/team.json](https://open-metadata.org/schema/entity/teams/team.json)** Type: `object` @@ -17,6 +17,15 @@ Type: `object` - **description** - Description of the team. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **href** `required` - Link to the resource corresponding to this entity. - $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) @@ -33,31 +42,6 @@ Type: `object` - List of entities owned by the team. - $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) -* **id** `required` - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** `required` - * $ref: [#/definitions/teamName](team.md#teamname) -* **displayName** - * Name used for display purposes. Example 'Data Science team'. - * Type: `string` -* **description** - * Description of the team. - * Type: `string` -* **href** `required` - * Link to the resource corresponding to this entity. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **profile** - * Team profile information. - * $ref: [../../type/profile.json](../types/profile.md) -* **deleted** - * When true the team has been deleted. - * Type: `boolean` -* **users** - * Users that are part of the team. - * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) -* **owns** - * List of entities owned by the team. - * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) ## Type definitions in this schema ### teamName @@ -67,4 +51,4 @@ Type: `object` - Length: between 1 and 64 -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/thread.md b/docs/openmetadata-apis/schemas/entities/thread.md index 211c27dcd39..a3ab582e9b2 100644 --- a/docs/openmetadata-apis/schemas/entities/thread.md +++ b/docs/openmetadata-apis/schemas/entities/thread.md @@ -2,7 +2,7 @@ This schema defines the Thread entity. A Thread is a collection of posts made by the users. The first post that starts a thread is **about** a data asset **from** a user. Other users can respond to this post by creating new posts in the thread. Note that bot users can also interact with a thread. A post can contains links that mention Users or other Data Assets. -**$id: **[https://open-metadata.org/schema/entity/feed/thread.json](https://open-metadata.org/schema/entity/feed/thread.json) +**$id: [https://open-metadata.org/schema/entity/feed/thread.json](https://open-metadata.org/schema/entity/feed/thread.json)** Type: `object` @@ -27,41 +27,23 @@ Type: `object` - **Items** - $ref: [#/definitions/post](#post) -* **id** `required` - * Unique identifier that identifies an entity instance. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **href** - * Link to the resource corresponding to this entity. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **threadTs** - * Timestamp of the when the first post created the thread. - * $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) -* **about** `required` - * Data asset about which this thread is created for with format <#E/{enties}/{entityName}/{field}/{fieldValue}. - * $ref: [../../type/basic.json#/definitions/entityLink](../types/basic.md#entitylink) -* **addressedTo** - * User or team this thread is addressed to in format <#E/{enties}/{entityName}/{field}/{fieldValue}. - * $ref: [../../type/basic.json#/definitions/entityLink](../types/basic.md#entitylink) -* **posts** `required` - * Type: `array` - * **Items** - * $ref: [#/definitions/post](thread.md#post) ## Type definitions in this schema ### post -* Post within a feed. -* Type: `object` -* **Properties** - * **message** `required` - * Message in markdown format. See markdown support for more details. - * Type: `string` - * **postTs** - * Timestamp of the post. - * Type: `string` - * String format must be a "date-time" - * **from** `required` - * The ID of the User (regular user or a bot) posting the message. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) + - Post within a feed. + - Type: `object` + - **Properties** + - **message** `required` + - Message in markdown format. See markdown support for more details. + - Type: `string` + - **postTs** + - Timestamp of the post. + - Type: `string` + - String format must be a "date-time" + - **from** `required` + - ID of User (regular user or a bot) posting the message. + - $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -_This document was updated on: Thursday, September 16, 2021_ + +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/topic.md b/docs/openmetadata-apis/schemas/entities/topic.md index 73cf12a6f90..cd9d82cac03 100644 --- a/docs/openmetadata-apis/schemas/entities/topic.md +++ b/docs/openmetadata-apis/schemas/entities/topic.md @@ -2,7 +2,7 @@ This schema defines the Topic entity. A topic is a feed into which message are published to by publishers and read from by consumers in a messaging service. -**$id: **[https://open-metadata.org/schema/entity/data/topic.json](https://open-metadata.org/schema/entity/data/topic.json) +**$id: [https://open-metadata.org/schema/entity/data/topic.json](https://open-metadata.org/schema/entity/data/topic.json)** Type: `object` @@ -17,11 +17,20 @@ Type: `object` - Name that uniquely identifies a topic in the format 'messagingServiceName.topicName'. - Type: `string` - **displayName** - - Display Name that identifies this topic. It could be title or label from the source services. - - Type: `string` + - Display Name that identifies this topic. It could be title or label from the source services. + - Type: `string` - **description** - Description of the topic instance. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **service** `required` - Link to the messaging cluster/service where this topic is hosted in. - $ref: [../../type/entityReference.json](../types/entityreference.md) @@ -71,66 +80,6 @@ Type: `object` - Link to the resource corresponding to this entity. - $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **id** `required` - * Unique identifier that identifies this topic instance. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** `required` - * Name that identifies the topic. - * $ref: [#/definitions/topicName](topic.md#topicname) -* **fullyQualifiedName** - * Name that uniquely identifies a topic in the format 'messagingServiceName.topicName'. - * Type: `string` -* **description** - * Description of the topic instance. - * Type: `string` -* **service** `required` - * Link to the messaging cluster/service where this topic is hosted in. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **partitions** `required` - * A number of partitions into which the topic is divided. - * Type: `integer` - * Range: ≥ 1 -* **schemaText** - * The Schema is used for message serialization. Optional as some topics may not have associated schemas. - * Type: `string` -* **schemaType** - * Schema used for message serialization. - * $ref: [#/definitions/schemaType](topic.md#schematype) -* **cleanupPolicies** - * Topic clean up policies. For Kafka - `cleanup.policy` configuration. - * Type: `array` - * **Items** - * $ref: [#/definitions/cleanupPolicy](topic.md#cleanuppolicy) -* **retentionTime** - * Retention time in milliseconds. For Kafka - `retention.ms` configuration. - * Type: `number` -* **replicationFactor** - * Replication Factor in integer (more than 1). - * Type: `integer` -* **maximumMessageSize** - * Maximum message size in bytes. For Kafka - `max.message.bytes` configuration. - * Type: `integer` -* **minimumInSyncReplicas** - * The minimum number of replicas in sync to control durability. For Kafka - `min.insync.replicas` configuration. - * Type: `integer` -* **retentionSize** - * Maximum size of a partition in bytes before old data is discarded. For Kafka - `retention.bytes` configuration. - * Type: `number` - * Default: _"-1"_ -* **owner** - * Owner of this topic. - * $ref: [../../type/entityReference.json](../types/entityreference.md) -* **followers** - * Followers of this table. - * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) -* **tags** - * Tags for this table. - * Type: `array` - * **Items** - * $ref: [../../type/tagLabel.json](../types/taglabel.md) -* **href** - * Link to the resource corresponding to this entity. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) ## Type definitions in this schema ### topicName @@ -159,4 +108,4 @@ Type: `object` 2. _"compact"_ -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/entities/user.md b/docs/openmetadata-apis/schemas/entities/user.md index 7687471b0c5..c7a2d913f8c 100644 --- a/docs/openmetadata-apis/schemas/entities/user.md +++ b/docs/openmetadata-apis/schemas/entities/user.md @@ -2,7 +2,7 @@ This schema defines the User entity. A user can be part of 0 or more teams. A special type of user called Bot is used for automation. A user can be an owner of zero or more data assets. A user can also follow zero or more data assets. -**$id: **[https://open-metadata.org/schema/entity/teams/user.json](https://open-metadata.org/schema/entity/teams/user.json) +**$id: [https://open-metadata.org/schema/entity/teams/user.json](https://open-metadata.org/schema/entity/teams/user.json)** Type: `object` @@ -17,6 +17,15 @@ This schema does not accept additional properties. - **displayName** - Name used for display purposes. Example 'FirstName LastName'. - Type: `string` + - **version** + - Metadata version of the entity. + - $ref: [../../type/basic.json#/definitions/entityVersion](../types/basic.md#entityversion) + - **updatedAt** + - Last update time corresponding to the new version of the entity. + - $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime) + - **updatedBy** + - User who made the update. + - Type: `string` - **email** `required` - Email address of the user. - $ref: [../../type/basic.json#/definitions/email](../types/basic.md#email) @@ -49,45 +58,6 @@ This schema does not accept additional properties. - List of entities followed by the user. - $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) -* **id** `required` - * Unique identifier that identifies a user entity instance. - * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid) -* **name** `required` - * $ref: [#/definitions/userName](user.md#username) -* **displayName** - * Name used for display purposes. Example 'FirstName LastName'. - * Type: `string` -* **email** `required` - * Email address of the user. - * $ref: [../../type/basic.json#/definitions/email](../types/basic.md#email) -* **href** `required` - * Link to the resource corresponding to this entity. - * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href) -* **timezone** - * The Timezone of the user. - * Type: `string` - * String format must be a "timezone" -* **deactivated** - * When true indicates the user has been deactivated. Users are deactivated instead of deleted. - * Type: `boolean` -* **isBot** - * When true indicates a special type of user called Bot. - * Type: `boolean` -* **isAdmin** - * When true indicates the user is an administrator for the system with superuser privileges. - * Type: `boolean` -* **profile** - * Profile of the user. - * $ref: [../../type/profile.json](../types/profile.md) -* **teams** - * Teams that the user belongs to. - * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) -* **owns** - * List of entities owned by the user. - * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) -* **follows** - * List of entities followed by the user. - * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist) ## Type definitions in this schema ### userName @@ -96,4 +66,4 @@ This schema does not accept additional properties. - Type: `string` - Length: between 1 and 64 -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/types/README.md b/docs/openmetadata-apis/schemas/types/README.md index 9778393aaf1..fe5f8a1ed2a 100644 --- a/docs/openmetadata-apis/schemas/types/README.md +++ b/docs/openmetadata-apis/schemas/types/README.md @@ -2,10 +2,10 @@ OpenMetadata supports many common types reused across different schemas. Defining common types and reusing them is foundational to building a consistent schema vocabulary. The types are organized as follows: -{% page-ref page="basic.md" %} - {% page-ref page="auditlog.md" %} +{% page-ref page="basic.md" %} + {% page-ref page="dailycount.md" %} {% page-ref page="entitylineage.md" %} @@ -22,5 +22,7 @@ OpenMetadata supports many common types reused across different schemas. Definin {% page-ref page="schedule.md" %} +{% page-ref page="taglabel.md" %} + {% page-ref page="usagedetails.md" %} diff --git a/docs/openmetadata-apis/schemas/types/auditlog.md b/docs/openmetadata-apis/schemas/types/auditlog.md index a0202a93009..37f55d9e566 100644 --- a/docs/openmetadata-apis/schemas/types/auditlog.md +++ b/docs/openmetadata-apis/schemas/types/auditlog.md @@ -2,7 +2,7 @@ This schema defines the Audit Log type to capture the audit trail of POST, PUT, and PATCH API operations. -**$id: **[https://open-metadata.org/schema/type/auditLog.json](https://open-metadata.org/schema/type/auditLog.json) +**$id: [https://open-metadata.org/schema/type/auditLog.json](https://open-metadata.org/schema/type/auditLog.json)** Type: `object` @@ -34,31 +34,4 @@ Type: `object` - Type of Entity that is modified by the operation. - Type: `string` -* **method** `required` - * HTTP Method used in a call. - * Type: `string` - * The value is restricted to the following: - 1. _"POST"_ - 2. _"PUT"_ - 3. _"PATCH"_ - 4. _"DELETE"_ -* **responseCode** `required` - * HTTP response code for the api request. - * Type: `integer` -* **path** `required` - * Requested API Path. - * Type: `string` -* **userName** `required` - * Name of the user who made the API request. - * Type: `string` -* **dateTime** - * Date when the API call is made. - * $ref: [basic.json#/definitions/dateTime](basic.md#datetime) -* **entityId** `required` - * Identifier of an entity that was modified by the operation. - * $ref: [basic.json#/definitions/uuid](basic.md#uuid) -* **entityType** `required` - * Type of Entity that is modified by the operation. - * Type: `string` - -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/types/basic.md b/docs/openmetadata-apis/schemas/types/basic.md index 0fa462383a5..5bda518eb38 100644 --- a/docs/openmetadata-apis/schemas/types/basic.md +++ b/docs/openmetadata-apis/schemas/types/basic.md @@ -2,7 +2,7 @@ This schema defines basic common types that are used by other schemas. -**$id: **[https://open-metadata.org/schema/type/basic.json](https://open-metadata.org/schema/type/basic.json) +**$id: [https://open-metadata.org/schema/type/basic.json](https://open-metadata.org/schema/type/basic.json)** ## Type definitions in this schema @@ -22,13 +22,6 @@ This schema defines basic common types that are used by other schemas. - Length: between 6 and 127 -### entityLink - - - Link to an entity or field within an entity using this format `<#E/{enties}/{entityName}/{field}/{fieldValue}`. - - Type: `string` - - The value must match this pattern: `^<#E/\S+/\S+>$` - - ### timestamp - Timestamp in unixTimeMillis. @@ -76,10 +69,26 @@ This schema defines basic common types that are used by other schemas. - String format must be a "date-time" +### entityVersion + + - Metadata version of the entity in the form `Major.Minor`. First version always starts from `0.1` when the entity is created. When the backward compatible changes are made to the entity, only the `Minor` version is incremented - example `1.0` is changed to `1.1`. When backward incompatible changes are made the `Major` version is incremented - example `1.1` to `2.0`. + - Type: `number` + - Default: `0.1` + - Range: ≥ 0.1 + - The value must be a multiple of `0.1` + + +### entityLink + + - Link to an entity or field within an entity using this format `<#E/{enties}/{entityName}/{field}/{fieldValue}`. + - Type: `string` + - The value must match this pattern: `^<#E/\S+/\S+>$` + + ### sqlQuery - SQL query statement. Example - 'select * from orders'. - Type: `string` -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/types/dailycount.md b/docs/openmetadata-apis/schemas/types/dailycount.md index a67d79a4113..fbe0d27ec1b 100644 --- a/docs/openmetadata-apis/schemas/types/dailycount.md +++ b/docs/openmetadata-apis/schemas/types/dailycount.md @@ -2,19 +2,18 @@ This schema defines the type for reporting the daily count of some measurement. For example, you might use this schema for the number of times a table is queried each day. -**$id: **[https://open-metadata.org/schema/type/dailyCount.json](https://open-metadata.org/schema/type/dailyCount.json) +**$id: [https://open-metadata.org/schema/type/dailyCount.json](https://open-metadata.org/schema/type/dailyCount.json)** Type: `object` This schema does not accept additional properties. ## Properties + - **count** `required` + - Daily count of a measurement on the given date. + - Type: `integer` + - Range: ≥ 0 + - **date** `required` + - $ref: [basic.json#/definitions/date](basic.md#date) -* **count** `required` - * Daily count of a measurement on the given date. - * Type: `integer` - * Range: ≥ 0 -* **date** `required` - * $ref: [basic.json#/definitions/date](basic.md#date) - -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/types/entitylineage.md b/docs/openmetadata-apis/schemas/types/entitylineage.md index f53552df9c8..56d6ac2043d 100644 --- a/docs/openmetadata-apis/schemas/types/entitylineage.md +++ b/docs/openmetadata-apis/schemas/types/entitylineage.md @@ -57,4 +57,4 @@ This schema does not accept additional properties. - Type: `string` -_This document was updated on: Thursday, October 14, 2021_ \ No newline at end of file +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/types/entityreference.md b/docs/openmetadata-apis/schemas/types/entityreference.md index 7217f6a9728..2c3d009adbe 100644 --- a/docs/openmetadata-apis/schemas/types/entityreference.md +++ b/docs/openmetadata-apis/schemas/types/entityreference.md @@ -2,7 +2,7 @@ This schema defines the EntityReference type used for referencing an entity. EntityReference is used for capturing relationships from one entity to another. For example, a table has an attribute called database of type EntityReference that captures the relationship of a table `belongs to a` database. -**$id: **[https://open-metadata.org/schema/type/entityReference.json](https://open-metadata.org/schema/type/entityReference.json) +**$id: [https://open-metadata.org/schema/type/entityReference.json](https://open-metadata.org/schema/type/entityReference.json)** Type: `object` @@ -28,21 +28,6 @@ This schema does not accept additional properties. - Link to the entity resource. - $ref: [basic.json#/definitions/href](basic.md#href) -* **id** `required` - * Unique identifier that identifies an entity instance. - * $ref: [basic.json#/definitions/uuid](basic.md#uuid) -* **type** `required` - * Entity type/class name - Examples: `database`, `table`, `metrics`, `redshift`, `mysql`, `bigquery`, `snowflake`... - * Type: `string` -* **name** - * Name of the entity instance. For entities such as tables, databases where the name is not unique, fullyQualifiedName is returned in this field. - * Type: `string` -* **description** - * Optional description of the entity. - * Type: `string` -* **href** - * Link to the entity resource. - * $ref: [basic.json#/definitions/href](basic.md#href) ## Type definitions in this schema ### entityReferenceList @@ -52,4 +37,4 @@ This schema does not accept additional properties. - $ref: [entityReference.json](entityreference.md) -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/types/entityusage.md b/docs/openmetadata-apis/schemas/types/entityusage.md index 375cab8f2ac..10f8e9eaf60 100644 --- a/docs/openmetadata-apis/schemas/types/entityusage.md +++ b/docs/openmetadata-apis/schemas/types/entityusage.md @@ -2,7 +2,7 @@ This schema defines the type used for capturing usage details of an entity. -**$id: **[https://open-metadata.org/schema/type/entityUsage.json](https://open-metadata.org/schema/type/entityUsage.json) +**$id: [https://open-metadata.org/schema/type/entityUsage.json](https://open-metadata.org/schema/type/entityUsage.json)** Type: `object` @@ -16,13 +16,4 @@ Type: `object` - **Items** - $ref: [usageDetails.json](usagedetails.md) -* **entity** `required` - * Entity for which usage is returned. - * $ref: [entityReference.json](entityreference.md) -* **usage** `required` - * List usage details per day. - * Type: `array` - * **Items** - * $ref: [usageDetails.json](usagedetails.md) - -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/types/jdbcconnection.md b/docs/openmetadata-apis/schemas/types/jdbcconnection.md index b5e56fe3e3e..131b96d57a9 100644 --- a/docs/openmetadata-apis/schemas/types/jdbcconnection.md +++ b/docs/openmetadata-apis/schemas/types/jdbcconnection.md @@ -2,7 +2,7 @@ This schema defines the type used for JDBC connection information. -**$id: **[https://open-metadata.org/schema/type/jdbcConnection.json](https://open-metadata.org/schema/type/jdbcConnection.json) +**$id: [https://open-metadata.org/schema/type/jdbcConnection.json](https://open-metadata.org/schema/type/jdbcConnection.json)** Type: `object` @@ -20,18 +20,6 @@ Type: `object` - Login password. - Type: `string` -* **driverClass** `required` - * JDBC driver class. - * $ref: [#/definitions/driverClass](jdbcconnection.md#driverclass) -* **connectionUrl** `required` - * JDBC connection URL. - * $ref: [#/definitions/connectionUrl](jdbcconnection.md#connectionurl) -* **userName** `required` - * Login user name. - * Type: `string` -* **password** `required` - * Login password. - * Type: `string` ## Type definitions in this schema ### driverClass @@ -48,12 +36,12 @@ Type: `object` ### jdbcInfo -* Type for capturing JDBC connector information. -* Type: `object` -* **Properties** - * **driverClass** `required` - * $ref: [#/definitions/driverClass](jdbcconnection.md#driverclass) - * **connectionUrl** `required` - * $ref: [#/definitions/connectionUrl](jdbcconnection.md#connectionurl) + - Type for capturing JDBC connector information. + - Type: `object` + - **Properties** + - **driverClass** `required` + - $ref: [#/definitions/driverClass](#driverclass) + - **connectionUrl** `required` + - $ref: [#/definitions/connectionUrl](#connectionurl) -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/types/paging.md b/docs/openmetadata-apis/schemas/types/paging.md index a69bf83d8cd..a42b1f77cda 100644 --- a/docs/openmetadata-apis/schemas/types/paging.md +++ b/docs/openmetadata-apis/schemas/types/paging.md @@ -2,7 +2,7 @@ Type used for cursor based pagination information in GET list responses. -**$id: **[https://open-metadata.org/schema/type/paging.json](https://open-metadata.org/schema/type/paging.json) +**$id: [https://open-metadata.org/schema/type/paging.json](https://open-metadata.org/schema/type/paging.json)** Type: `object` @@ -17,14 +17,4 @@ Type: `object` - Total number of entries available to page through. - Type: `integer` -* **before** - * Before cursor is used for getting the previous page (see API pagination for details). - * Type: `string` -* **after** - * After the cursor is used for getting the next page (see API pagination for details). - * Type: `string` -* **total** `required` - * The Total number of entries available to page through. - * Type: `integer` - -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/types/profile.md b/docs/openmetadata-apis/schemas/types/profile.md index ae09c51232d..57102a38e54 100644 --- a/docs/openmetadata-apis/schemas/types/profile.md +++ b/docs/openmetadata-apis/schemas/types/profile.md @@ -2,7 +2,7 @@ This schema defines the type for a profile of a user, team, or organization. -**$id: **[https://open-metadata.org/schema/type/profile.json](https://open-metadata.org/schema/type/profile.json) +**$id: [https://open-metadata.org/schema/type/profile.json](https://open-metadata.org/schema/type/profile.json)** Type: `object` @@ -10,35 +10,33 @@ Type: `object` - **images** - $ref: [#/definitions/imageList](#imagelist) -* **images** - * $ref: [#/definitions/imageList](profile.md#imagelist) ## Type definitions in this schema ### imageList -* Links to a list of images of varying resolutions/sizes. -* Type: `object` -* **Properties** - * **image** - * Type: `string` - * String format must be a "uri" - * **image24** - * Type: `string` - * String format must be a "uri" - * **image32** - * Type: `string` - * String format must be a "uri" - * **image48** - * Type: `string` - * String format must be a "uri" - * **image72** - * Type: `string` - * String format must be a "uri" - * **image192** - * Type: `string` - * String format must be a "uri" - * **image512** - * Type: `string` - * String format must be a "uri" + - Links to a list of images of varying resolutions/sizes. + - Type: `object` + - **Properties** + - **image** + - Type: `string` + - String format must be a "uri" + - **image24** + - Type: `string` + - String format must be a "uri" + - **image32** + - Type: `string` + - String format must be a "uri" + - **image48** + - Type: `string` + - String format must be a "uri" + - **image72** + - Type: `string` + - String format must be a "uri" + - **image192** + - Type: `string` + - String format must be a "uri" + - **image512** + - Type: `string` + - String format must be a "uri" -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/types/schedule.md b/docs/openmetadata-apis/schemas/types/schedule.md index b446d35dec8..f07852bbfd3 100644 --- a/docs/openmetadata-apis/schemas/types/schedule.md +++ b/docs/openmetadata-apis/schemas/types/schedule.md @@ -2,7 +2,7 @@ This schema defines the type used for the schedule. The schedule has a start time and repeat frequency. -**$id: **[https://open-metadata.org/schema/type/schedule.json](https://open-metadata.org/schema/type/schedule.json) +**$id: [https://open-metadata.org/schema/type/schedule.json](https://open-metadata.org/schema/type/schedule.json)** Type: `object` @@ -14,11 +14,4 @@ Type: `object` - Repeat frequency in ISO 8601 duration format. Example - 'P23DT23H'. - $ref: [basic.json#/definitions/duration](basic.md#duration) -* **startDate** - * Start date and time of the schedule. - * $ref: [basic.json#/definitions/dateTime](basic.md#datetime) -* **repeatFrequency** - * Repeat frequency in ISO 8601 duration format. Example - 'P23DT23H'. - * $ref: [basic.json#/definitions/duration](basic.md#duration) - -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/types/taglabel.md b/docs/openmetadata-apis/schemas/types/taglabel.md index 28c61281943..4e5eadf5cc7 100644 --- a/docs/openmetadata-apis/schemas/types/taglabel.md +++ b/docs/openmetadata-apis/schemas/types/taglabel.md @@ -2,33 +2,35 @@ This schema defines the type for labeling an entity with a Tag. -**$id: **[https://open-metadata.org/schema/type/tagLabel.json](https://open-metadata.org/schema/type/tagLabel.json) +**$id: [https://open-metadata.org/schema/type/tagLabel.json](https://open-metadata.org/schema/type/tagLabel.json)** Type: `object` +This schema does not accept additional properties. + ## Properties + - **tagFQN** + - Type: `string` + - Length: ≤ 45 + - **labelType** + - Label type describes how a tag label was applied. 'Manual' indicates the tag label was applied by a person. 'Derived' indicates a tag label was derived using the associated tag relationship (see TagCategory.json for more details). 'Propagated` indicates a tag label was propagated from upstream based on lineage. 'Automated' is used when a tool was used to determine the tag label. + - Type: `string` + - The value is restricted to the following: + 1. _"Manual"_ + 2. _"Propagated"_ + 3. _"Automated"_ + 4. _"Derived"_ + - Default: _"Manual"_ + - **state** + - 'Suggested' state is used when a tag label is suggested by users or tools. Owner of the entity must confirm the suggested labels before it is marked as 'Confirmed'. + - Type: `string` + - The value is restricted to the following: + 1. _"Suggested"_ + 2. _"Confirmed"_ + - Default: _"Confirmed"_ + - **href** + - Link to the tag resource. + - $ref: [basic.json#/definitions/href](basic.md#href) -* **tagFQN** - * Type: `string` - * Length: ≤ 45 -* **labelType** - * Label type describes how a tag label was applied. 'Manual' indicates the tag label was applied by a person. 'Derived' indicates a tag label was derived using the associated tag relationship (see TagCategory.json for more details). 'Propagated\` indicates a tag label was propagated from upstream based on lineage. 'Automated' is used when a tool was used to determine the tag label. - * Type: `string` - * The value is restricted to the following: - 1. _"Manual"_ - 2. _"Propagated"_ - 3. _"Automated"_ - 4. _"Derived"_ - * Default: _"Manual"_ -* **state** - * 'Suggested' state is used when a tag label is suggested by users or tools. The owner of the entity must confirm the suggested labels before it is marked as 'Confirmed'. - * Type: `string` - * The value is restricted to the following: - 1. _"Suggested"_ - 2. _"Confirmed"_ - * Default: _"Confirmed"_ -* **href** - * Link to the tag resource. - * $ref: [basic.json#/definitions/href](basic.md#href) -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file diff --git a/docs/openmetadata-apis/schemas/types/usagedetails.md b/docs/openmetadata-apis/schemas/types/usagedetails.md index da6318ed7f5..7c1a5569434 100644 --- a/docs/openmetadata-apis/schemas/types/usagedetails.md +++ b/docs/openmetadata-apis/schemas/types/usagedetails.md @@ -2,7 +2,7 @@ This schema defines the type for usage details. Daily, weekly, and monthly aggregation of usage is computed along with the percentile rank based on the usage for a given day. -**$id: **[https://open-metadata.org/schema/type/usageDetails.json](https://open-metadata.org/schema/type/usageDetails.json) +**$id: [https://open-metadata.org/schema/type/usageDetails.json](https://open-metadata.org/schema/type/usageDetails.json)** Type: `object` @@ -20,33 +20,21 @@ Type: `object` - Date in UTC. - $ref: [basic.json#/definitions/date](basic.md#date) -* **dailyStats** `required` - * Daily usage stats of a data asset on the start date. - * $ref: [#/definitions/usageStats](usagedetails.md#usagestats) -* **weeklyStats** - * Weekly (last 7 days) rolling usage stats of a data asset on the start date. - * $ref: [#/definitions/usageStats](usagedetails.md#usagestats) -* **monthlyStats** - * Monthly (last 30 days) rolling usage stats of a data asset on the start date. - * $ref: [#/definitions/usageStats](usagedetails.md#usagestats) -* **date** `required` - * Date in UTC. - * $ref: [basic.json#/definitions/date](basic.md#date) ## Type definitions in this schema ### usageStats -* Type used to return usage statistics. -* Type: `object` -* This schema does not accept additional properties. -* **Properties** - * **count** `required` - * Usage count of a data asset on the start date. - * Type: `integer` - * Range: ≥ 0 - * **percentileRank** - * Optional daily percentile rank data asset use when relevant. - * Type: `number` - * Range: between 0 and 100 + - Type used to return usage statistics. + - Type: `object` + - This schema does not accept additional properties. + - **Properties** + - **count** `required` + - Usage count of a data asset on the start date. + - Type: `integer` + - Range: ≥ 0 + - **percentileRank** + - Optional daily percentile rank data asset use when relevant. + - Type: `number` + - Range: between 0 and 100 -_This document was updated on: Thursday, September 16, 2021_ +_This document was updated on: Monday, October 18, 2021_ \ No newline at end of file