Documentation Update (#774)

docs/README.md

@@ -5,7 +5,7 @@ Metadata enables you to unlock the value of data assets in the common use cases
 
 ### **OpenMetadata is an open standard with a centralized metadata store that unifies all your data assets end-to-end to enable data discovery, user collaboration, and tool interoperability.**
 
-![](.gitbook/assets/openmetadata-overview%20%281%29.png)
+![](<../.gitbook/assets/openmetadata-overview (1).png>)
 
 OpenMetadata includes the following:
 
@@ -18,4 +18,3 @@ OpenMetadata includes the following:
 ## License
 
 OpenMetadata is released under [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0)
-

docs/SUMMARY.md

@@ -94,7 +94,8 @@
       * [Vertica](install/metadata-ingestion/connectors/database-services/vertica.md)
     * [Messaging Services](install/metadata-ingestion/connectors/messaging-services/README.md)
       * [Kafka](install/metadata-ingestion/connectors/messaging-services/kafka.md)
-  * [Scheduler](install/metadata-ingestion/scheduler.md)
+    * [Processors](install/metadata-ingestion/connectors/processors/README.md)
+      * [PII](install/metadata-ingestion/connectors/processors/pii.md)
   * [Airflow](install/metadata-ingestion/airflow.md)
 
 ## Open Source Community
@@ -112,4 +113,3 @@
     * [BulkSink](open-source-community/developer/build-a-connector/bulksink.md)
   * [Run Integration Tests](open-source-community/developer/run-integration-tests.md)
   * [UX Style Guide](open-source-community/developer/ux-style-guide.md)
-

docs/install/enable-security/google-sso/create-ingestion-service-account.md

@@ -28,7 +28,7 @@ description: This is a guide to create ingestion bot service account.
 Make sure the Service Account Id is **ingestion-bot** and click on **CREATE AND CONTINUE**. If you chose a different Service Account Id, add it to the default bots list in [Configure OpenMetadata Server](https://github.com/StreamlineData/catalog/tree/3d53fa7c645ea55f846b06d0210ac63f8c38463f/docs/install/install/google-catalog-config.md)
 {% endhint %}
 
-![](../../../.gitbook/assets/ingestion-bot-service-account.png)
+![](../../../../.gitbook/assets/ingestion-bot-service-account.png)
 
 * Click on **Select a role** and give the **Owner** role. Then click **Continue.**
 
@@ -71,4 +71,3 @@ Make sure the Service Account Id is **ingestion-bot** and click on **CREATE AND
 ![Alt text](https://user-images.githubusercontent.com/83201188/125935663-d047f025-ae85-4cc7-9900-632586cc31be.png)
 
 * **The private-key/service-account JSON file will be downloaded**
-

docs/install/enable-security/google-sso/google-server-creds.md

@@ -3,9 +3,9 @@
 ## Step 1: Create the account
 
 * Go to [Create Google Cloud Account](https://console.cloud.google.com)
-* Click on `Create Project`
+*   Click on `Create Project`
 
-  ![Create Project](../../../.gitbook/assets/g-create-project%20%282%29%20%282%29%20%282%29%20%282%29%20%282%29%20%283%29%20%281%29.png)
+    ![Create Project](<../../../../.gitbook/assets/g-create-project (2) (2) (2) (2) (2) (3) (1).png>)
 
 ## Step 2: Create a new project
 
@@ -20,7 +20,7 @@
 
 * Select the project you created above and click on **APIs & Services** on the left-side panel.
 
-![OAuth Consent](../../../.gitbook/assets/g-project-oauth-consent.png)
+![OAuth Consent](../../../../.gitbook/assets/g-project-oauth-consent.png)
 
 * Click on the **OAuth Consent Screen** available on the left-hand side panel.
 * Provide the User Type. Chose **External** if you are testing otherwise chose **Internal**
@@ -63,7 +63,7 @@
 
 * After selecting the **Application Type**, name your project and give the authorized URIs
 
-![Create OAuth](../../../.gitbook/assets/g-create-oauth.png)
+![Create OAuth](../../../../.gitbook/assets/g-create-oauth.png)
 
 * Click **Create**
 * You will get the credentials
@@ -73,11 +73,10 @@
 ## Step 5: Where to find the credentials
 
 * Go to **Credentials**
-* Click on the **pencil icon \(Edit OAuth Client\)** on the right side of the screen
+* Click on the **pencil icon (Edit OAuth Client)** on the right side of the screen
 
 ![Alt text](https://user-images.githubusercontent.com/83201188/123220675-cad08b80-d4eb-11eb-9c46-2c7c81b02509.png)
 
 * You will find the **client Id** and **client secret** in the top right corner
 
 ![Alt text](https://user-images.githubusercontent.com/83201188/123220680-cc01b880-d4eb-11eb-9479-1379f69a063b.png)
-

docs/install/enable-security/okta-sso/okta-server-creds.md

@@ -14,7 +14,7 @@
 
 ![Alt text](https://user-images.githubusercontent.com/83201188/123376300-e600d100-d5a7-11eb-8ceb-e90b5e265ce9.png)
 
-* Click **Applications -> Applications** available on the left side of the panel.
+* Click **Applications -> Applications** available on the left side of the panel.
 
 ![Alt text](https://user-images.githubusercontent.com/83201188/123376308-e7ca9480-d5a7-11eb-84f8-9d8dce921c68.png)
 
@@ -34,18 +34,18 @@
 ![Alt text](https://user-images.githubusercontent.com/83201188/123376314-e9945800-d5a7-11eb-866b-5af5ca003f1e.png)
 
 * Once selected, click **Next**.
-* Provide the necessary input and click **Save**.
+* Select **Refresh token**, **Implicit(Hybrid)**. Once selected Implicit option, select **Allow ID token with implicit grant type**. In Refresh token behavior. Select **Use persistent token**. Provide the necessary input and click **Save**.
 
-![Alt text](https://user-images.githubusercontent.com/83201188/123376319-eac58500-d5a7-11eb-9167-a0d1cb8aedce.png)
+![](../../../../.gitbook/assets/screenshot-from-2021-09-20-14-14-14.png)
 
 * The app is now configured.
 
-![Alt text](https://user-images.githubusercontent.com/83201188/123376795-b43c3a00-d5a8-11eb-8a83-f146959f31c9.png)
+![](<../../../../.gitbook/assets/screenshot-from-2021-09-20-14-11-53 (1).png>)
 
 ## Step 4: Where to find the credentials
 
 * Once the app is configured, now the **Client Id** and **Client secret** can be used.
-* You can also go to **Application -&gt; Application** like in step 2.
+* You can also go to **Application -> Application** like in step 2.
 * You should be able to see your application in the list.
 
 ![Alt text](https://user-images.githubusercontent.com/83201188/123376803-b605fd80-d5a8-11eb-94ce-42f38dca99a1.png)
@@ -53,7 +53,7 @@
 * Click on your application.
 * You will find your **Client id**, **Client secret,** and **Okta domain**.
 
-![Alt text](https://user-images.githubusercontent.com/83201188/123376795-b43c3a00-d5a8-11eb-8a83-f146959f31c9.png)
+![](../../../../.gitbook/assets/screenshot-from-2021-09-20-14-59-07.png)
 
 * To get your **issuer** and **audience** click on **Sign On** available on top of the form.
 
@@ -61,9 +61,8 @@
 
 ## Step 5: Adding the details in openmetadata-security.yaml
 
-* Once the **Client Id**, **Client secret**, **issuer,** and **audience** are generated.
+*   Once the **Client Id**, **Client secret**, **issuer,** and **audience** are generated.
 
-  Add those details in the openmetadata-security.yaml file in the respective field.
+    Add those details in the openmetadata-security.yaml file in the respective field.
 
 ![Alt text](https://user-images.githubusercontent.com/83201188/123380400-054e2d00-d5ad-11eb-9937-2813f69cb268.png)
-

docs/install/metadata-ingestion/README.md

@@ -1,11 +1,10 @@
 # Metadata Ingestion
 
-OpenMetadata Ingestion is a simple framework to build connectors and ingest metadata of various systems through [OpenMetadata APIs](https://docs.open-metadata.org/openmetadata-apis/apis). It could be used in an orchestration framework\(e.g. Apache Airflow\) to ingest metadata.
+OpenMetadata Ingestion is a simple framework to build connectors and ingest metadata of various systems through [OpenMetadata APIs](../../openmetadata-apis/apis/). It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata.
 
 ## Guides
 
-* [Ingest Sample Data](https://docs.open-metadata.org/install/metadata-ingestion/ingest-sample-data)
-* [Explore Connectors & Install](https://docs.open-metadata.org/install/metadata-ingestion/connectors/)
-* [Run Simple Scheduler](https://docs.open-metadata.org/install/metadata-ingestion/scheduler)
-* [Configure Airflow](https://docs.open-metadata.org/install/metadata-ingestion/airflow)
+* [Ingest Sample Data](ingest-sample-data.md)
+* [Explore Connectors & Install](connectors/)
+* [Configure Airflow](airflow.md)
 

docs/install/metadata-ingestion/airflow.md

@@ -2,9 +2,11 @@
 
 We highly recommend using Airflow or similar schedulers to run Metadata Connectors. Below is the sample code example you can refer to integrate with Airflow
 
-## Airflow Example for Hive
+## Airflow Example for Sample Data
 
 ```python
+import pathlib
+import json
 from datetime import timedelta
 from airflow import DAG
 
@@ -25,22 +27,44 @@ default_args = {
     "execution_timeout": timedelta(minutes=60),
 }
 
+config = """
+{
+  "source": {
+    "type": "sample-data",
+    "config": {
+      "sample_data_folder": "./examples/sample_data"
+    }
+  },
+  "sink": {
+    "type": "metadata-rest",
+    "config": {}
+  },
+  "metadata_server": {
+    "type": "metadata-server",
+    "config": {
+      "api_endpoint": "http://localhost:8585/api",
+      "auth_provider_type": "no-auth"
+    }
+  }
+}
+"""
 
 def metadata_ingestion_workflow():
-    config = load_config_file("examples/workflows/hive.json")
-    workflow = Workflow.create(config)
-    workflow.run()
+    workflow_config = json.loads(config)
+    workflow = Workflow.create(workflow_config)
+    workflow.execute()
     workflow.raise_from_status()
     workflow.print_status()
     workflow.stop()
 
 
 with DAG(
-    "hive_metadata_ingestion_workflow"
+    "sample_data",
     default_args=default_args,
     description="An example DAG which runs a OpenMetadata ingestion workflow",
-    schedule_interval=timedelta(days=1),
-    start_date=days_ago(30),
+    start_date=days_ago(1),
+    is_paused_upon_creation=False,
+    schedule_interval='*/5 * * * *', 
     catchup=False,
 ) as dag:
     ingest_task = PythonOperator(
@@ -53,13 +77,12 @@ we are using a python method like below
 
 ```python
 def metadata_ingestion_workflow():
-    config = load_config_file("examples/workflows/hive.json")
-    workflow = Workflow.create(config)
-    workflow.run()
+    workflow_config = json.loads(config)
+    workflow = Workflow.create(workflow_config)
+    workflow.execute()
     workflow.raise_from_status()
     workflow.print_status()
-    workflow.stop()
+    workflow.stop
 ```
 
-Create a Workflow instance and pass a hive configuration which will read metadata from Hive and ingest it into the OpenMetadata Server. You can customize this configuration or add different connectors please refer to our [examples](https://github.com/open-metadata/OpenMetadata/tree/main/ingestion/examples/workflows) and refer to [Connectors](connectors/).
-
+Create a Workflow instance and pass a sample-data configuration which will read metadata from Json files and ingest it into the OpenMetadata Server. You can customize this configuration or add different connectors please refer to our [examples](https://github.com/open-metadata/OpenMetadata/tree/main/ingestion/examples/workflows) and refer to [Connectors](connectors/).

docs/install/metadata-ingestion/connectors/dashboard-services/README.md

@@ -2,8 +2,14 @@
 
 OpenMetadata supports connectors to some popular dashboard services. We will continue as a community to provide more integrations to get all your metadata into a central repository.
 
-{% page-ref page="looker.md" %}
+{% content-ref url="looker.md" %}
+[looker.md](looker.md)
+{% endcontent-ref %}
 
-{% page-ref page="superset.md" %}
+{% content-ref url="superset.md" %}
+[superset.md](superset.md)
+{% endcontent-ref %}
 
-{% page-ref page="tableau.md" %}
+{% content-ref url="tableau.md" %}
+[tableau.md](tableau.md)
+{% endcontent-ref %}

docs/install/metadata-ingestion/connectors/database-services/README.md

@@ -2,29 +2,61 @@
 
 OpenMetadata supports connectors to some popular data sources. We will continue as a community to provide more integrations to get all your metadata into a central repository.
 
-{% page-ref page="athena.md" %}
+{% content-ref url="athena.md" %}
+[athena.md](athena.md)
+{% endcontent-ref %}
 
-{% page-ref page="bigquery.md" %}
+{% content-ref url="bigquery.md" %}
+[bigquery.md](bigquery.md)
+{% endcontent-ref %}
 
-{% page-ref page="elastic-search.md" %}
+{% content-ref url="elastic-search.md" %}
+[elastic-search.md](elastic-search.md)
+{% endcontent-ref %}
 
-{% page-ref page="hive.md" %}
+{% content-ref url="mssql.md" %}
+[mssql.md](mssql.md)
+{% endcontent-ref %}
 
-{% page-ref page="mssql.md" %}
+{% content-ref url="mysql.md" %}
+[mysql.md](mysql.md)
+{% endcontent-ref %}
 
-{% page-ref page="mysql.md" %}
+{% content-ref url="hive.md" %}
+[hive.md](hive.md)
+{% endcontent-ref %}
 
-{% page-ref page="oracle.md" %}
+{% content-ref url="oracle.md" %}
+[oracle.md](oracle.md)
+{% endcontent-ref %}
 
-{% page-ref page="postgres.md" %}
+{% content-ref url="postgres.md" %}
+[postgres.md](postgres.md)
+{% endcontent-ref %}
 
-{% page-ref page="redshift-usage.md" %}
+{% content-ref url="redshift.md" %}
+[redshift.md](redshift.md)
+{% endcontent-ref %}
 
-{% page-ref page="salesforce.md" %}
+{% content-ref url="redshift-usage.md" %}
+[redshift-usage.md](redshift-usage.md)
+{% endcontent-ref %}
 
-{% page-ref page="snowflake.md" %}
+{% content-ref url="salesforce.md" %}
+[salesforce.md](salesforce.md)
+{% endcontent-ref %}
 
-{% page-ref page="snowflake-usage.md" %}
+{% content-ref url="snowflake.md" %}
+[snowflake.md](snowflake.md)
+{% endcontent-ref %}
+
+{% content-ref url="snowflake-usage.md" %}
+[snowflake-usage.md](snowflake-usage.md)
+{% endcontent-ref %}
+
+{% content-ref url="vertica.md" %}
+[vertica.md](vertica.md)
+{% endcontent-ref %}
 
 {% page-ref page="trino.md" %}
 

docs/install/metadata-ingestion/connectors/database-services/elastic-search.md

@@ -33,7 +33,7 @@ metadata ingest -c ./pipelines/metadata_to_es.json
 
 ### Configuration
 
-{% code title="metadata\_to\_es.json" %}
+{% code title="metadata_to_es.json" %}
 ```javascript
 {
   "source": {
@@ -48,9 +48,9 @@ metadata ingest -c ./pipelines/metadata_to_es.json
 
 Below is the configuration to publish Elastic Search data into the OpenMetadata service.
 
-Add Optionally `file` stage and `elasticsearch` bulk\_sink along with `metadata-server` config
+Add Optionally `file` stage and `elasticsearch` bulk_sink along with `metadata-server` config
 
-{% code title="metadata\_to\_es.json" %}
+{% code title="metadata_to_es.json" %}
 ```javascript
 {
   "source": {
@@ -89,4 +89,3 @@ Add Optionally `file` stage and `elasticsearch` bulk\_sink along with `metadata-
 
 ```
 {% endcode %}
-

docs/install/metadata-ingestion/connectors/database-services/redshift-usage.md

@@ -31,7 +31,7 @@ metadata ingest -c ./examples/workflows/redshift_usage.json
 
 ### Configuration
 
-{% code title="redshift\_usage.json" %}
+{% code title="redshift_usage.json" %}
 ```javascript
 {
   "source": {
@@ -52,16 +52,16 @@ metadata ingest -c ./examples/workflows/redshift_usage.json
 
 1. **username** - pass the Redshift username. We recommend creating a user with read-only permissions to all the databases in your Redshift installation
 2. **password** - password for the username
-3. **service\_name** - Service Name for this Redshift cluster. If you added Redshift cluster through OpenMetadata UI, make sure the service name matches the same.
-4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata
+3. **service_name** - Service Name for this Redshift cluster. If you added the Redshift cluster through OpenMetadata UI, make sure the service name matches the same.
+4. **filter_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata
 
 ## Publish to OpenMetadata
 
 Below is the configuration to publish Redshift Usage data into the OpenMetadata service.
 
-Add optionally`query-parser` processor, `table-usage` stage and `metadata-usage` bulk\_sink along with `metadata-server` config
+Add optionally`query-parser` processor, `table-usage` stage and `metadata-usage` bulk_sink along with `metadata-server` config
 
-{% code title="redshift\_usage.json" %}
+{% code title="redshift_usage.json" %}
 ```javascript
 {
   "source": {
@@ -112,4 +112,3 @@ Add optionally`query-parser` processor, `table-usage` stage and `metadata-usage`
 
 ```
 {% endcode %}
-

docs/install/metadata-ingestion/connectors/database-services/snowflake-usage.md

@@ -31,7 +31,7 @@ metadata ingest -c ./examples/workflows/snowflake_usage.json
 
 ### Configuration
 
-{% code title="snowflake\_usage.json" %}
+{% code title="snowflake_usage.json" %}
 ```javascript
 {
   "source": {
@@ -51,17 +51,17 @@ metadata ingest -c ./examples/workflows/snowflake_usage.json
 
 1. **username** - pass the Snowflake username.
 2. **password** - the password for the Snowflake username.
-3. **service\_name** - Service Name for this Snowflake cluster. If you added the Snowflake cluster through OpenMetadata UI, make sure the service name matches the same.
-4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata.
+3. **service_name** - Service Name for this Snowflake cluster. If you added the Snowflake cluster through OpenMetadata UI, make sure the service name matches the same.
+4. **filter_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata.
 5. **database -** Database name from where data is to be fetched.
 
 ### Publish to OpenMetadata
 
 Below is the configuration to publish Snowflake Usage data into the OpenMetadata service.
 
-Add Optionally `query-parser` processor, `table-usage` stage  and`metadata-usage` bulk\_sink along with `metadata-server` config
+Add Optionally `query-parser` processor, `table-usage` stage and`metadata-usage` bulk_sink along with `metadata-server` config
 
-{% code title="snowflake\_usage.json" %}
+{% code title="snowflake_usage.json" %}
 ```javascript
 {
   "source": {
@@ -112,4 +112,3 @@ Add Optionally `query-parser` processor, `table-usage` stage  and`metadata-usage
 
 ```
 {% endcode %}
-

docs/install/metadata-ingestion/connectors/database-services/vertica.md

@@ -49,11 +49,10 @@ metadata ingest -c ./examples/workflows/vertica.json
 ```
 {% endcode %}
 
-1. **username** - pass the Vertica username.
-2. **password** - the password for the Vertica username.
-3. **service\_name** - Service Name for this Vertica cluster. If you added the Vertica cluster through OpenMetadata UI, make sure the service name matches the same.
-4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata.
-5. **database -** Database name from where data is to be fetched.
+1. **username **- pass the Vertica username.
+2. **password** - password for the username.
+3. **service_name** - Service Name for this Vertica cluster. If you added Vertica cluster through OpenMetadata UI, make sure the service name matches the same.
+4. **filter_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata
 
 ### Publish to OpenMetadata
 

docs/install/metadata-ingestion/connectors/messaging-services/kafka.md

@@ -50,8 +50,8 @@ metadata ingest -c ./pipelines/confluent_kafka.json
 ```
 {% endcode %}
 
-1. **service\_name** - Service Name for this Kafka cluster. If you added Kafka cluster through OpenMetadata UI, make sure the service name matches the same.
-2. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata
+1. **service_name** - Service Name for this Kafka cluster. If you added Kafka cluster through OpenMetadata UI, make sure the service name matches the same.
+2. **filter_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata
 
 ## Publish to OpenMetadata
 
@@ -95,4 +95,3 @@ Add optionally `pii` processor and `metadata-rest-tables` sink along with `metad
 }
 ```
 {% endcode %}
-

docs/install/metadata-ingestion/scheduler.md

@@ -1,65 +0,0 @@
----
-description: >-
-  This guide will help install Simple Scheduler and schedule connectors for
-  ingestion.
----
-
-# Scheduler
-
-{% hint style="info" %}
-**Prerequisites**
-
-OpenMetadata is built using Java, DropWizard, Jetty, and MySQL.
-
-1. Python 3.7 or above
-2. Create an env
-
-   ```bash
-   python3 -m venv env
-   ```
-
-3. Activating the environment
-
-   ```bash
-   source env/bin/activate
-   ```
-{% endhint %}
-
-## Install dependencies
-
-```text
-cd ingestion
-python3 -m venv env
-source env/bin/activate
-pip install '.[scheduler]'
-python ingestion_scheduler/scheduler.py
-```
-
-{% hint style="warning" %}
-**Note:**
-
-Different Connectors require different dependencies, please go through [Connectors](https://docs.open-metadata.org/install/metadata-ingestion/connectors) Documentation install dependencies as needed.
-{% endhint %}
-
-## Scheduler UI
-
-### Main Page
-
-Loads all the Json connectors inside the pipeline directory as cron jobs.
-
-![](../../.gitbook/assets/screenshot-from-2021-07-26-21-08-17%20%281%29%20%282%29%20%282%29%20%282%29%20%283%29%20%284%29%20%284%29%20%285%29%20%283%29%20%281%29%20%285%29.png)
-
-### Custom run a job
-
-* Click on the **Custom Run** button under the **Action** column of the jobs.
-* Click on **Run.**
-* The Job will start running the ingestion.
-
-![After Clicking 'Custom Run' Button](../../.gitbook/assets/screenshot-from-2021-07-26-21-08-30.png)
-
-**Status of an executed job**
-
-* Click on the **Executions** tab under Navigation Bar.
-
-![Status of the executions](../../.gitbook/assets/screenshot-from-2021-07-26-23-57-46.png)
-

docs/install/run-openmetadata.md

@@ -13,8 +13,8 @@ description: >-
 {% hint style="info" %}
 **Prerequisites**
 
-* Docker >= 20.10.x
-* Minimum allocated memory to Docker >= 4GB  \(Preferences -> Advanced -> Resources\)
+* Docker >= 20.10.x
+* Minimum allocated memory to Docker >= 4GB  (Preferences -> Advanced -> Resources)
 {% endhint %}
 
 ```bash
@@ -28,9 +28,9 @@ docker-compose up
 1. Docker for OpenMetadata will depend on Mysql Container to be up, It may take few seconds to run.
 2. Once OpenMetadata UI is accessible, Go to Scheduler UI -[ http://localhost:7777](http://localhost:7777), to invoke the pipelines to ingest data. 
 
-![Scheduler UI \(http://localhost:7777\)](../.gitbook/assets/localhost_7777_.png)
+![Scheduler UI (http://localhost:7777)](../../.gitbook/assets/localhost\_7777\_.png)
 
-![Invoking a Pipeline for Ingestion](../.gitbook/assets/localhost_7777_-1-.png)
+![Invoking a Pipeline for Ingestion](../../.gitbook/assets/localhost\_7777\_-1-.png)
 
 The above command brings up all the necessary services
 
@@ -105,45 +105,41 @@ cd openmetadata-0.3.0-SNAPSHOT
 #### macOS
 
 1. Setup Database
-   * Install MySQL
+   *   Install MySQL
 
-     ```text
-      brew install mysql
-     ```
+       ```
+        brew install mysql
+       ```
+   *   Configure MySQL
 
-   * Configure MySQL
+       ```
+       mysqladmin -u root password 'yourpassword'
+       mysql -u root -p
+       ```
+   *   Setup Database
 
-     ```text
-     mysqladmin -u root password 'yourpassword'
-     mysql -u root -p
-     ```
+       ```
+       mysql -u root -p
+       CREATE DATABASE openmetadata_db;
+       CREATE USER 'openmetadata_user'@'localhost' IDENTIFIED BY 'openmetadata_password';
+       GRANT ALL PRIVILEGES ON openmetadata_db.* TO 'openmetadata_user'@'localhost' WITH GRANT OPTION;
+       commit;
+       ```
+2.  Run bootstrap scripts to initialize the database and tables
 
-   * Setup Database
+    ```
+       cd openmetadata-0.4.0
+       ./bootstrap/bootstrap_storage.sh migrate
+    ```
+3.  Start the OpenMetadata Server
 
-     ```text
-     mysql -u root -p
-     CREATE DATABASE openmetadata_db;
-     CREATE USER 'openmetadata_user'@'localhost' IDENTIFIED BY 'openmetadata_password';
-     GRANT ALL PRIVILEGES ON openmetadata_db.* TO 'openmetadata_user'@'localhost' WITH GRANT OPTION;
-     commit;
-     ```
-2. Run bootstrap scripts to initialize the database and tables
-
-   ```text
-      cd openmetadata-0.3.0
-      ./bootstrap/bootstrap_storage.sh migrate
-   ```
-
-3. Start the OpenMetadata Server
-
-   ```text
-      cd openmetadata-0.3.0 
-      ./bin/openmetadata.sh start
-   ```
+    ```
+       cd openmetadata-0.4.0 
+       ./bin/openmetadata.sh start
+    ```
 
 ### Ingest Sample Data
 
 Previous steps start OpenMetadata server. To start using it we need to run ElasticSearch and ingest sample metadata. Please follow the below guide:
 
 [Ingest Sample Data](metadata-ingestion/ingest-sample-data.md)
-

docs/open-source-community/developer/build-a-connector/README.md

@@ -16,7 +16,7 @@ A workflow consists of [Source](source.md), [Processor](processor.md) and [Sink]
 
 Workflow execution happens in serial fashion.
 
-1. **Workflow** runs the **source** component first.  The **source** retrieves a record from external sources and emits the record downstream.
+1. The** Workflow** runs the **source** component first.  The **source** retrieves a record from external sources and emits the record downstream.
 2. If the **processor** component is configured, the **workflow** sends the record to the **processor** next.
 3. There can be multiple **processor** components attached to the **workflow**.  The **workflow** passes a record to each **processor** in the order they are configured.
 4. Once a **processor** is finished, it sends the modified record to **sink**.
@@ -24,13 +24,22 @@ Workflow execution happens in serial fashion.
 
 In the cases where we need aggregation over the records, we can use **stage** to write to a file or other store. Use the file written to in **stage** and pass it to **bulksink** to publish to external services such as **openmetadata** or **elasticsearch**.
 
-{% page-ref page="source.md" %}
+{% content-ref url="source.md" %}
+[source.md](source.md)
+{% endcontent-ref %}
 
-{% page-ref page="processor.md" %}
+{% content-ref url="processor.md" %}
+[processor.md](processor.md)
+{% endcontent-ref %}
 
-{% page-ref page="sink.md" %}
+{% content-ref url="sink.md" %}
+[sink.md](sink.md)
+{% endcontent-ref %}
 
-{% page-ref page="stage.md" %}
-
-{% page-ref page="bulksink.md" %}
+{% content-ref url="stage.md" %}
+[stage.md](stage.md)
+{% endcontent-ref %}
 
+{% content-ref url="bulksink.md" %}
+[bulksink.md](bulksink.md)
+{% endcontent-ref %}

docs/open-source-community/developer/build-code-run-tests.md

@@ -12,11 +12,11 @@ The following commands must be run from the top-level directory.
 
 If you wish to skip the unit tests you can do this by adding `-DskipTests` to the command line.
 
-## Create a distribution \(packaging\)
+## Create a distribution (packaging)
 
 You can create a _distribution_ as follows.
 
-```text
+```
 $ mvn clean install
 
 # Create the binary distribution.
@@ -25,7 +25,7 @@ $ cd dist && mvn package
 
 The binaries will be created at:
 
-```text
+```
 dist/target/open-metadata-<version>.pom
 dist/target/open-metadata-<version>.tar.gz
 ```
@@ -34,40 +34,39 @@ dist/target/open-metadata-<version>.tar.gz
 
 Add a new Run/Debug configuration like the below screenshot.
 
-![Intellij Run Configuration](../../.gitbook/assets/image%20%281%29.png)
+![Intellij Run Configuration](<../../../.gitbook/assets/image (1).png>)
 
 ## Add missing dependency
 
 Right-click on catalog-rest-service
 
-![](../../.gitbook/assets/image-1-.png)
+![](../../../.gitbook/assets/image-1-.png)
 
 Click on "Open Module Settings"
 
-![](../../.gitbook/assets/image-2-.png)
+![](../../../.gitbook/assets/image-2-.png)
 
 Go to "Dependencies"
 
-![](../../.gitbook/assets/image-3-.png)
+![](../../../.gitbook/assets/image-3-.png)
 
 Click “+” at the bottom of the dialog box and click "Add"
 
-![](../../.gitbook/assets/image-4-.png)
+![](../../../.gitbook/assets/image-4-.png)
 
 Click on Library
 
-![](../../.gitbook/assets/image-5-.png)
+![](../../../.gitbook/assets/image-5-.png)
 
 In that list look for "jersey-client:2.25.1"
 
-![](../../.gitbook/assets/image-6-.png)
+![](../../../.gitbook/assets/image-6-.png)
 
 Select it and click "OK". Now run/debug the application.
 
 ## Coding Style
 
-1. [Refer to coding guidelines](https://github.com/open-metadata/OpenMetadata/blob/main/docs/open-source-community/developer/coding-style.md)
-2. Configure IntelliJ to disable the \[wild-card imports\]
-
-   \([https://www.jetbrains.com/help/idea/creating-and-optimizing-imports.html\#disable-wildcard-imports](https://www.jetbrains.com/help/idea/creating-and-optimizing-imports.html#disable-wildcard-imports)\)
+1. [Refer to coding guidelines](coding-style.md)
+2.  Configure IntelliJ to disable the \[wild-card imports]
 
+    ([https://www.jetbrains.com/help/idea/creating-and-optimizing-imports.html#disable-wildcard-imports](https://www.jetbrains.com/help/idea/creating-and-optimizing-imports.html#disable-wildcard-imports))

docs/open-source-community/developer/how-to-contribute.md

@@ -8,13 +8,13 @@ We ❤️ all contributions, big and small!
 
 Look for issues under [github/issues tab](https://github.com/open-metadata/OpenMetadata/issues) . If you have a feature request or found a bug please file an issue. This will help us track and will help community overall as well.
 
-![./images/new-issue.png](../../.gitbook/assets/new-issue.png)
+![./images/new-issue.png](../../../.gitbook/assets/new-issue.png)
 
 ## Fork Github project
 
 OpenMetadata Github repository can be accessed here [https://github.com/open-metadata/OpenMetadata](https://github.com/open-metadata/OpenMetadata) .
 
-![./images/fork-github.png](../../.gitbook/assets/fork-github%20%281%29.png)
+![./images/fork-github.png](<../../../.gitbook/assets/fork-github (1).png>)
 
 Create a local clone of your fork
 
@@ -49,15 +49,13 @@ git push origin HEAD:refs/heads/issue-200
 ## Open a PR
 
 1. Go to [https://github.com/open-metadata/OpenMetadata/pulls](https://github.com/open-metadata/OpenMetadata/pulls)
-2. It should show an option to open a pull request. ![./images/pull-request-1.png](../../.gitbook/assets/pull-request-1.png)
-3. If not, click on "New Pull request"
+2. It should show an option to open a pull request. ![./images/pull-request-1.png](../../../.gitbook/assets/pull-request-1.png)
+3.  If not, click on "New Pull request"
 
-   ![./images/pull-request.png](../../.gitbook/assets/pull-request.png)
-
-4. Select your fork repository and branch ![./images/pull-request-2.png](../../.gitbook/assets/pull-request-2.png)
+    ![./images/pull-request.png](../../../.gitbook/assets/pull-request.png)
+4. Select your fork repository and branch ![./images/pull-request-2.png](../../../.gitbook/assets/pull-request-2.png)
 5. Click "Create pull request"
 
 ## We are here to help
 
-Please reach out to us anytime you need any help. [Slack](https://slack.open-metadata.org/) would be fastest way to get a response.
-
+Please reach out to us anytime you need any help. [Slack](https://slack.open-metadata.org) would be fastest way to get a response.

docs/openmetadata-apis/schemas/entities/README.md

@@ -2,7 +2,7 @@
 
 OpenMetadata has the following high-level core entities and relationships:
 
-![](../../../.gitbook/assets/openmetadata-entities-relationship.png)
+![](../../../../.gitbook/assets/openmetadata-entities-relationship.png)
 
 1. **Data Assets** - These entities represent data, such as databases, tables, and topics, and assets created using data, such as Dashboards, Reports, Metrics, and ML Features. It also includes entities such as Pipelines that are used for creating data assets.  
 2. **Services** - Services represent platforms and services used for storing and processing data. It includes Online Data Stores, Data Warehouses, ETL tools, Dashboard services, etc.  
@@ -11,40 +11,67 @@ OpenMetadata has the following high-level core entities and relationships:
 5. **Glossary and Tags** - Entities for defining business glossary that includes hierarchical tags.
 
 ## List of Schema Entities
-{% page-ref page="bot.md" %}
 
-{% page-ref page="chart.md" %}
+{% content-ref url="bot.md" %}
+[bot.md](bot.md)
+{% endcontent-ref %}
 
-{% page-ref page="dashboard.md" %}
+{% content-ref url="chart.md" %}
+[chart.md](chart.md)
+{% endcontent-ref %}
 
-{% page-ref page="dashboardservice.md" %}
+{% content-ref url="dashboard.md" %}
+[dashboard.md](dashboard.md)
+{% endcontent-ref %}
 
-{% page-ref page="database.md" %}
+{% content-ref url="dashboardservice.md" %}
+[dashboardservice.md](dashboardservice.md)
+{% endcontent-ref %}
 
-{% page-ref page="databaseservice.md" %}
+{% content-ref url="database.md" %}
+[database.md](database.md)
+{% endcontent-ref %}
 
-{% page-ref page="messagingservice.md" %}
+{% content-ref url="databaseservice.md" %}
+[databaseservice.md](databaseservice.md)
+{% endcontent-ref %}
 
-{% page-ref page="thread.md" %}
+{% content-ref url="messagingservice.md" %}
+[messagingservice.md](messagingservice.md)
+{% endcontent-ref %}
 
-{% page-ref page="metrics.md" %}
+{% content-ref url="thread.md" %}
+[thread.md](thread.md)
+{% endcontent-ref %}
 
-{% page-ref page="model.md" %}
+{% content-ref url="metrics.md" %}
+[metrics.md](metrics.md)
+{% endcontent-ref %}
 
-{% page-ref page="pipeline.md" %}
+{% content-ref url="pipeline.md" %}
+[pipeline.md](pipeline.md)
+{% endcontent-ref %}
 
-{% page-ref page="pipelineservice.md" %}
+{% content-ref url="report.md" %}
+[report.md](report.md)
+{% endcontent-ref %}
 
-{% page-ref page="report.md" %}
+{% content-ref url="table.md" %}
+[table.md](table.md)
+{% endcontent-ref %}
 
-{% page-ref page="table.md" %}
+{% content-ref url="team.md" %}
+[team.md](team.md)
+{% endcontent-ref %}
 
-{% page-ref page="tagcategory.md" %}
+{% content-ref url="topic.md" %}
+[topic.md](topic.md)
+{% endcontent-ref %}
 
-{% page-ref page="task.md" %}
+{% content-ref url="user.md" %}
+[user.md](user.md)
+{% endcontent-ref %}
 
-{% page-ref page="team.md" %}
-
-{% page-ref page="topic.md" %}
-
-{% page-ref page="user.md" %}
+{% content-ref url="tagcategory.md" %}
+[tagcategory.md](tagcategory.md)
+{% endcontent-ref %}

docs/openmetadata-apis/schemas/entities/bot.md

@@ -2,7 +2,7 @@
 
 This schema defines Bot entity. A bot automates tasks, such as adding description, identifying the importance of data. It runs as a special user in the system.
 
-**$id: https://open-metadata.org/schema/entity/bots.json**
+**$id: **[https://open-metadata.org/schema/entity/bots.json](https://open-metadata.org/schema/entity/bots.json)
 
 Type: `object`
 
@@ -24,4 +24,21 @@ Type: `object`
    - Link to the resource corresponding to this bot.
    - $ref: [../type/basic.json#/definitions/href](../types/basic.md#href)
 
-_This document was updated on: Thursday, October 14, 2021_
+* **id**
+  * Unique identifier of a bot instance.
+  * $ref: [../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name**
+  * Name of the bot.
+  * Type: `string`
+  * Length: between 1 and 64
+* **displayName**
+  * Name used for display purposes. Example 'FirstName LastName'.
+  * Type: `string`
+* **description**
+  * Description of the bot.
+  * Type: `string`
+* **href**
+  * Link to the resource corresponding to this bot.
+  * $ref: [../type/basic.json#/definitions/href](../types/basic.md#href)
+
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/chart.md

@@ -2,7 +2,7 @@
 
 This schema defines the Chart entity. Charts are built using tables or sql queries by analyzing the data. Charts can be part of Dashboard.
 
-**$id: https://open-metadata.org/schema/entity/data/chart.json**
+**$id: **[https://open-metadata.org/schema/entity/data/chart.json](https://open-metadata.org/schema/entity/data/chart.json)
 
 Type: `object`
 
@@ -54,6 +54,52 @@ Type: `object`
    - Latest usage information for this database.
    - $ref: [../../type/usageDetails.json](../types/usagedetails.md)
 
+* **id** `required`
+  * Unique identifier that identifies a chart instance.
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name** `required`
+  * Name that identifies this Chart.
+  * Type: `string`
+  * Length: between 1 and 64
+* **displayName**
+  * Display Name that identifies this Chart. It could be a title or label from the source services.
+  * Type: `string`
+* **fullyQualifiedName**
+  * A unique name that identifies a dashboard in the format 'ServiceName.ChartName'.
+  * Type: `string`
+  * Length: between 1 and 64
+* **description**
+  * Description of the dashboard, what it is, and how to use it.
+  * Type: `string`
+* **chartType**
+  * $ref: [#/definitions/chartType](chart.md#charttype)
+* **chartUrl**
+  * Chart URL, pointing to its own Service URL.
+  * Type: `string`
+  * String format must be a "uri"
+* **href**
+  * Link to the resource corresponding to this entity.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
+* **owner**
+  * Owner of this dashboard.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **tables**
+  * Link to table used in this chart.
+  * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
+* **followers**
+  * Followers of this chart.
+  * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
+* **tags**
+  * Tags for this chart.
+  * Type: `array`
+    * **Items**
+    * $ref: [../../type/tagLabel.json](../types/taglabel.md)
+* **service** `required`
+  * Link to service where this dashboard is hosted in.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **usageSummary**
+  * Latest usage information for this database.
+  * $ref: [../../type/usageDetails.json](../types/usagedetails.md)
 
 ## Type definitions in this schema
 ### chartType
@@ -73,5 +119,4 @@ Type: `object`
    10. _"Other"_
 
 
-
-_This document was updated on: Thursday, October 14, 2021_
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/dashboard.md

@@ -2,7 +2,7 @@
 
 This schema defines the Dashboard entity. Dashboards are computed from data and visually present data, metrics, and KPIs. They are updated in real-time and allow interactive data exploration.
 
-**$id: https://open-metadata.org/schema/entity/data/dashboard.json**
+**$id: **[https://open-metadata.org/schema/entity/data/dashboard.json](https://open-metadata.org/schema/entity/data/dashboard.json)
 
 Type: `object`
 
@@ -54,4 +54,51 @@ Type: `object`
    - Latest usage information for this database.
    - $ref: [../../type/usageDetails.json](../types/usagedetails.md)
 
-_This document was updated on: Thursday, October 14, 2021_
+* **id** `required`
+  * Unique identifier that identifies a dashboard instance.
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name** `required`
+  * Name that identifies this dashboard.
+  * Type: `string`
+  * Length: between 1 and 64
+* **displayName**
+  * Display Name that identifies this Dashboard. It could be a title or label from the source services.
+  * Type: `string`
+* **fullyQualifiedName**
+  * A unique name that identifies a dashboard in the format 'ServiceName.DashboardName'.
+  * Type: `string`
+  * Length: between 1 and 64
+* **description**
+  * Description of the dashboard, what it is, and how to use it.
+  * Type: `string`
+* **dashboardUrl**
+  * Dashboard URL.
+  * Type: `string`
+  * String format must be a "uri"
+* **charts**
+  * All the charts are included in this Dashboard.
+  * Type: `array`
+    * **Items**
+    * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **href**
+  * Link to the resource corresponding to this entity.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
+* **owner**
+  * Owner of this dashboard.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **followers**
+  * Followers of this dashboard.
+  * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
+* **tags**
+  * Tags for this dashboard.
+  * Type: `array`
+    * **Items**
+    * $ref: [../../type/tagLabel.json](../types/taglabel.md)
+* **service** `required`
+  * Link to service where this dashboard is hosted in.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **usageSummary**
+  * Latest usage information for this database.
+  * $ref: [../../type/usageDetails.json](../types/usagedetails.md)
+
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/dashboardservice.md

@@ -2,7 +2,7 @@
 
 This schema defines the Dashboard Service entity, such as Looker and Superset.
 
-**$id: https://open-metadata.org/schema/entity/services/dashboardService.json**
+**$id: **[https://open-metadata.org/schema/entity/services/dashboardService.json](https://open-metadata.org/schema/entity/services/dashboardService.json)
 
 Type: `object`
 
@@ -40,6 +40,35 @@ Type: `object`
    - Link to the resource corresponding to this dashboard service.
    - $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
 
+* **id** `required`
+  * Unique identifier of this dashboard service instance.
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name** `required`
+  * Name that identifies this dashboard service.
+  * Type: `string`
+  * Length: between 1 and 64
+* **serviceType** `required`
+  * Type of dashboard services such as Looker or Superset...
+  * $ref: [#/definitions/dashboardServiceType](dashboardservice.md#dashboardservicetype)
+* **description**
+  * Description of a dashboard service instance.
+  * Type: `string`
+* **dashboardUrl** `required`
+  * Dashboard Service URL. This will be used to make REST API calls to Dashboard Service.
+  * Type: `string`
+  * String format must be a "uri"
+* **username**
+  * Username to log in to Dashboard Service.
+  * Type: `string`
+* **password**
+  * Password to log in to Dashboard Service.
+  * Type: `string`
+* **ingestionSchedule**
+  * Schedule for running metadata ingestion jobs.
+  * $ref: [../../type/schedule.json](../types/schedule.md)
+* **href**
+  * Link to the resource corresponding to this messaging service.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
 
 ## Type definitions in this schema
 ### dashboardServiceType
@@ -53,5 +82,4 @@ Type: `object`
    4. _"Redash"_
 
 
-
-_This document was updated on: Thursday, October 14, 2021_
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/database.md

@@ -2,7 +2,7 @@
 
 This schema defines the Database entity. A database also referred to as Database Catalog is a collection of tables.
 
-**$id: https://open-metadata.org/schema/entity/data/database.json**
+**$id: **[https://open-metadata.org/schema/entity/data/database.json](https://open-metadata.org/schema/entity/data/database.json)
 
 Type: `object`
 
@@ -38,6 +38,33 @@ Type: `object`
    - References to tables in the database.
    - $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
 
+* **id**
+  * Unique identifier that identifies this database instance.
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name** `required`
+  * Name that identifies the database.
+  * $ref: [#/definitions/databaseName](database.md#databasename)
+* **fullyQualifiedName**
+  * Name that uniquely identifies a database in the format 'ServiceName.DatabaseName'.
+  * Type: `string`
+* **description**
+  * Description of the database instance.
+  * Type: `string`
+* **href**
+  * Link to the resource corresponding to this entity.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
+* **owner**
+  * Owner of this database.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **service** `required`
+  * Link to the database cluster/service where this database is hosted in.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **usageSummary**
+  * Latest usage information for this database.
+  * $ref: [../../type/usageDetails.json](../types/usagedetails.md)
+* **tables**
+  * References to tables in the database.
+  * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
 
 ## Type definitions in this schema
 ### databaseName
@@ -48,5 +75,4 @@ Type: `object`
  - Length: between 1 and 64
 
 
-
-_This document was updated on: Thursday, October 14, 2021_
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/databaseservice.md

@@ -2,7 +2,7 @@
 
 This schema defines the Database Service entity, such as MySQL, BigQuery, Redshift, Postgres, or Snowflake. Alternative terms such as Database Cluster, Database Server instance are also used for database service.
 
-**$id: https://open-metadata.org/schema/entity/services/databaseService.json**
+**$id: **[https://open-metadata.org/schema/entity/services/databaseService.json](https://open-metadata.org/schema/entity/services/databaseService.json)
 
 Type: `object`
 
@@ -33,24 +33,45 @@ Type: `object`
    - Schedule for running metadata ingestion jobs.
    - $ref: [../../type/schedule.json](../types/schedule.md)
 
+* **id** `required`
+  * Unique identifier of this database service instance.
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name** `required`
+  * Name that identifies this database service.
+  * Type: `string`
+  * Length: between 1 and 64
+* **serviceType** `required`
+  * Type of database service such as MySQL, BigQuery, Snowflake, Redshift, Postgres...
+  * $ref: [#/definitions/databaseServiceType](databaseservice.md#databaseservicetype)
+* **description**
+  * Description of a database service instance.
+  * Type: `string`
+* **href** `required`
+  * Link to the resource corresponding to this database service.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
+* **jdbc** `required`
+  * JDBC connection information.
+  * $ref: [../../type/jdbcConnection.json#/definitions/jdbcInfo](../types/jdbcconnection.md#jdbcinfo)
+* **ingestionSchedule**
+  * Schedule for running metadata ingestion jobs.
+  * $ref: [../../type/schedule.json](../types/schedule.md)
 
 ## Type definitions in this schema
 ### databaseServiceType
 
- - Type of database service such as MySQL, BigQuery, Snowflake, Redshift, Postgres...
- - Type: `string`
- - The value is restricted to the following: 
-   1. _"BigQuery"_
-   2. _"MySQL"_
-   3. _"Redshift"_
-   4. _"Snowflake"_
-   5. _"Postgres"_
-   6. _"MSSQL"_
-   7. _"Hive"_
-   8. _"Oracle"_
-   9. _"Athena"_
-   10. _"Presto"_
-   11. _"Trino"_
-   12. _"Vertica"_
+* Type of database service such as MySQL, BigQuery, Snowflake, Redshift, Postgres...
+* Type: `string`
+* The value is restricted to the following: 
+  1. _"BigQuery"_
+  2. _"MySQL"_
+  3. _"Redshift"_
+  4. _"Snowflake"_
+  5. _"Postgres"_
+  6. _"MSSQL"_
+  7. _"Hive"_
+  8. _"Oracle"_
+  9. _"Athena"_
+  10. _"Presto"_
+  11. _"Vertica"_
 
-_This document was updated on: Thursday, October 14, 2021_
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/messagingservice.md

@@ -2,7 +2,7 @@
 
 This schema defines the Messaging Service entity, such as Kafka and Pulsar.
 
-**$id: https://open-metadata.org/schema/entity/services/messagingService.json**
+**$id: **[https://open-metadata.org/schema/entity/services/messagingService.json](https://open-metadata.org/schema/entity/services/messagingService.json)
 
 Type: `object`
 
@@ -37,6 +37,32 @@ Type: `object`
    - Link to the resource corresponding to this messaging service.
    - $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
 
+* **id** `required`
+  * Unique identifier of this messaging service instance.
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name** `required`
+  * Name that identifies this messaging service.
+  * Type: `string`
+  * Length: between 1 and 64
+* **serviceType** `required`
+  * Type of messaging services such as Kafka or Pulsar...
+  * $ref: [#/definitions/messagingServiceType](messagingservice.md#messagingservicetype)
+* **description**
+  * Description of a messaging service instance.
+  * Type: `string`
+* **brokers** `required`
+  * Multiple bootstrap addresses for Kafka. Single proxy address for Pulsar.
+  * $ref: [#/definitions/brokers](messagingservice.md#brokers)
+* **schemaRegistry**
+  * Schema registry URL.
+  * Type: `string`
+  * String format must be a "uri"
+* **ingestionSchedule**
+  * Schedule for running metadata ingestion jobs.
+  * $ref: [../../type/schedule.json](../types/schedule.md)
+* **href**
+  * Link to the resource corresponding to this messaging service.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
 
 ## Type definitions in this schema
 ### messagingServiceType
@@ -50,9 +76,4 @@ Type: `object`
 
 ### brokers
 
- - Multiple bootstrap addresses for Kafka. Single proxy address for Pulsar.
- - Type: `array`
-   - **Items**    
-   - Type: `string`
-
-_This document was updated on: Thursday, October 14, 2021_
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/metrics.md

@@ -2,7 +2,7 @@
 
 This schema defines the Metrics entity. Metrics are measurements computed from data such as `Monthly Active Users`. Some of the metrics that measures used to determine performance against an objective are called KPIs or Key Performance Indicators, such as `User Retention`.
 
-**$id: https://open-metadata.org/schema/entity/data/metrics.json**
+**$id: **[https://open-metadata.org/schema/entity/data/metrics.json](https://open-metadata.org/schema/entity/data/metrics.json)
 
 Type: `object`
 
@@ -37,4 +37,31 @@ Type: `object`
    - Latest usage information for this database.
    - $ref: [../../type/usageDetails.json](../types/usagedetails.md)
 
-_This document was updated on: Thursday, October 14, 2021_
+* **id** `required`
+  * Unique identifier that identifies this metrics instance.
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name** `required`
+  * Name that identifies this metrics instance uniquely.
+  * Type: `string`
+  * Length: between 1 and 64
+* **fullyQualifiedName**
+  * A unique name that identifies a metric in the format 'ServiceName.MetricName'.
+  * Type: `string`
+  * Length: between 1 and 64
+* **description**
+  * Description of metrics instance, what it is, and how to use it.
+  * Type: `string`
+* **href**
+  * Link to the resource corresponding to this entity.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
+* **owner**
+  * Owner of these metrics.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **service** `required`
+  * Link to service where this metrics is hosted in.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **usageSummary**
+  * Latest usage information for this database.
+  * $ref: [../../type/usageDetails.json](../types/usagedetails.md)
+
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/pipeline.md

@@ -2,7 +2,7 @@
 
 This schema defines the Pipeline entity. A pipeline enables the flow of data from source to destination through a series of processing steps. ETL is a type of pipeline where the series of steps Extract, Transform and Load the data.
 
-**$id: https://open-metadata.org/schema/entity/data/pipeline.json**
+**$id: **[https://open-metadata.org/schema/entity/data/pipeline.json](https://open-metadata.org/schema/entity/data/pipeline.json)
 
 Type: `object`
 
@@ -60,4 +60,28 @@ Type: `object`
    - Link to service where this pipeline is hosted in.
    - $ref: [../../type/entityReference.json](../types/entityreference.md)
 
-_This document was updated on: Thursday, October 14, 2021_
+* **id** `required`
+  * Unique identifier that identifies a pipeline instance.
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name** `required`
+  * Name that identifies this pipeline instance uniquely.
+  * Type: `string`
+  * Length: between 1 and 64
+* **fullyQualifiedName**
+  * A unique name that identifies a pipeline in the format 'ServiceName.PipelineName'.
+  * Type: `string`
+  * Length: between 1 and 64
+* **description**
+  * Description of this pipeline.
+  * Type: `string`
+* **href**
+  * Link to the resource corresponding to this entity.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
+* **owner**
+  * Owner of this pipeline.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **service** `required`
+  * Link to service where this pipeline is hosted in.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/report.md

@@ -2,7 +2,7 @@
 
 This schema defines the Report entity. Reports are static information computed from data periodically that includes data in text, table, and visual form.
 
-**$id: https://open-metadata.org/schema/entity/data/report.json**
+**$id: **[https://open-metadata.org/schema/entity/data/report.json](https://open-metadata.org/schema/entity/data/report.json)
 
 Type: `object`
 
@@ -37,4 +37,31 @@ Type: `object`
    - Latest usage information for this database.
    - $ref: [../../type/usageDetails.json](../types/usagedetails.md)
 
-_This document was updated on: Thursday, October 14, 2021_
+* **id** `required`
+  * The Unique identifier that identifies this report.
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name** `required`
+  * Name that identifies this report instance uniquely.
+  * Type: `string`
+  * Length: between 1 and 64
+* **fullyQualifiedName**
+  * A unique name that identifies a report in the format 'ServiceName.ReportName'.
+  * Type: `string`
+  * Length: between 1 and 64
+* **description**
+  * Description of this report instance.
+  * Type: `string`
+* **href**
+  * Link to the resource corresponding to this report.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
+* **owner**
+  * Owner of this pipeline.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **service** `required`
+  * Link to service where this report is hosted in.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **usageSummary**
+  * Latest usage information for this database.
+  * $ref: [../../type/usageDetails.json](../types/usagedetails.md)
+
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/table.md

@@ -2,7 +2,7 @@
 
 This schema defines the Table entity. A Table organizes data in rows and columns and is defined by a Schema. OpenMetadata does not have a separate abstraction for Schema. Both Table and Schema are captured in this entity.
 
-**$id: https://open-metadata.org/schema/entity/data/table.json**
+**$id: **[https://open-metadata.org/schema/entity/data/table.json](https://open-metadata.org/schema/entity/data/table.json)
 
 Type: `object`
 
@@ -69,106 +69,157 @@ Type: `object`
      - **Items**
      - $ref: [#/definitions/tableProfile](#tableprofile)
 
+* **id** `required`
+  * Unique identifier of this table instance.
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name** `required`
+  * Name of a table. Expected to be unique within a database.
+  * $ref: [#/definitions/tableName](table.md#tablename)
+* **description**
+  * Description of a table.
+  * Type: `string`
+* **href**
+  * Link to this table resource.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
+* **tableType**
+  * $ref: [#/definitions/tableType](table.md#tabletype)
+* **fullyQualifiedName**
+  * Fully qualified name of a table in the form `serviceName.databaseName.tableName`.
+  * Type: `string`
+* **columns** `required`
+  * Columns in this table.
+  * Type: `array`
+    * **Items**
+    * $ref: [#/definitions/column](table.md#column)
+* **tableConstraints**
+  * Table constraints.
+  * Type: `array`
+    * **Items**
+    * $ref: [#/definitions/tableConstraint](table.md#tableconstraint)
+* **usageSummary**
+  * Latest usage information for this table.
+  * $ref: [../../type/usageDetails.json](../types/usagedetails.md)
+* **owner**
+  * Owner of this table.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **followers**
+  * Followers of this table.
+  * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
+* **database**
+  * Reference to Database that contains this table.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **viewDefinition**
+  * View Definition in SQL. Applies to TableType.View only.
+  * $ref: [../../type/basic.json#/definitions/sqlQuery](../types/basic.md#sqlquery)
+* **tags**
+  * Tags for this table.
+  * Type: `array`
+    * **Items**
+    * $ref: [../../type/tagLabel.json](../types/taglabel.md)
+* **joins**
+  * Details of other tables this table is frequently joined with.
+  * $ref: [#/definitions/tableJoins](table.md#tablejoins)
+* **sampleData**
+  * Sample data for a table.
+  * $ref: [#/definitions/tableData](table.md#tabledata)
+* **tableProfile**
+  * Data profile for a table.
+  * Type: `array`
+    * **Items**
+    * $ref: [#/definitions/tableProfile](table.md#tableprofile)
 
 ## Type definitions in this schema
 ### tableType
 
- - This schema defines the type used for describing different types of tables.
- - Type: `string`
- - The value is restricted to the following: 
-   1. _"Regular"_
-   2. _"External"_
-   3. _"View"_
-   4. _"SecureView"_
-   5. _"MaterializedView"_
+* This schema defines the type used for describing different types of tables.
+* Type: `string`
+* The value is restricted to the following: 
+  1. _"Regular"_
+  2. _"External"_
+  3. _"View"_
+  4. _"SecureView"_
+  5. _"MaterializedView"_
 
+### columnDataType
 
-### dataType
+* This enum defines the type of data stored in a column.
+* Type: `string`
+* The value is restricted to the following: 
+  1. _"NUMBER"_
+  2. _"TINYINT"_
+  3. _"SMALLINT"_
+  4. _"INT"_
+  5. _"BIGINT"_
+  6. _"FLOAT"_
+  7. _"DOUBLE"_
+  8. _"DECIMAL"_
+  9. _"NUMERIC"_
+  10. _"TIMESTAMP"_
+  11. _"TIME"_
+  12. _"DATE"_
+  13. _"DATETIME"_
+  14. _"INTERVAL"_
+  15. _"STRING"_
+  16. _"MEDIUMTEXT"_
+  17. _"TEXT"_
+  18. _"CHAR"_
+  19. _"VARCHAR"_
+  20. _"BOOLEAN"_
+  21. _"BINARY"_
+  22. _"VARBINARY"_
+  23. _"ARRAY"_
+  24. _"BLOB"_
+  25. _"LONGBLOB"_
+  26. _"MEDIUMBLOB"_
+  27. _"MAP"_
+  28. _"STRUCT"_
+  29. _"UNION"_
+  30. _"SET"_
+  31. _"GEOGRAPHY"_
+  32. _"ENUM"_
+  33. _"JSON"_
 
- - This enum defines the type of data stored in a column.
- - Type: `string`
- - The value is restricted to the following: 
-   1. _"NUMBER"_
-   2. _"TINYINT"_
-   3. _"SMALLINT"_
-   4. _"INT"_
-   5. _"BIGINT"_
-   6. _"BYTEINT"_
-   7. _"FLOAT"_
-   8. _"DOUBLE"_
-   9. _"DECIMAL"_
-   10. _"NUMERIC"_
-   11. _"TIMESTAMP"_
-   12. _"TIME"_
-   13. _"DATE"_
-   14. _"DATETIME"_
-   15. _"INTERVAL"_
-   16. _"STRING"_
-   17. _"MEDIUMTEXT"_
-   18. _"TEXT"_
-   19. _"CHAR"_
-   20. _"VARCHAR"_
-   21. _"BOOLEAN"_
-   22. _"BINARY"_
-   23. _"VARBINARY"_
-   24. _"ARRAY"_
-   25. _"BLOB"_
-   26. _"LONGBLOB"_
-   27. _"MEDIUMBLOB"_
-   28. _"MAP"_
-   29. _"STRUCT"_
-   30. _"UNION"_
-   31. _"SET"_
-   32. _"GEOGRAPHY"_
-   33. _"ENUM"_
-   34. _"JSON"_
-
-
-### constraint
-
- - This enum defines the type for column constraint.
- - Type: `string`
- - The value is restricted to the following: 
-   1. _"NULL"_
-   2. _"NOT_NULL"_
-   3. _"UNIQUE"_
-   4. _"PRIMARY_KEY"_
- - Default: _"NULL"_
+### columnConstraint
 
+* This enum defines the type for column constraint.
+* Type: `string`
+* The value is restricted to the following: 
+  1. _"NULL"_
+  2. _"NOT_NULL"_
+  3. _"UNIQUE"_
+  4. _"PRIMARY_KEY"_
+* Default: _"NULL"_
 
 ### tableConstraint
 
- - This enum defines the type for table constraint.
- - Type: `object`
- - **Properties**
-   - **constraintType**
-     - Type: `string`
-     - The value is restricted to the following: 
-       1. _"UNIQUE"_
-       2. _"PRIMARY_KEY"_
-       3. _"FOREIGN_KEY"_
-   - **columns**
-     - List of column names corresponding to the constraint.
-     - Type: `array`
-       - **Items**
-       - Type: `string`
-
+* This enum defines the type for table constraint.
+* Type: `object`
+* **Properties**
+  * **constraintType**
+    * Type: `string`
+    * The value is restricted to the following: 
+      1. _"UNIQUE"_
+      2. _"PRIMARY_KEY"_
+      3. _"FOREIGN_KEY"_
+  * **columns**
+    * List of column names corresponding to the constraint.
+    * Type: `array`
+      * **Items**
+      * Type: `string`
 
 ### columnName
 
- - Local name (not fully qualified name) of the column. ColumnName is `-` when the column is not named in struct dataType. For example, BigQuery supports struct with unnamed fields.
- - Type: `string`
- - The value must match this pattern: `^[^.]*$`
- - Length: between 1 and 64
-
+* Local name (not fully qualified name) of the column.
+* Type: `string`
+* The value must match this pattern: `^[^.]*$`
+* Length: between 1 and 64
 
 ### tableName
 
- - Local name (not fully qualified name) of a table.
- - Type: `string`
- - The value must match this pattern: `^[^.]*$`
- - Length: between 1 and 64
-
+* Local name (not fully qualified name) of a table.
+* Type: `string`
+* The value must match this pattern: `^[^.]*$`
+* Length: between 1 and 64
 
 ### fullyQualifiedColumnName
 
@@ -179,106 +230,84 @@ Type: `object`
 
 ### column
 
- - This schema defines the type for a column in a table.
- - Type: `object`
- - This schema <u>does not</u> accept additional properties.
- - **Properties**
-   - **name** `required`
-     - $ref: [#/definitions/columnName](#columnname)
-   - **dataType** `required`
-     - Data type of the column (int, date etc.).
-     - $ref: [#/definitions/dataType](#datatype)
-   - **arrayDataType**
-     - Data type used array in dataType. For example, `array<int>` has dataType as `array` and arrayDataType as `int`.
-     - $ref: [#/definitions/dataType](#datatype)
-   - **dataLength**
-     - Length of `char`, `varchar`, `binary`, `varbinary` `dataTypes`, else null. For example, `varchar(20)` has dataType as `varchar` and dataLength as `20`.
-     - Type: `integer`
-   - **dataTypeDisplay**
-     - Display name used for dataType. This is useful for complex types, such as `array<int>, map<int,string>, struct<>, and union types.
-     - Type: `string`
-   - **description**
-     - Description of the column.
-     - Type: `string`
-   - **fullyQualifiedName**
-     - $ref: [#/definitions/fullyQualifiedColumnName](#fullyqualifiedcolumnname)
-   - **tags**
-     - Tags associated with the column.
-     - Type: `array`
-       - **Items**
-       - $ref: [../../type/tagLabel.json](../types/taglabel.md)
-   - **constraint**
-     - Column level constraint.
-     - $ref: [#/definitions/constraint](#constraint)
-   - **ordinalPosition**
-     - Ordinal position of the column.
-     - Type: `integer`
-   - **jsonSchema**
-     - Json schema only if the dataType is JSON else null.
-     - Type: `string`
-   - **children**
-     - Child columns if dataType or arrayDataType is `map`, `struct`, or `union` else `null`.
-     - Type: `array`
-       - **Items**
-       - $ref: [#/definitions/column](#column)
-
+* This schema defines the type for a column in a table.
+* Type: `object`
+* **Properties**
+  * **name** `required`
+    * $ref: [#/definitions/columnName](table.md#columnname)
+  * **columnDataType** `required`
+    * Data type of the column (int, date etc.).
+    * $ref: [#/definitions/columnDataType](table.md#columndatatype)
+  * **description**
+    * Description of the column.
+    * Type: `string`
+  * **fullyQualifiedName**
+    * $ref: [#/definitions/fullyQualifiedColumnName](table.md#fullyqualifiedcolumnname)
+  * **tags**
+    * Tags associated with the column.
+    * Type: `array`
+      * **Items**
+      * $ref: [../../type/tagLabel.json](../types/taglabel.md)
+  * **columnConstraint**
+    * Column level constraint.
+    * $ref: [#/definitions/columnConstraint](table.md#columnconstraint)
+  * **ordinalPosition**
+    * Ordinal position of the column.
+    * Type: `integer`
 
 ### columnJoins
 
- - This schema defines the type to capture how frequently a column is joined with columns in the other tables.
- - Type: `object`
- - This schema <u>does not</u> accept additional properties.
- - **Properties**
-   - **columnName**
-     - $ref: [#/definitions/columnName](#columnname)
-   - **joinedWith**
-     - Fully qualified names of the columns that this column is joined with.
-     - Type: `array`
-       - **Items**
-       - Type: `object`
- - **Properties**
-   - **fullyQualifiedName**
-     - $ref: [#/definitions/fullyQualifiedColumnName](#fullyqualifiedcolumnname)
-   - **joinCount**
-     - Type: `integer`
-
+* This schema defines the type to capture how frequently a column is joined with columns in the other tables.
+* Type: `object`
+* This schema does not accept additional properties.
+* **Properties**
+  * **columnName**
+    * $ref: [#/definitions/columnName](table.md#columnname)
+  * **joinedWith**
+    * Fully qualified names of the columns that this column is joined with.
+    * Type: `array`
+      * **Items**
+      * Type: `object`
+* **Properties**
+  * **fullyQualifiedName**
+    * $ref: [#/definitions/fullyQualifiedColumnName](table.md#fullyqualifiedcolumnname)
+  * **joinCount**
+    * Type: `integer`
 
 ### tableJoins
 
- - This schema defines the type to capture information about how columns in this table are joined with columns in the other tables.
- - Type: `object`
- - This schema <u>does not</u> accept additional properties.
- - **Properties**
-   - **startDate**
-     - Date can be only from today going back to last 29 days.
-     - $ref: [../../type/basic.json#/definitions/date](../types/basic.md#date)
-   - **dayCount**
-     - Type: `integer`
-     - Default: `1`
-   - **columnJoins**
-     - Type: `array`
-       - **Items**
-         - $ref: [#/definitions/columnJoins](#columnjoins)
-
+* This schema defines the type to capture information about how columns in this table are joined with columns in the other tables.
+* Type: `object`
+* This schema does not accept additional properties.
+* **Properties**
+  * **startDate**
+    * Date can be only from today going back to last 29 days.
+    * $ref: [../../type/basic.json#/definitions/date](../types/basic.md#date)
+  * **dayCount**
+    * Type: `integer`
+    * Default: `1`
+  * **columnJoins**
+    * Type: `array`
+      * **Items**
+        * $ref: [#/definitions/columnJoins](table.md#columnjoins)
 
 ### tableData
 
- - This schema defines the type to capture rows of sample data for a table.
- - Type: `object`
- - This schema <u>does not</u> accept additional properties.
- - **Properties**
- - **columns**
-   - List of local column names (not fully qualified column names) of the table.
-   - Type: `array`
-     - **Items**
-     - $ref: [#/definitions/columnName](#columnname)
- - **rows**
-   - Data for multiple rows of the table.
-   - Type: `array`
-     - **Items**
-     - Data for a single row of the table within the same order as columns fields.
-     - Type: `array`
-
+* This schema defines the type to capture rows of sample data for a table.
+* Type: `object`
+* This schema does not accept additional properties.
+* **Properties**
+* **columns**
+  * List of local column names (not fully qualified column names) of the table.
+  * Type: `array`
+    * **Items**
+    * $ref: [#/definitions/columnName](table.md#columnname)
+* **rows**
+  * Data for multiple rows of the table.
+  * Type: `array`
+    * **Items**
+    * Data for a single row of the table within the same order as columns fields.
+    * Type: `array`
 
 ### columnProfile
 
@@ -317,27 +346,23 @@ Type: `object`
      - Type: `number`
 
 
-### tableProfile
+* This schema defines the type to capture the table's data profile.
+* Type: `object`
+* This schema does not accept additional properties.
+  * **Properties**
+    * **profileDate**
+      * Data one which profile is taken.
+      * $ref: [../../type/basic.json#/definitions/date](../types/basic.md#date)
+    * **columnCount**
+      * No.of columns in the table.
+      * Type: `number`
+    * **rowCount**
+      * No.of rows in the table.
+      * Type: `number`
+    * **columnProfile**
+      * List of local column profiles of the table.
+      * Type: `array`
+    * **Items**
+      * $ref: [#/definitions/columnProfile](table.md#columnprofile)
 
- - This schema defines the type to capture the table's data profile.
- - Type: `object`
- - This schema <u>does not</u> accept additional properties.
-   - **Properties**
-     - **profileDate**
-       - Data one which profile is taken.
-       - $ref: [../../type/basic.json#/definitions/date](../types/basic.md#date)
-     - **columnCount**
-       - No.of columns in the table.
-       - Type: `number`
-     - **rowCount**
-       - No.of rows in the table.
-       - Type: `number`
-     - **columnProfile**
-       - List of local column profiles of the table.
-       - Type: `array`
-     - **Items**
-       - $ref: [#/definitions/columnProfile](#columnprofile)
-
-
-
-_This document was updated on: Thursday, October 14, 2021_
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/tagcategory.md

@@ -2,7 +2,7 @@
 
 This schema defines the Tag Category entity. A Tag Category contains tags called Primary Tags. Primary Tags can further have children Tags called Secondary Tags. Only two levels of tags are supported currently.
 
-**$id: https://open-metadata.org/schema/entity/tags/tagCategory.json**
+**$id: **[https://open-metadata.org/schema/entity/tags/tagCategory.json](https://open-metadata.org/schema/entity/tags/tagCategory.json)
 
 Type: `object`
 
@@ -31,6 +31,24 @@ This schema <u>does not</u> accept additional properties.
      - **Items**
      - $ref: [#/definitions/tag](#tag)
 
+* **name** `required`
+  * $ref: [#/definitions/tagName](tagcategory.md#tagname)
+* **description** `required`
+  * Description of the tag category.
+  * Type: `string`
+* **categoryType** `required`
+  * $ref: [#/definitions/tagCategoryType](tagcategory.md#tagcategorytype)
+* **href**
+  * Link to the resource corresponding to the tag category.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
+* **usageCount**
+  * Count how many times the tags from this tag category are used.
+  * Type: `integer`
+* **children**
+  * Tags under this category.
+  * Type: `array`
+    * **Items**
+    * $ref: [#/definitions/tag](tagcategory.md#tag)
 
 ## Type definitions in this schema
 ### tagName
@@ -51,5 +69,4 @@ This schema <u>does not</u> accept additional properties.
 
 ### tag
 
-
-_This document was updated on: Thursday, October 14, 2021_
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/team.md

@@ -2,7 +2,7 @@
 
 This schema defines the Team entity. A Team is a group of zero or more users. Teams can own zero or more data assets.
 
-**$id: https://open-metadata.org/schema/entity/teams/team.json**
+**$id: **[https://open-metadata.org/schema/entity/teams/team.json](https://open-metadata.org/schema/entity/teams/team.json)
 
 Type: `object`
 
@@ -33,6 +33,31 @@ Type: `object`
    - List of entities owned by the team.
    - $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
 
+* **id** `required`
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name** `required`
+  * $ref: [#/definitions/teamName](team.md#teamname)
+* **displayName**
+  * Name used for display purposes. Example 'Data Science team'.
+  * Type: `string`
+* **description**
+  * Description of the team.
+  * Type: `string`
+* **href** `required`
+  * Link to the resource corresponding to this entity.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
+* **profile**
+  * Team profile information.
+  * $ref: [../../type/profile.json](../types/profile.md)
+* **deleted**
+  * When true the team has been deleted.
+  * Type: `boolean`
+* **users**
+  * Users that are part of the team.
+  * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
+* **owns**
+  * List of entities owned by the team.
+  * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
 
 ## Type definitions in this schema
 ### teamName
@@ -42,5 +67,4 @@ Type: `object`
  - Length: between 1 and 64
 
 
-
-_This document was updated on: Thursday, October 14, 2021_
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/thread.md

@@ -2,7 +2,7 @@
 
 This schema defines the Thread entity. A Thread is a collection of posts made by the users. The first post that starts a thread is **about** a data asset **from** a user. Other users can respond to this post by creating new posts in the thread. Note that bot users can also interact with a thread. A post can contains links that mention Users or other Data Assets.
 
-**$id: https://open-metadata.org/schema/entity/feed/thread.json**
+**$id: **[https://open-metadata.org/schema/entity/feed/thread.json](https://open-metadata.org/schema/entity/feed/thread.json)
 
 Type: `object`
 
@@ -27,24 +27,41 @@ Type: `object`
      - **Items**
      - $ref: [#/definitions/post](#post)
 
+* **id** `required`
+  * Unique identifier that identifies an entity instance.
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **href**
+  * Link to the resource corresponding to this entity.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
+* **threadTs**
+  * Timestamp of the when the first post created the thread.
+  * $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime)
+* **about** `required`
+  * Data asset about which this thread is created for with format <#E/{enties}/{entityName}/{field}/{fieldValue}.
+  * $ref: [../../type/basic.json#/definitions/entityLink](../types/basic.md#entitylink)
+* **addressedTo**
+  * User or team this thread is addressed to in format <#E/{enties}/{entityName}/{field}/{fieldValue}.
+  * $ref: [../../type/basic.json#/definitions/entityLink](../types/basic.md#entitylink)
+* **posts** `required`
+  * Type: `array`
+    * **Items**
+    * $ref: [#/definitions/post](thread.md#post)
 
 ## Type definitions in this schema
 ### post
 
- - Post within a feed.
- - Type: `object`
- - **Properties**
-   - **message** `required`
-     - Message in markdown format. See markdown support for more details.
-     - Type: `string`
-   - **postTs**
-     - Timestamp of the post.
-     - Type: `string`
-     - String format must be a "date-time"
-   - **from** `required`
-     - ID of User (regular user or a bot) posting the message.
-     - $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* Post within a feed.
+* Type: `object`
+* **Properties**
+  * **message** `required`
+    * Message in markdown format. See markdown support for more details.
+    * Type: `string`
+    * **postTs**
+      * Timestamp of the post.
+      * Type: `string`
+      * String format must be a "date-time"
+      * **from** `required`
+        * The ID of the User (regular user or a bot) posting the message.
+        * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
 
-
-
-_This document was updated on: Thursday, October 14, 2021_
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/topic.md

@@ -2,7 +2,7 @@
 
 This schema defines the Topic entity. A topic is a feed into which message are published to by publishers and read from by consumers in a messaging service.
 
-**$id: https://open-metadata.org/schema/entity/data/topic.json**
+**$id: **[https://open-metadata.org/schema/entity/data/topic.json](https://open-metadata.org/schema/entity/data/topic.json)
 
 Type: `object`
 
@@ -71,6 +71,66 @@ Type: `object`
    - Link to the resource corresponding to this entity.
    - $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
 
+* **id** `required`
+  * Unique identifier that identifies this topic instance.
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name** `required`
+  * Name that identifies the topic.
+  * $ref: [#/definitions/topicName](topic.md#topicname)
+* **fullyQualifiedName**
+  * Name that uniquely identifies a topic in the format 'messagingServiceName.topicName'.
+  * Type: `string`
+* **description**
+  * Description of the topic instance.
+  * Type: `string`
+* **service** `required`
+  * Link to the messaging cluster/service where this topic is hosted in.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **partitions** `required`
+  * A number of partitions into which the topic is divided.
+  * Type: `integer`
+  * Range:  ≥ 1
+* **schemaText**
+  * The Schema is used for message serialization. Optional as some topics may not have associated schemas.
+  * Type: `string`
+* **schemaType**
+  * Schema used for message serialization.
+  * $ref: [#/definitions/schemaType](topic.md#schematype)
+* **cleanupPolicies**
+  * Topic clean up policies. For Kafka - `cleanup.policy` configuration.
+  * Type: `array`
+    * **Items**
+    * $ref: [#/definitions/cleanupPolicy](topic.md#cleanuppolicy)
+* **retentionTime**
+  * Retention time in milliseconds. For Kafka - `retention.ms` configuration.
+  * Type: `number`
+* **replicationFactor**
+  * Replication Factor in integer (more than 1).
+  * Type: `integer`
+* **maximumMessageSize**
+  * Maximum message size in bytes. For Kafka - `max.message.bytes` configuration.
+  * Type: `integer`
+* **minimumInSyncReplicas**
+  * The minimum number of replicas in sync to control durability. For Kafka - `min.insync.replicas` configuration.
+  * Type: `integer`
+* **retentionSize**
+  * Maximum size of a partition in bytes before old data is discarded. For Kafka - `retention.bytes` configuration.
+  * Type: `number`
+  * Default: _"-1"_
+* **owner**
+  * Owner of this topic.
+  * $ref: [../../type/entityReference.json](../types/entityreference.md)
+* **followers**
+  * Followers of this table.
+  * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
+* **tags**
+  * Tags for this table.
+  * Type: `array`
+    * **Items**
+    * $ref: [../../type/tagLabel.json](../types/taglabel.md)
+* **href**
+  * Link to the resource corresponding to this entity.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
 
 ## Type definitions in this schema
 ### topicName
@@ -99,5 +159,4 @@ Type: `object`
    2. _"compact"_
 
 
-
-_This document was updated on: Thursday, October 14, 2021_
+_This document was updated on: Thursday, September 16, 2021_

docs/openmetadata-apis/schemas/entities/user.md

@@ -2,7 +2,7 @@
 
 This schema defines the User entity. A user can be part of 0 or more teams. A special type of user called Bot is used for automation. A user can be an owner of zero or more data assets. A user can also follow zero or more data assets.
 
-**$id: https://open-metadata.org/schema/entity/teams/user.json**
+**$id: **[https://open-metadata.org/schema/entity/teams/user.json](https://open-metadata.org/schema/entity/teams/user.json)
 
 Type: `object`
 
@@ -49,6 +49,45 @@ This schema <u>does not</u> accept additional properties.
    - List of entities followed by the user.
    - $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
 
+* **id** `required`
+  * Unique identifier that identifies a user entity instance.
+  * $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
+* **name** `required`
+  * $ref: [#/definitions/userName](user.md#username)
+* **displayName**
+  * Name used for display purposes. Example 'FirstName LastName'.
+  * Type: `string`
+* **email** `required`
+  * Email address of the user.
+  * $ref: [../../type/basic.json#/definitions/email](../types/basic.md#email)
+* **href** `required`
+  * Link to the resource corresponding to this entity.
+  * $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
+* **timezone**
+  * The Timezone of the user.
+  * Type: `string`
+  * String format must be a "timezone"
+* **deactivated**
+  * When true indicates the user has been deactivated. Users are deactivated instead of deleted.
+  * Type: `boolean`
+* **isBot**
+  * When true indicates a special type of user called Bot.
+  * Type: `boolean`
+* **isAdmin**
+  * When true indicates the user is an administrator for the system with superuser privileges.
+  * Type: `boolean`
+* **profile**
+  * Profile of the user.
+  * $ref: [../../type/profile.json](../types/profile.md)
+* **teams**
+  * Teams that the user belongs to.
+  * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
+* **owns**
+  * List of entities owned by the user.
+  * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
+* **follows**
+  * List of entities followed by the user.
+  * $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
 
 ## Type definitions in this schema
 ### userName
@@ -57,5 +96,4 @@ This schema <u>does not</u> accept additional properties.
  - Type: `string`
  - Length: between 1 and 64
 
-
-_This document was updated on: Thursday, October 14, 2021_
+_This document was updated on: Thursday, September 16, 2021_