GitBook: [#177] Documentation Update - Airflow
BIN
.gitbook/assets/blank-diagram-page-2-1-.png
Normal file
|
After Width: | Height: | Size: 37 KiB |
BIN
.gitbook/assets/create-user.png
Normal file
|
After Width: | Height: | Size: 77 KiB |
BIN
.gitbook/assets/fork-github (1).png
Normal file
|
After Width: | Height: | Size: 277 KiB |
BIN
.gitbook/assets/fork-github (2).png
Normal file
|
After Width: | Height: | Size: 277 KiB |
BIN
.gitbook/assets/fork-github (3).png
Normal file
|
After Width: | Height: | Size: 277 KiB |
BIN
.gitbook/assets/fork-github (4).png
Normal file
|
After Width: | Height: | Size: 277 KiB |
BIN
.gitbook/assets/fork-github.png
Normal file
|
After Width: | Height: | Size: 277 KiB |
BIN
.gitbook/assets/g-create-oauth (1).png
Normal file
|
After Width: | Height: | Size: 169 KiB |
BIN
.gitbook/assets/g-create-oauth.png
Normal file
|
After Width: | Height: | Size: 169 KiB |
|
After Width: | Height: | Size: 81 KiB |
|
After Width: | Height: | Size: 81 KiB |
|
After Width: | Height: | Size: 81 KiB |
BIN
.gitbook/assets/g-create-project (2) (2) (2) (2) (2) (3) (1).png
Normal file
|
After Width: | Height: | Size: 81 KiB |
BIN
.gitbook/assets/g-project-oauth-consent.png
Normal file
|
After Width: | Height: | Size: 131 KiB |
BIN
.gitbook/assets/image (1).png
Normal file
|
After Width: | Height: | Size: 188 KiB |
BIN
.gitbook/assets/image-1-.png
Normal file
|
After Width: | Height: | Size: 99 KiB |
BIN
.gitbook/assets/image-2-.png
Normal file
|
After Width: | Height: | Size: 181 KiB |
BIN
.gitbook/assets/image-3-.png
Normal file
|
After Width: | Height: | Size: 232 KiB |
BIN
.gitbook/assets/image-4-.png
Normal file
|
After Width: | Height: | Size: 210 KiB |
BIN
.gitbook/assets/image-5-.png
Normal file
|
After Width: | Height: | Size: 52 KiB |
BIN
.gitbook/assets/image-6-.png
Normal file
|
After Width: | Height: | Size: 6.9 KiB |
BIN
.gitbook/assets/image.png
Normal file
|
After Width: | Height: | Size: 189 KiB |
BIN
.gitbook/assets/ingestion-bot-service-account.png
Normal file
|
After Width: | Height: | Size: 150 KiB |
BIN
.gitbook/assets/localhost_7777_-1-.png
Normal file
|
After Width: | Height: | Size: 481 KiB |
BIN
.gitbook/assets/localhost_7777_.png
Normal file
|
After Width: | Height: | Size: 388 KiB |
BIN
.gitbook/assets/new-issue.png
Normal file
|
After Width: | Height: | Size: 516 KiB |
78302
.gitbook/assets/openmetadat-style-guide.pdf
Normal file
BIN
.gitbook/assets/openmetadata-entities-relationship.png
Normal file
|
After Width: | Height: | Size: 81 KiB |
BIN
.gitbook/assets/openmetadata-overview (1).png
Normal file
|
After Width: | Height: | Size: 161 KiB |
BIN
.gitbook/assets/openmetadata-overview.png
Normal file
|
After Width: | Height: | Size: 146 KiB |
BIN
.gitbook/assets/openmetadata-sandbox.png
Normal file
|
After Width: | Height: | Size: 453 KiB |
78302
.gitbook/assets/openmetadata-style-guide (1).pdf
Normal file
78302
.gitbook/assets/openmetadata-style-guide (2).pdf
Normal file
78302
.gitbook/assets/openmetadata-style-guide (3).pdf
Normal file
78302
.gitbook/assets/openmetadata-style-guide (4).pdf
Normal file
78302
.gitbook/assets/openmetadata-style-guide.pdf
Normal file
BIN
.gitbook/assets/pull-request-1.png
Normal file
|
After Width: | Height: | Size: 340 KiB |
BIN
.gitbook/assets/pull-request-2.png
Normal file
|
After Width: | Height: | Size: 1015 KiB |
BIN
.gitbook/assets/pull-request.png
Normal file
|
After Width: | Height: | Size: 366 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 111 KiB |
BIN
.gitbook/assets/screenshot-from-2021-07-26-21-08-30.png
Normal file
|
After Width: | Height: | Size: 153 KiB |
BIN
.gitbook/assets/screenshot-from-2021-07-26-21-09-24.png
Normal file
|
After Width: | Height: | Size: 137 KiB |
BIN
.gitbook/assets/screenshot-from-2021-07-26-23-57-46.png
Normal file
|
After Width: | Height: | Size: 131 KiB |
BIN
.gitbook/assets/screenshot-from-2021-09-20-14-11-53 (1).png
Normal file
|
After Width: | Height: | Size: 49 KiB |
BIN
.gitbook/assets/screenshot-from-2021-09-20-14-11-53.png
Normal file
|
After Width: | Height: | Size: 43 KiB |
BIN
.gitbook/assets/screenshot-from-2021-09-20-14-14-14.png
Normal file
|
After Width: | Height: | Size: 42 KiB |
BIN
.gitbook/assets/screenshot-from-2021-09-20-14-56-55.png
Normal file
|
After Width: | Height: | Size: 75 KiB |
BIN
.gitbook/assets/screenshot-from-2021-09-20-14-59-07.png
Normal file
|
After Width: | Height: | Size: 75 KiB |
BIN
.gitbook/assets/welcome (1).png
Normal file
|
After Width: | Height: | Size: 40 KiB |
BIN
.gitbook/assets/welcome (2).png
Normal file
|
After Width: | Height: | Size: 63 KiB |
BIN
.gitbook/assets/welcome.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
@ -4,7 +4,7 @@ Metadata enables you to unlock the value of data assets in the common use cases
|
||||
|
||||
### **OpenMetadata is an open standard with a centralized metadata store that unifies all your data assets end-to-end to enable data discovery, user collaboration, and tool interoperability.**
|
||||
|
||||

|
||||
.png>)
|
||||
|
||||
OpenMetadata includes the following:
|
||||
|
||||
@ -17,4 +17,3 @@ OpenMetadata includes the following:
|
||||
## License
|
||||
|
||||
OpenMetadata is released under [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0)
|
||||
|
||||
|
||||
@ -91,7 +91,6 @@
|
||||
* [Kafka](install/metadata-ingestion/connectors/messaging-services/kafka.md)
|
||||
* [Processors](install/metadata-ingestion/connectors/processors/README.md)
|
||||
* [PII](install/metadata-ingestion/connectors/processors/pii.md)
|
||||
* [Scheduler](install/metadata-ingestion/scheduler.md)
|
||||
* [Airflow](install/metadata-ingestion/airflow.md)
|
||||
|
||||
## Open Source Community
|
||||
@ -109,4 +108,3 @@
|
||||
* [BulkSink](open-source-community/developer/build-a-connector/bulksink.md)
|
||||
* [Run Integration Tests](open-source-community/developer/run-integration-tests.md)
|
||||
* [UX Style Guide](open-source-community/developer/ux-style-guide.md)
|
||||
|
||||
|
||||
@ -28,7 +28,7 @@ description: This is a guide to create ingestion bot service account.
|
||||
Make sure the Service Account Id is **ingestion-bot** and click on **CREATE AND CONTINUE**. If you chose a different Service Account Id, add it to the default bots list in [Configure OpenMetadata Server](https://github.com/StreamlineData/catalog/tree/3d53fa7c645ea55f846b06d0210ac63f8c38463f/docs/install/install/google-catalog-config.md)
|
||||
{% endhint %}
|
||||
|
||||

|
||||

|
||||
|
||||
* Click on **Select a role** and give the **Owner** role. Then click **Continue.**
|
||||
|
||||
@ -71,4 +71,3 @@ Make sure the Service Account Id is **ingestion-bot** and click on **CREATE AND
|
||||

|
||||
|
||||
* **The private-key/service-account JSON file will be downloaded**
|
||||
|
||||
|
||||
@ -3,9 +3,9 @@
|
||||
## Step 1: Create the account
|
||||
|
||||
* Go to [Create Google Cloud Account](https://console.cloud.google.com)
|
||||
* Click on `Create Project`
|
||||
* Click on `Create Project`
|
||||
|
||||

|
||||
 (2) (2) (2) (2) (3) (1).png>)
|
||||
|
||||
## Step 2: Create a new project
|
||||
|
||||
@ -20,7 +20,7 @@
|
||||
|
||||
* Select the project you created above and click on **APIs & Services** on the left-side panel.
|
||||
|
||||

|
||||

|
||||
|
||||
* Click on the **OAuth Consent Screen** available on the left-hand side panel.
|
||||
* Provide the User Type. Chose **External** if you are testing otherwise chose **Internal**
|
||||
@ -63,7 +63,7 @@
|
||||
|
||||
* After selecting the **Application Type**, name your project and give the authorized URIs
|
||||
|
||||

|
||||

|
||||
|
||||
* Click **Create**
|
||||
* You will get the credentials
|
||||
@ -73,11 +73,10 @@
|
||||
## Step 5: Where to find the credentials
|
||||
|
||||
* Go to **Credentials**
|
||||
* Click on the **pencil icon \(Edit OAuth Client\)** on the right side of the screen
|
||||
* Click on the **pencil icon (Edit OAuth Client)** on the right side of the screen
|
||||
|
||||

|
||||
|
||||
* You will find the **client Id** and **client secret** in the top right corner
|
||||
|
||||

|
||||
|
||||
|
||||
@ -14,7 +14,7 @@
|
||||
|
||||

|
||||
|
||||
* Click **Applications -> Applications** available on the left side of the panel.
|
||||
* Click **Applications -> Applications** available on the left side of the panel.
|
||||
|
||||

|
||||
|
||||
@ -34,18 +34,18 @@
|
||||

|
||||
|
||||
* Once selected, click **Next**.
|
||||
* Select **Refresh token**, **Implicit\(Hybrid\)**. Once selected Implicit option, select **Allow ID token with implicit grant type**. In Refresh token behavior. Select **Use persistent token**. Provide the necessary input and click **Save**.
|
||||
* Select **Refresh token**, **Implicit(Hybrid)**. Once selected Implicit option, select **Allow ID token with implicit grant type**. In Refresh token behavior. Select **Use persistent token**. Provide the necessary input and click **Save**.
|
||||
|
||||

|
||||

|
||||
|
||||
* The app is now configured.
|
||||
|
||||

|
||||
.png>)
|
||||
|
||||
## Step 4: Where to find the credentials
|
||||
|
||||
* Once the app is configured, now the **Client Id** and **Client secret** can be used.
|
||||
* You can also go to **Application -> Application** like in step 2.
|
||||
* You can also go to **Application -> Application** like in step 2.
|
||||
* You should be able to see your application in the list.
|
||||
|
||||

|
||||
@ -53,7 +53,7 @@
|
||||
* Click on your application.
|
||||
* You will find your **Client id**, **Client secret,** and **Okta domain**.
|
||||
|
||||

|
||||

|
||||
|
||||
* To get your **issuer** and **audience** click on **Sign On** available on top of the form.
|
||||
|
||||
@ -61,9 +61,8 @@
|
||||
|
||||
## Step 5: Adding the details in openmetadata-security.yaml
|
||||
|
||||
* Once the **Client Id**, **Client secret**, **issuer,** and **audience** are generated.
|
||||
* Once the **Client Id**, **Client secret**, **issuer,** and **audience** are generated.
|
||||
|
||||
Add those details in the openmetadata-security.yaml file in the respective field.
|
||||
Add those details in the openmetadata-security.yaml file in the respective field.
|
||||
|
||||

|
||||
|
||||
|
||||
@ -1,13 +1,10 @@
|
||||
# Metadata Ingestion
|
||||
|
||||
OpenMetadata Ingestion is a simple framework to build connectors and ingest metadata of various systems through [OpenMetadata APIs](../../openmetadata-apis/apis/). It could be used in an orchestration framework\(e.g. Apache Airflow\) to ingest metadata.
|
||||
OpenMetadata Ingestion is a simple framework to build connectors and ingest metadata of various systems through [OpenMetadata APIs](../../openmetadata-apis/apis/). It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata.
|
||||
|
||||
## Guides
|
||||
|
||||
* [Ingest Sample Data](ingest-sample-data.md)
|
||||
* [Explore Connectors & Install](connectors/)
|
||||
* [Run Simple Scheduler](scheduler.md)
|
||||
* [Configure Airflow](airflow.md)
|
||||
|
||||
|
||||
|
||||
|
||||
@ -2,10 +2,11 @@
|
||||
|
||||
We highly recommend using Airflow or similar schedulers to run Metadata Connectors. Below is the sample code example you can refer to integrate with Airflow
|
||||
|
||||
## Airflow Example for Hive
|
||||
## Airflow Example for Sample Data
|
||||
|
||||
```python
|
||||
import pathlib
|
||||
import json
|
||||
from datetime import timedelta
|
||||
from airflow import DAG
|
||||
|
||||
@ -27,12 +28,30 @@ default_args = {
|
||||
"execution_timeout": timedelta(minutes=60)
|
||||
}
|
||||
|
||||
config = """
|
||||
{
|
||||
"source": {
|
||||
"type": "sample-data",
|
||||
"config": {
|
||||
"sample_data_folder": "./examples/sample_data"
|
||||
}
|
||||
},
|
||||
"sink": {
|
||||
"type": "metadata-rest",
|
||||
"config": {}
|
||||
},
|
||||
"metadata_server": {
|
||||
"type": "metadata-server",
|
||||
"config": {
|
||||
"api_endpoint": "http://localhost:8585/api",
|
||||
"auth_provider_type": "no-auth"
|
||||
}
|
||||
}
|
||||
}
|
||||
"""
|
||||
|
||||
def metadata_ingestion_workflow():
|
||||
config_file = pathlib.Path("/tmp/sample_data.json")
|
||||
workflow_config = load_config_file(config_file)
|
||||
if workflow_config.get('cron'):
|
||||
del workflow_config['cron']
|
||||
workflow_config = json.loads(config)
|
||||
workflow = Workflow.create(workflow_config)
|
||||
workflow.execute()
|
||||
workflow.raise_from_status()
|
||||
@ -41,11 +60,12 @@ def metadata_ingestion_workflow():
|
||||
|
||||
|
||||
with DAG(
|
||||
"sample_metadata_ingestion_workflow",
|
||||
"sample_data",
|
||||
default_args=default_args,
|
||||
description="An example DAG which runs a OpenMetadata ingestion workflow",
|
||||
schedule_interval=timedelta(days=1),
|
||||
start_date=days_ago(1),
|
||||
is_paused_upon_creation=False,
|
||||
schedule_interval='*/5 * * * *',
|
||||
catchup=False,
|
||||
) as dag:
|
||||
ingest_task = PythonOperator(
|
||||
@ -58,16 +78,12 @@ we are using a python method like below
|
||||
|
||||
```python
|
||||
def metadata_ingestion_workflow():
|
||||
config_file = pathlib.Path("/tmp/sample_data.json")
|
||||
workflow_config = load_config_file(config_file)
|
||||
if workflow_config.get('cron'):
|
||||
del workflow_config['cron']
|
||||
workflow_config = json.loads(config)
|
||||
workflow = Workflow.create(workflow_config)
|
||||
workflow.execute()
|
||||
workflow.raise_from_status()
|
||||
workflow.print_status()
|
||||
workflow.stop()
|
||||
workflow.stop
|
||||
```
|
||||
|
||||
Create a Workflow instance and pass a hive configuration which will read metadata from Hive and ingest it into the OpenMetadata Server. You can customize this configuration or add different connectors please refer to our [examples](https://github.com/open-metadata/OpenMetadata/tree/main/ingestion/examples/workflows) and refer to [Connectors](connectors/).
|
||||
|
||||
Create a Workflow instance and pass a sample-data configuration which will read metadata from Json files and ingest it into the OpenMetadata Server. You can customize this configuration or add different connectors please refer to our [examples](https://github.com/open-metadata/OpenMetadata/tree/main/ingestion/examples/workflows) and refer to [Connectors](connectors/).
|
||||
|
||||
@ -2,11 +2,14 @@
|
||||
|
||||
OpenMetadata supports connectors to some popular dashboard services. We will continue as a community to provide more integrations to get all your metadata into a central repository.
|
||||
|
||||
{% page-ref page="looker.md" %}
|
||||
{% content-ref url="looker.md" %}
|
||||
[looker.md](looker.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="../database-services/salesforce.md" %}
|
||||
|
||||
{% page-ref page="superset.md" %}
|
||||
|
||||
{% page-ref page="tableau.md" %}
|
||||
{% content-ref url="superset.md" %}
|
||||
[superset.md](superset.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% content-ref url="tableau.md" %}
|
||||
[tableau.md](tableau.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
@ -2,23 +2,59 @@
|
||||
|
||||
OpenMetadata supports connectors to some popular data sources. We will continue as a community to provide more integrations to get all your metadata into a central repository.
|
||||
|
||||
{% page-ref page="athena.md" %}
|
||||
{% content-ref url="athena.md" %}
|
||||
[athena.md](athena.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="bigquery.md" %}
|
||||
{% content-ref url="bigquery.md" %}
|
||||
[bigquery.md](bigquery.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="elastic-search.md" %}
|
||||
{% content-ref url="elastic-search.md" %}
|
||||
[elastic-search.md](elastic-search.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="mssql.md" %}
|
||||
{% content-ref url="mssql.md" %}
|
||||
[mssql.md](mssql.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="mysql.md" %}
|
||||
{% content-ref url="mysql.md" %}
|
||||
[mysql.md](mysql.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="oracle.md" %}
|
||||
{% content-ref url="hive.md" %}
|
||||
[hive.md](hive.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="postgres.md" %}
|
||||
{% content-ref url="oracle.md" %}
|
||||
[oracle.md](oracle.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="redshift-usage.md" %}
|
||||
{% content-ref url="postgres.md" %}
|
||||
[postgres.md](postgres.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="snowflake.md" %}
|
||||
{% content-ref url="redshift.md" %}
|
||||
[redshift.md](redshift.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="snowflake-usage.md" %}
|
||||
{% content-ref url="redshift-usage.md" %}
|
||||
[redshift-usage.md](redshift-usage.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% content-ref url="salesforce.md" %}
|
||||
[salesforce.md](salesforce.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% content-ref url="snowflake.md" %}
|
||||
[snowflake.md](snowflake.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% content-ref url="snowflake-usage.md" %}
|
||||
[snowflake-usage.md](snowflake-usage.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% content-ref url="vertica.md" %}
|
||||
[vertica.md](vertica.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
|
||||
@ -32,7 +32,7 @@ metadata ingest -c ./examples/workflows/metadata_to_es.json
|
||||
|
||||
### Configuration
|
||||
|
||||
{% code title="metadata\_to\_es.json" %}
|
||||
{% code title="metadata_to_es.json" %}
|
||||
```javascript
|
||||
{
|
||||
"source": {
|
||||
@ -47,9 +47,9 @@ metadata ingest -c ./examples/workflows/metadata_to_es.json
|
||||
|
||||
Below is the configuration to publish Elastic Search data into the OpenMetadata service.
|
||||
|
||||
Add Optionally `file` stage and `elasticsearch` bulk\_sink along with `metadata-server` config
|
||||
Add Optionally `file` stage and `elasticsearch` bulk_sink along with `metadata-server` config
|
||||
|
||||
{% code title="metadata\_to\_es.json" %}
|
||||
{% code title="metadata_to_es.json" %}
|
||||
```javascript
|
||||
{
|
||||
"source": {
|
||||
@ -87,4 +87,3 @@ Add Optionally `file` stage and `elasticsearch` bulk\_sink along with `metadata-
|
||||
}
|
||||
```
|
||||
{% endcode %}
|
||||
|
||||
|
||||
@ -30,7 +30,7 @@ metadata ingest -c ./examples/workflows/redshift_usage.json
|
||||
|
||||
### Configuration
|
||||
|
||||
{% code title="redshift\_usage.json" %}
|
||||
{% code title="redshift_usage.json" %}
|
||||
```javascript
|
||||
{
|
||||
"source": {
|
||||
@ -51,16 +51,16 @@ metadata ingest -c ./examples/workflows/redshift_usage.json
|
||||
|
||||
1. **username** - pass the Redshift username. We recommend creating a user with read-only permissions to all the databases in your Redshift installation
|
||||
2. **password** - password for the username
|
||||
3. **service\_name** - Service Name for this Redshift cluster. If you added the Redshift cluster through OpenMetadata UI, make sure the service name matches the same.
|
||||
4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata
|
||||
3. **service_name** - Service Name for this Redshift cluster. If you added the Redshift cluster through OpenMetadata UI, make sure the service name matches the same.
|
||||
4. **filter_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata
|
||||
|
||||
## Publish to OpenMetadata
|
||||
|
||||
Below is the configuration to publish Redshift Usage data into the OpenMetadata service.
|
||||
|
||||
Add optionally`query-parser` processor, `table-usage` stage and `metadata-usage` bulk\_sink along with `metadata-server` config
|
||||
Add optionally`query-parser` processor, `table-usage` stage and `metadata-usage` bulk_sink along with `metadata-server` config
|
||||
|
||||
{% code title="redshift\_usage.json" %}
|
||||
{% code title="redshift_usage.json" %}
|
||||
```javascript
|
||||
{
|
||||
"source": {
|
||||
@ -110,4 +110,3 @@ Add optionally`query-parser` processor, `table-usage` stage and `metadata-usage`
|
||||
}
|
||||
```
|
||||
{% endcode %}
|
||||
|
||||
|
||||
@ -30,7 +30,7 @@ metadata ingest -c ./examples/workflows/snowflake_usage.json
|
||||
|
||||
### Configuration
|
||||
|
||||
{% code title="snowflake\_usage.json" %}
|
||||
{% code title="snowflake_usage.json" %}
|
||||
```javascript
|
||||
{
|
||||
"source": {
|
||||
@ -50,17 +50,17 @@ metadata ingest -c ./examples/workflows/snowflake_usage.json
|
||||
|
||||
1. **username** - pass the Snowflake username.
|
||||
2. **password** - the password for the Snowflake username.
|
||||
3. **service\_name** - Service Name for this Snowflake cluster. If you added the Snowflake cluster through OpenMetadata UI, make sure the service name matches the same.
|
||||
4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata.
|
||||
3. **service_name** - Service Name for this Snowflake cluster. If you added the Snowflake cluster through OpenMetadata UI, make sure the service name matches the same.
|
||||
4. **filter_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata.
|
||||
5. **database -** Database name from where data is to be fetched.
|
||||
|
||||
### Publish to OpenMetadata
|
||||
|
||||
Below is the configuration to publish Snowflake Usage data into the OpenMetadata service.
|
||||
|
||||
Add Optionally `query-parser` processor, `table-usage` stage and`metadata-usage` bulk\_sink along with `metadata-server` config
|
||||
Add Optionally `query-parser` processor, `table-usage` stage and`metadata-usage` bulk_sink along with `metadata-server` config
|
||||
|
||||
{% code title="snowflake\_usage.json" %}
|
||||
{% code title="snowflake_usage.json" %}
|
||||
```javascript
|
||||
{
|
||||
"source": {
|
||||
@ -110,4 +110,3 @@ Add Optionally `query-parser` processor, `table-usage` stage and`metadata-usage`
|
||||
}
|
||||
```
|
||||
{% endcode %}
|
||||
|
||||
|
||||
@ -49,10 +49,10 @@ metadata ingest -c ./examples/workflows/vertica.json
|
||||
```
|
||||
{% endcode %}
|
||||
|
||||
1. **username** - pass the Vertica username.
|
||||
1. **username **- pass the Vertica username.
|
||||
2. **password** - password for the username.
|
||||
3. **service\_name** - Service Name for this Vertica cluster. If you added Vertica cluster through OpenMetadata UI, make sure the service name matches the same.
|
||||
4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata
|
||||
3. **service_name** - Service Name for this Vertica cluster. If you added Vertica cluster through OpenMetadata UI, make sure the service name matches the same.
|
||||
4. **filter_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata
|
||||
|
||||
### Publish to OpenMetadata
|
||||
|
||||
@ -97,4 +97,3 @@ Add optionally `pii` processor and `metadata-rest` sink along with `metadata-ser
|
||||
|
||||
```
|
||||
{% endcode %}
|
||||
|
||||
|
||||
@ -30,7 +30,7 @@ metadata ingest -c ./examples/workflows/confluent_kafka.json
|
||||
|
||||
### Configuration
|
||||
|
||||
{% code title="confluent\_kafka.json" %}
|
||||
{% code title="confluent_kafka.json" %}
|
||||
```javascript
|
||||
{
|
||||
"source": {
|
||||
@ -48,8 +48,8 @@ metadata ingest -c ./examples/workflows/confluent_kafka.json
|
||||
```
|
||||
{% endcode %}
|
||||
|
||||
1. **service\_name** - Service Name for this Kafka cluster. If you added Kafka cluster through OpenMetadata UI, make sure the service name matches the same.
|
||||
2. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata
|
||||
1. **service_name** - Service Name for this Kafka cluster. If you added Kafka cluster through OpenMetadata UI, make sure the service name matches the same.
|
||||
2. **filter_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata
|
||||
|
||||
## Publish to OpenMetadata
|
||||
|
||||
@ -57,7 +57,7 @@ Below is the configuration to publish Kafka data into the OpenMetadata service.
|
||||
|
||||
Add optionally `pii` processor and `metadata-rest` sink along with `metadata-server` config
|
||||
|
||||
{% code title="confluent\_kafka.json" %}
|
||||
{% code title="confluent_kafka.json" %}
|
||||
```javascript
|
||||
{
|
||||
"source": {
|
||||
@ -93,4 +93,3 @@ Add optionally `pii` processor and `metadata-rest` sink along with `metadata-ser
|
||||
}
|
||||
```
|
||||
{% endcode %}
|
||||
|
||||
|
||||
@ -1,65 +0,0 @@
|
||||
---
|
||||
description: >-
|
||||
This guide will help install Simple Scheduler and schedule connectors for
|
||||
ingestion.
|
||||
---
|
||||
|
||||
# Scheduler
|
||||
|
||||
{% hint style="info" %}
|
||||
**Prerequisites**
|
||||
|
||||
OpenMetadata is built using Java, DropWizard, Jetty, and MySQL.
|
||||
|
||||
1. Python 3.7 or above
|
||||
2. Create an env
|
||||
|
||||
```bash
|
||||
python3 -m venv env
|
||||
```
|
||||
|
||||
3. Activating the environment
|
||||
|
||||
```bash
|
||||
source env/bin/activate
|
||||
```
|
||||
{% endhint %}
|
||||
|
||||
## Install dependencies
|
||||
|
||||
```text
|
||||
cd ingestion
|
||||
python3 -m venv env
|
||||
source env/bin/activate
|
||||
pip install '.[scheduler]'
|
||||
python ingestion_scheduler/scheduler.py
|
||||
```
|
||||
|
||||
{% hint style="warning" %}
|
||||
**Note:**
|
||||
|
||||
Different Connectors require different dependencies, please go through [Connectors](https://docs.open-metadata.org/install/metadata-ingestion/connectors) Documentation install dependencies as needed.
|
||||
{% endhint %}
|
||||
|
||||
## Scheduler UI
|
||||
|
||||
### Main Page
|
||||
|
||||
Loads all the Json connectors inside the pipeline directory as cron jobs.
|
||||
|
||||

|
||||
|
||||
### Custom run a job
|
||||
|
||||
* Click on the **Custom Run** button under the **Action** column of the jobs.
|
||||
* Click on **Run.**
|
||||
* The Job will start running the ingestion.
|
||||
|
||||

|
||||
|
||||
**Status of an executed job**
|
||||
|
||||
* Click on the **Executions** tab under Navigation Bar.
|
||||
|
||||

|
||||
|
||||
@ -13,8 +13,8 @@ description: >-
|
||||
{% hint style="info" %}
|
||||
**Prerequisites**
|
||||
|
||||
* Docker >= 20.10.x
|
||||
* Minimum allocated memory to Docker >= 4GB \(Preferences -> Advanced -> Resources\)
|
||||
* Docker >= 20.10.x
|
||||
* Minimum allocated memory to Docker >= 4GB (Preferences -> Advanced -> Resources)
|
||||
{% endhint %}
|
||||
|
||||
```bash
|
||||
@ -28,9 +28,9 @@ docker-compose up
|
||||
1. Docker for OpenMetadata will depend on Mysql Container to be up, It may take few seconds to run.
|
||||
2. Once OpenMetadata UI is accessible, Go to Scheduler UI -[ http://localhost:7777](http://localhost:7777), to invoke the pipelines to ingest data.
|
||||
|
||||

|
||||

|
||||
|
||||

|
||||

|
||||
|
||||
The above command brings up all the necessary services
|
||||
|
||||
@ -81,45 +81,41 @@ cd openmetadata-0.4.0
|
||||
#### macOS
|
||||
|
||||
1. Setup Database
|
||||
* Install MySQL
|
||||
* Install MySQL
|
||||
|
||||
```text
|
||||
brew install mysql
|
||||
```
|
||||
```
|
||||
brew install mysql
|
||||
```
|
||||
* Configure MySQL
|
||||
|
||||
* Configure MySQL
|
||||
```
|
||||
mysqladmin -u root password 'yourpassword'
|
||||
mysql -u root -p
|
||||
```
|
||||
* Setup Database
|
||||
|
||||
```text
|
||||
mysqladmin -u root password 'yourpassword'
|
||||
mysql -u root -p
|
||||
```
|
||||
```
|
||||
mysql -u root -p
|
||||
CREATE DATABASE openmetadata_db;
|
||||
CREATE USER 'openmetadata_user'@'localhost' IDENTIFIED BY 'openmetadata_password';
|
||||
GRANT ALL PRIVILEGES ON openmetadata_db.* TO 'openmetadata_user'@'localhost' WITH GRANT OPTION;
|
||||
commit;
|
||||
```
|
||||
2. Run bootstrap scripts to initialize the database and tables
|
||||
|
||||
* Setup Database
|
||||
```
|
||||
cd openmetadata-0.4.0
|
||||
./bootstrap/bootstrap_storage.sh migrate
|
||||
```
|
||||
3. Start the OpenMetadata Server
|
||||
|
||||
```text
|
||||
mysql -u root -p
|
||||
CREATE DATABASE openmetadata_db;
|
||||
CREATE USER 'openmetadata_user'@'localhost' IDENTIFIED BY 'openmetadata_password';
|
||||
GRANT ALL PRIVILEGES ON openmetadata_db.* TO 'openmetadata_user'@'localhost' WITH GRANT OPTION;
|
||||
commit;
|
||||
```
|
||||
2. Run bootstrap scripts to initialize the database and tables
|
||||
|
||||
```text
|
||||
cd openmetadata-0.4.0
|
||||
./bootstrap/bootstrap_storage.sh migrate
|
||||
```
|
||||
|
||||
3. Start the OpenMetadata Server
|
||||
|
||||
```text
|
||||
cd openmetadata-0.4.0
|
||||
./bin/openmetadata.sh start
|
||||
```
|
||||
```
|
||||
cd openmetadata-0.4.0
|
||||
./bin/openmetadata.sh start
|
||||
```
|
||||
|
||||
### Ingest Sample Data
|
||||
|
||||
Previous steps start OpenMetadata server. To start using it we need to run ElasticSearch and ingest sample metadata. Please follow the below guide:
|
||||
|
||||
[Ingest Sample Data](metadata-ingestion/ingest-sample-data.md)
|
||||
|
||||
|
||||
@ -16,7 +16,7 @@ A workflow consists of [Source](source.md), [Processor](processor.md) and [Sink]
|
||||
|
||||
Workflow execution happens in a serial fashion.
|
||||
|
||||
1. The **Workflow** runs the **source** component first. The **source** retrieves a record from external sources and emits the record downstream.
|
||||
1. The** Workflow** runs the **source** component first. The **source** retrieves a record from external sources and emits the record downstream.
|
||||
2. If the **processor** component is configured, the **workflow** sends the record to the **processor** next.
|
||||
3. There can be multiple **processor** components attached to the **workflow**. The **workflow** passes a record to each **processor** in the order they are configured.
|
||||
4. Once a **processor** is finished, it sends the modified record to the **sink**.
|
||||
@ -24,13 +24,22 @@ Workflow execution happens in a serial fashion.
|
||||
|
||||
In the cases where we need aggregation over the records, we can use the **stage** to write to a file or other store. Use the file written to in **stage** and pass it to **bulk sink** to publish to external services such as **openmetadata** or **elasticsearch**.
|
||||
|
||||
{% page-ref page="source.md" %}
|
||||
{% content-ref url="source.md" %}
|
||||
[source.md](source.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="processor.md" %}
|
||||
{% content-ref url="processor.md" %}
|
||||
[processor.md](processor.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="sink.md" %}
|
||||
{% content-ref url="sink.md" %}
|
||||
[sink.md](sink.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="stage.md" %}
|
||||
|
||||
{% page-ref page="bulksink.md" %}
|
||||
{% content-ref url="stage.md" %}
|
||||
[stage.md](stage.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% content-ref url="bulksink.md" %}
|
||||
[bulksink.md](bulksink.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
@ -12,11 +12,11 @@ The following commands must be run from the top-level directory.
|
||||
|
||||
If you wish to skip the unit tests you can do this by adding`-DskipTests`to the command line.
|
||||
|
||||
## Create a distribution \(packaging\)
|
||||
## Create a distribution (packaging)
|
||||
|
||||
You can create a _distribution_ as follows.
|
||||
|
||||
```text
|
||||
```
|
||||
$ mvn clean install
|
||||
|
||||
# Create the binary distribution.
|
||||
@ -25,7 +25,7 @@ $ cd dist && mvn package
|
||||
|
||||
The binaries will be created at:
|
||||
|
||||
```text
|
||||
```
|
||||
dist/target/open-metadata-<version>.pom
|
||||
dist/target/open-metadata-<version>.tar.gz
|
||||
```
|
||||
@ -34,40 +34,39 @@ dist/target/open-metadata-<version>.tar.gz
|
||||
|
||||
Add a new Run/Debug configuration like the below screenshot.
|
||||
|
||||

|
||||
.png>)
|
||||
|
||||
## Add missing dependency
|
||||
|
||||
Right-click on catalog-rest-service
|
||||
|
||||

|
||||

|
||||
|
||||
Click on "Open Module Settings"
|
||||
|
||||

|
||||

|
||||
|
||||
Go to "Dependencies"
|
||||
|
||||

|
||||

|
||||
|
||||
Click “+” at the bottom of the dialog box and click "Add"
|
||||
|
||||

|
||||

|
||||
|
||||
Click on Library
|
||||
|
||||

|
||||

|
||||
|
||||
In that list look for "jersey-client:2.25.1"
|
||||
|
||||

|
||||

|
||||
|
||||
Select it and click "OK". Now run/debug the application.
|
||||
|
||||
## Coding Style
|
||||
|
||||
1. [Refer to coding guidelines](coding-style.md)
|
||||
2. Configure IntelliJ to disable the \[wild-card imports\]
|
||||
|
||||
\([https://www.jetbrains.com/help/idea/creating-and-optimizing-imports.html\#disable-wildcard-imports](https://www.jetbrains.com/help/idea/creating-and-optimizing-imports.html#disable-wildcard-imports)\)
|
||||
2. Configure IntelliJ to disable the \[wild-card imports]
|
||||
|
||||
([https://www.jetbrains.com/help/idea/creating-and-optimizing-imports.html#disable-wildcard-imports](https://www.jetbrains.com/help/idea/creating-and-optimizing-imports.html#disable-wildcard-imports))
|
||||
|
||||
@ -8,13 +8,13 @@ We ❤️ all contributions, big and small!
|
||||
|
||||
Look for issues under [Github/issues tab](https://github.com/open-metadata/OpenMetadata/issues). If you have a feature request or found a bug please file an issue. This will help us track and will help the community overall as well.
|
||||
|
||||

|
||||

|
||||
|
||||
## Fork Github project
|
||||
|
||||
OpenMetadata Github repository can be accessed here [https://github.com/open-metadata/OpenMetadata](https://github.com/open-metadata/OpenMetadata).
|
||||
|
||||

|
||||
.png>)
|
||||
|
||||
Create a local clone of your fork
|
||||
|
||||
@ -49,15 +49,13 @@ git push origin HEAD:refs/heads/issue-200
|
||||
## Open a PR
|
||||
|
||||
1. Go to [https://github.com/open-metadata/OpenMetadata/pulls](https://github.com/open-metadata/OpenMetadata/pulls)
|
||||
2. It should show an option to open a pull request. 
|
||||
3. If not, click on "New Pull request"
|
||||
2. It should show an option to open a pull request. 
|
||||
3. If not, click on "New Pull request"
|
||||
|
||||

|
||||
|
||||
4. Select your fork repository and branch 
|
||||

|
||||
4. Select your fork repository and branch 
|
||||
5. Click "Create pull request"
|
||||
|
||||
## We are here to help
|
||||
|
||||
Please reach out to us anytime you need any help. [Slack](https://slack.open-metadata.org/) would be fastest way to get a response.
|
||||
|
||||
Please reach out to us anytime you need any help. [Slack](https://slack.open-metadata.org) would be fastest way to get a response.
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
OpenMetadata has the following high-level core entities and relationships:
|
||||
|
||||

|
||||

|
||||
|
||||
1. **Data Assets** - These entities represent data, such as databases, tables, and topics, and assets created using data, such as Dashboards, Reports, Metrics, and ML Features. It also includes entities such as Pipelines that are used for creating data assets.
|
||||
2. **Services** - Services represent platforms and services used for storing and processing data. It includes Online Data Stores, Data Warehouses, ETL tools, Dashboard services, etc.
|
||||
@ -12,35 +12,66 @@ OpenMetadata has the following high-level core entities and relationships:
|
||||
|
||||
## List of Schema Entities
|
||||
|
||||
{% page-ref page="bot.md" %}
|
||||
{% content-ref url="bot.md" %}
|
||||
[bot.md](bot.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="chart.md" %}
|
||||
{% content-ref url="chart.md" %}
|
||||
[chart.md](chart.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="dashboard.md" %}
|
||||
{% content-ref url="dashboard.md" %}
|
||||
[dashboard.md](dashboard.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="dashboardservice.md" %}
|
||||
{% content-ref url="dashboardservice.md" %}
|
||||
[dashboardservice.md](dashboardservice.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="database.md" %}
|
||||
{% content-ref url="database.md" %}
|
||||
[database.md](database.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="databaseservice.md" %}
|
||||
{% content-ref url="databaseservice.md" %}
|
||||
[databaseservice.md](databaseservice.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="messagingservice.md" %}
|
||||
{% content-ref url="messagingservice.md" %}
|
||||
[messagingservice.md](messagingservice.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="thread.md" %}
|
||||
{% content-ref url="thread.md" %}
|
||||
[thread.md](thread.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="metrics.md" %}
|
||||
{% content-ref url="metrics.md" %}
|
||||
[metrics.md](metrics.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="pipeline.md" %}
|
||||
{% content-ref url="pipeline.md" %}
|
||||
[pipeline.md](pipeline.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="report.md" %}
|
||||
{% content-ref url="report.md" %}
|
||||
[report.md](report.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="table.md" %}
|
||||
{% content-ref url="table.md" %}
|
||||
[table.md](table.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="team.md" %}
|
||||
{% content-ref url="team.md" %}
|
||||
[team.md](team.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="topic.md" %}
|
||||
{% content-ref url="topic.md" %}
|
||||
[topic.md](topic.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% page-ref page="user.md" %}
|
||||
|
||||
{% page-ref page="tagcategory.md" %}
|
||||
{% content-ref url="user.md" %}
|
||||
[user.md](user.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
{% content-ref url="tagcategory.md" %}
|
||||
[tagcategory.md](tagcategory.md)
|
||||
{% endcontent-ref %}
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines Bot entity. A bot automates tasks, such as adding description, identifying the importance of data. It runs as a special user in the system.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/bots.json](https://open-metadata.org/schema/entity/bots.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/bots.json](https://open-metadata.org/schema/entity/bots.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -10,7 +10,7 @@ Type: `object`
|
||||
|
||||
* **id**
|
||||
* Unique identifier of a bot instance.
|
||||
* $ref: [../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name**
|
||||
* Name of the bot.
|
||||
* Type: `string`
|
||||
@ -23,7 +23,6 @@ Type: `object`
|
||||
* Type: `string`
|
||||
* **href**
|
||||
* Link to the resource corresponding to this bot.
|
||||
* $ref: [../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the Chart entity. Charts are built using tables or SQL queries by analyzing the data. Charts can be part of Dashboard.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/data/chart.json](https://open-metadata.org/schema/entity/data/chart.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/data/chart.json](https://open-metadata.org/schema/entity/data/chart.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -10,7 +10,7 @@ Type: `object`
|
||||
|
||||
* **id** `required`
|
||||
* Unique identifier that identifies a chart instance.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name** `required`
|
||||
* Name that identifies this Chart.
|
||||
* Type: `string`
|
||||
@ -26,23 +26,23 @@ Type: `object`
|
||||
* Description of the dashboard, what it is, and how to use it.
|
||||
* Type: `string`
|
||||
* **chartType**
|
||||
* $ref: [\#/definitions/chartType](chart.md#charttype)
|
||||
* $ref: [#/definitions/chartType](chart.md#charttype)
|
||||
* **chartUrl**
|
||||
* Chart URL, pointing to its own Service URL.
|
||||
* Type: `string`
|
||||
* String format must be a "uri"
|
||||
* **href**
|
||||
* Link to the resource corresponding to this entity.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
* **owner**
|
||||
* Owner of this dashboard.
|
||||
* $ref: [../../type/entityReference.json](../types/entityreference.md)
|
||||
* **tables**
|
||||
* Link to table used in this chart.
|
||||
* $ref: [../../type/entityReference.json\#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* **followers**
|
||||
* Followers of this chart.
|
||||
* $ref: [../../type/entityReference.json\#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* **tags**
|
||||
* Tags for this chart.
|
||||
* Type: `array`
|
||||
@ -74,4 +74,3 @@ Type: `object`
|
||||
10. _"Other"_
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the Dashboard entity. Dashboards are computed from data and visually present data, metrics, and KPIs. They are updated in real-time and allow interactive data exploration.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/data/dashboard.json](https://open-metadata.org/schema/entity/data/dashboard.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/data/dashboard.json](https://open-metadata.org/schema/entity/data/dashboard.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -10,7 +10,7 @@ Type: `object`
|
||||
|
||||
* **id** `required`
|
||||
* Unique identifier that identifies a dashboard instance.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name** `required`
|
||||
* Name that identifies this dashboard.
|
||||
* Type: `string`
|
||||
@ -36,13 +36,13 @@ Type: `object`
|
||||
* $ref: [../../type/entityReference.json](../types/entityreference.md)
|
||||
* **href**
|
||||
* Link to the resource corresponding to this entity.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
* **owner**
|
||||
* Owner of this dashboard.
|
||||
* $ref: [../../type/entityReference.json](../types/entityreference.md)
|
||||
* **followers**
|
||||
* Followers of this dashboard.
|
||||
* $ref: [../../type/entityReference.json\#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* **tags**
|
||||
* Tags for this dashboard.
|
||||
* Type: `array`
|
||||
@ -56,4 +56,3 @@ Type: `object`
|
||||
* $ref: [../../type/usageDetails.json](../types/usagedetails.md)
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the Messaging Service entity, such as Kafka and Pulsar.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/services/dashboardService.json](https://open-metadata.org/schema/entity/services/dashboardService.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/services/dashboardService.json](https://open-metadata.org/schema/entity/services/dashboardService.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -10,14 +10,14 @@ Type: `object`
|
||||
|
||||
* **id** `required`
|
||||
* Unique identifier of this dashboard service instance.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name** `required`
|
||||
* Name that identifies this dashboard service.
|
||||
* Type: `string`
|
||||
* Length: between 1 and 64
|
||||
* **serviceType** `required`
|
||||
* Type of dashboard services such as Looker or Superset...
|
||||
* $ref: [\#/definitions/dashboardServiceType](dashboardservice.md#dashboardservicetype)
|
||||
* $ref: [#/definitions/dashboardServiceType](dashboardservice.md#dashboardservicetype)
|
||||
* **description**
|
||||
* Description of a dashboard service instance.
|
||||
* Type: `string`
|
||||
@ -36,7 +36,7 @@ Type: `object`
|
||||
* $ref: [../../type/schedule.json](../types/schedule.md)
|
||||
* **href**
|
||||
* Link to the resource corresponding to this messaging service.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
|
||||
## Type definitions in this schema
|
||||
|
||||
@ -51,4 +51,3 @@ Type: `object`
|
||||
4. _"Redash"_
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the Database entity. A database also referred to as Database Catalog is a collection of tables.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/data/database.json](https://open-metadata.org/schema/entity/data/database.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/data/database.json](https://open-metadata.org/schema/entity/data/database.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -10,10 +10,10 @@ Type: `object`
|
||||
|
||||
* **id**
|
||||
* Unique identifier that identifies this database instance.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name** `required`
|
||||
* Name that identifies the database.
|
||||
* $ref: [\#/definitions/databaseName](database.md#databasename)
|
||||
* $ref: [#/definitions/databaseName](database.md#databasename)
|
||||
* **fullyQualifiedName**
|
||||
* Name that uniquely identifies a database in the format 'ServiceName.DatabaseName'.
|
||||
* Type: `string`
|
||||
@ -22,7 +22,7 @@ Type: `object`
|
||||
* Type: `string`
|
||||
* **href**
|
||||
* Link to the resource corresponding to this entity.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
* **owner**
|
||||
* Owner of this database.
|
||||
* $ref: [../../type/entityReference.json](../types/entityreference.md)
|
||||
@ -34,7 +34,7 @@ Type: `object`
|
||||
* $ref: [../../type/usageDetails.json](../types/usagedetails.md)
|
||||
* **tables**
|
||||
* References to tables in the database.
|
||||
* $ref: [../../type/entityReference.json\#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
|
||||
## Type definitions in this schema
|
||||
|
||||
@ -46,4 +46,3 @@ Type: `object`
|
||||
* Length: between 1 and 64
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the Database Service entity, such as MySQL, BigQuery, Redshift, Postgres, or Snowflake. Alternative terms such as Database Cluster, Database Server instance are also used for database service.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/services/databaseService.json](https://open-metadata.org/schema/entity/services/databaseService.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/services/databaseService.json](https://open-metadata.org/schema/entity/services/databaseService.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -10,23 +10,23 @@ Type: `object`
|
||||
|
||||
* **id** `required`
|
||||
* Unique identifier of this database service instance.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name** `required`
|
||||
* Name that identifies this database service.
|
||||
* Type: `string`
|
||||
* Length: between 1 and 64
|
||||
* **serviceType** `required`
|
||||
* Type of database service such as MySQL, BigQuery, Snowflake, Redshift, Postgres...
|
||||
* $ref: [\#/definitions/databaseServiceType](databaseservice.md#databaseservicetype)
|
||||
* $ref: [#/definitions/databaseServiceType](databaseservice.md#databaseservicetype)
|
||||
* **description**
|
||||
* Description of a database service instance.
|
||||
* Type: `string`
|
||||
* **href** `required`
|
||||
* Link to the resource corresponding to this database service.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
* **jdbc** `required`
|
||||
* JDBC connection information.
|
||||
* $ref: [../../type/jdbcConnection.json\#/definitions/jdbcInfo](../types/jdbcconnection.md#jdbcinfo)
|
||||
* $ref: [../../type/jdbcConnection.json#/definitions/jdbcInfo](../types/jdbcconnection.md#jdbcinfo)
|
||||
* **ingestionSchedule**
|
||||
* Schedule for running metadata ingestion jobs.
|
||||
* $ref: [../../type/schedule.json](../types/schedule.md)
|
||||
@ -51,4 +51,3 @@ Type: `object`
|
||||
11. _"Vertica"_
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the Messaging Service entity, such as Kafka and Pulsar.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/services/messagingService.json](https://open-metadata.org/schema/entity/services/messagingService.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/services/messagingService.json](https://open-metadata.org/schema/entity/services/messagingService.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -10,20 +10,20 @@ Type: `object`
|
||||
|
||||
* **id** `required`
|
||||
* Unique identifier of this messaging service instance.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name** `required`
|
||||
* Name that identifies this messaging service.
|
||||
* Type: `string`
|
||||
* Length: between 1 and 64
|
||||
* **serviceType** `required`
|
||||
* Type of messaging services such as Kafka or Pulsar...
|
||||
* $ref: [\#/definitions/messagingServiceType](messagingservice.md#messagingservicetype)
|
||||
* $ref: [#/definitions/messagingServiceType](messagingservice.md#messagingservicetype)
|
||||
* **description**
|
||||
* Description of a messaging service instance.
|
||||
* Type: `string`
|
||||
* **brokers** `required`
|
||||
* Multiple bootstrap addresses for Kafka. Single proxy address for Pulsar.
|
||||
* $ref: [\#/definitions/brokers](messagingservice.md#brokers)
|
||||
* $ref: [#/definitions/brokers](messagingservice.md#brokers)
|
||||
* **schemaRegistry**
|
||||
* Schema registry URL.
|
||||
* Type: `string`
|
||||
@ -33,7 +33,7 @@ Type: `object`
|
||||
* $ref: [../../type/schedule.json](../types/schedule.md)
|
||||
* **href**
|
||||
* Link to the resource corresponding to this messaging service.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
|
||||
## Type definitions in this schema
|
||||
|
||||
@ -53,4 +53,3 @@ Type: `object`
|
||||
* Type: `string`
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the Metrics entity. Metrics are measurements computed from data such as `Monthly Active Users`. Some of the metrics that measures used to determine performance against an objective are called KPIs or Key Performance Indicators, such as `User Retention`.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/data/metrics.json](https://open-metadata.org/schema/entity/data/metrics.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/data/metrics.json](https://open-metadata.org/schema/entity/data/metrics.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -10,7 +10,7 @@ Type: `object`
|
||||
|
||||
* **id** `required`
|
||||
* Unique identifier that identifies this metrics instance.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name** `required`
|
||||
* Name that identifies this metrics instance uniquely.
|
||||
* Type: `string`
|
||||
@ -24,7 +24,7 @@ Type: `object`
|
||||
* Type: `string`
|
||||
* **href**
|
||||
* Link to the resource corresponding to this entity.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
* **owner**
|
||||
* Owner of these metrics.
|
||||
* $ref: [../../type/entityReference.json](../types/entityreference.md)
|
||||
@ -36,4 +36,3 @@ Type: `object`
|
||||
* $ref: [../../type/usageDetails.json](../types/usagedetails.md)
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the Pipeline entity. A pipeline enables the flow of data from source to destination through a series of processing steps. ETL is a type of pipeline where the series of steps Extract, Transform and Load the data.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/data/pipeline.json](https://open-metadata.org/schema/entity/data/pipeline.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/data/pipeline.json](https://open-metadata.org/schema/entity/data/pipeline.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -10,7 +10,7 @@ Type: `object`
|
||||
|
||||
* **id** `required`
|
||||
* Unique identifier that identifies a pipeline instance.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name** `required`
|
||||
* Name that identifies this pipeline instance uniquely.
|
||||
* Type: `string`
|
||||
@ -24,7 +24,7 @@ Type: `object`
|
||||
* Type: `string`
|
||||
* **href**
|
||||
* Link to the resource corresponding to this entity.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
* **owner**
|
||||
* Owner of this pipeline.
|
||||
* $ref: [../../type/entityReference.json](../types/entityreference.md)
|
||||
@ -33,4 +33,3 @@ Type: `object`
|
||||
* $ref: [../../type/entityReference.json](../types/entityreference.md)
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the Report entity. Reports are static information computed from data periodically that includes data in text, table, and visual form.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/data/report.json](https://open-metadata.org/schema/entity/data/report.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/data/report.json](https://open-metadata.org/schema/entity/data/report.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -10,7 +10,7 @@ Type: `object`
|
||||
|
||||
* **id** `required`
|
||||
* The Unique identifier that identifies this report.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name** `required`
|
||||
* Name that identifies this report instance uniquely.
|
||||
* Type: `string`
|
||||
@ -24,7 +24,7 @@ Type: `object`
|
||||
* Type: `string`
|
||||
* **href**
|
||||
* Link to the resource corresponding to this report.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
* **owner**
|
||||
* Owner of this pipeline.
|
||||
* $ref: [../../type/entityReference.json](../types/entityreference.md)
|
||||
@ -36,4 +36,3 @@ Type: `object`
|
||||
* $ref: [../../type/usageDetails.json](../types/usagedetails.md)
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the Table entity. A Table organizes data in rows and columns and is defined by a Schema. OpenMetadata does not have a separate abstraction for Schema. Both Table and Schema are captured in this entity.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/data/table.json](https://open-metadata.org/schema/entity/data/table.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/data/table.json](https://open-metadata.org/schema/entity/data/table.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -10,18 +10,18 @@ Type: `object`
|
||||
|
||||
* **id** `required`
|
||||
* Unique identifier of this table instance.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name** `required`
|
||||
* Name of a table. Expected to be unique within a database.
|
||||
* $ref: [\#/definitions/tableName](table.md#tablename)
|
||||
* $ref: [#/definitions/tableName](table.md#tablename)
|
||||
* **description**
|
||||
* Description of a table.
|
||||
* Type: `string`
|
||||
* **href**
|
||||
* Link to this table resource.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
* **tableType**
|
||||
* $ref: [\#/definitions/tableType](table.md#tabletype)
|
||||
* $ref: [#/definitions/tableType](table.md#tabletype)
|
||||
* **fullyQualifiedName**
|
||||
* Fully qualified name of a table in the form `serviceName.databaseName.tableName`.
|
||||
* Type: `string`
|
||||
@ -29,12 +29,12 @@ Type: `object`
|
||||
* Columns in this table.
|
||||
* Type: `array`
|
||||
* **Items**
|
||||
* $ref: [\#/definitions/column](table.md#column)
|
||||
* $ref: [#/definitions/column](table.md#column)
|
||||
* **tableConstraints**
|
||||
* Table constraints.
|
||||
* Type: `array`
|
||||
* **Items**
|
||||
* $ref: [\#/definitions/tableConstraint](table.md#tableconstraint)
|
||||
* $ref: [#/definitions/tableConstraint](table.md#tableconstraint)
|
||||
* **usageSummary**
|
||||
* Latest usage information for this table.
|
||||
* $ref: [../../type/usageDetails.json](../types/usagedetails.md)
|
||||
@ -43,13 +43,13 @@ Type: `object`
|
||||
* $ref: [../../type/entityReference.json](../types/entityreference.md)
|
||||
* **followers**
|
||||
* Followers of this table.
|
||||
* $ref: [../../type/entityReference.json\#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* **database**
|
||||
* Reference to Database that contains this table.
|
||||
* $ref: [../../type/entityReference.json](../types/entityreference.md)
|
||||
* **viewDefinition**
|
||||
* View Definition in SQL. Applies to TableType.View only.
|
||||
* $ref: [../../type/basic.json\#/definitions/sqlQuery](../types/basic.md#sqlquery)
|
||||
* $ref: [../../type/basic.json#/definitions/sqlQuery](../types/basic.md#sqlquery)
|
||||
* **tags**
|
||||
* Tags for this table.
|
||||
* Type: `array`
|
||||
@ -57,15 +57,15 @@ Type: `object`
|
||||
* $ref: [../../type/tagLabel.json](../types/taglabel.md)
|
||||
* **joins**
|
||||
* Details of other tables this table is frequently joined with.
|
||||
* $ref: [\#/definitions/tableJoins](table.md#tablejoins)
|
||||
* $ref: [#/definitions/tableJoins](table.md#tablejoins)
|
||||
* **sampleData**
|
||||
* Sample data for a table.
|
||||
* $ref: [\#/definitions/tableData](table.md#tabledata)
|
||||
* $ref: [#/definitions/tableData](table.md#tabledata)
|
||||
* **tableProfile**
|
||||
* Data profile for a table.
|
||||
* Type: `array`
|
||||
* **Items**
|
||||
* $ref: [\#/definitions/tableProfile](table.md#tableprofile)
|
||||
* $ref: [#/definitions/tableProfile](table.md#tableprofile)
|
||||
|
||||
## Type definitions in this schema
|
||||
|
||||
@ -125,9 +125,9 @@ Type: `object`
|
||||
* Type: `string`
|
||||
* The value is restricted to the following:
|
||||
1. _"NULL"_
|
||||
2. _"NOT\_NULL"_
|
||||
2. _"NOT_NULL"_
|
||||
3. _"UNIQUE"_
|
||||
4. _"PRIMARY\_KEY"_
|
||||
4. _"PRIMARY_KEY"_
|
||||
* Default: _"NULL"_
|
||||
|
||||
### tableConstraint
|
||||
@ -139,8 +139,8 @@ Type: `object`
|
||||
* Type: `string`
|
||||
* The value is restricted to the following:
|
||||
1. _"UNIQUE"_
|
||||
2. _"PRIMARY\_KEY"_
|
||||
3. _"FOREIGN\_KEY"_
|
||||
2. _"PRIMARY_KEY"_
|
||||
3. _"FOREIGN_KEY"_
|
||||
* **columns**
|
||||
* List of column names corresponding to the constraint.
|
||||
* Type: `array`
|
||||
@ -149,14 +149,14 @@ Type: `object`
|
||||
|
||||
### columnName
|
||||
|
||||
* Local name \(not fully qualified name\) of the column.
|
||||
* Local name (not fully qualified name) of the column.
|
||||
* Type: `string`
|
||||
* The value must match this pattern: `^[^.]*$`
|
||||
* Length: between 1 and 64
|
||||
|
||||
### tableName
|
||||
|
||||
* Local name \(not fully qualified name\) of a table.
|
||||
* Local name (not fully qualified name) of a table.
|
||||
* Type: `string`
|
||||
* The value must match this pattern: `^[^.]*$`
|
||||
* Length: between 1 and 64
|
||||
@ -173,15 +173,15 @@ Type: `object`
|
||||
* Type: `object`
|
||||
* **Properties**
|
||||
* **name** `required`
|
||||
* $ref: [\#/definitions/columnName](table.md#columnname)
|
||||
* $ref: [#/definitions/columnName](table.md#columnname)
|
||||
* **columnDataType** `required`
|
||||
* Data type of the column \(int, date etc.\).
|
||||
* $ref: [\#/definitions/columnDataType](table.md#columndatatype)
|
||||
* Data type of the column (int, date etc.).
|
||||
* $ref: [#/definitions/columnDataType](table.md#columndatatype)
|
||||
* **description**
|
||||
* Description of the column.
|
||||
* Type: `string`
|
||||
* **fullyQualifiedName**
|
||||
* $ref: [\#/definitions/fullyQualifiedColumnName](table.md#fullyqualifiedcolumnname)
|
||||
* $ref: [#/definitions/fullyQualifiedColumnName](table.md#fullyqualifiedcolumnname)
|
||||
* **tags**
|
||||
* Tags associated with the column.
|
||||
* Type: `array`
|
||||
@ -189,7 +189,7 @@ Type: `object`
|
||||
* $ref: [../../type/tagLabel.json](../types/taglabel.md)
|
||||
* **columnConstraint**
|
||||
* Column level constraint.
|
||||
* $ref: [\#/definitions/columnConstraint](table.md#columnconstraint)
|
||||
* $ref: [#/definitions/columnConstraint](table.md#columnconstraint)
|
||||
* **ordinalPosition**
|
||||
* Ordinal position of the column.
|
||||
* Type: `integer`
|
||||
@ -201,7 +201,7 @@ Type: `object`
|
||||
* This schema does not accept additional properties.
|
||||
* **Properties**
|
||||
* **columnName**
|
||||
* $ref: [\#/definitions/columnName](table.md#columnname)
|
||||
* $ref: [#/definitions/columnName](table.md#columnname)
|
||||
* **joinedWith**
|
||||
* Fully qualified names of the columns that this column is joined with.
|
||||
* Type: `array`
|
||||
@ -209,7 +209,7 @@ Type: `object`
|
||||
* Type: `object`
|
||||
* **Properties**
|
||||
* **fullyQualifiedName**
|
||||
* $ref: [\#/definitions/fullyQualifiedColumnName](table.md#fullyqualifiedcolumnname)
|
||||
* $ref: [#/definitions/fullyQualifiedColumnName](table.md#fullyqualifiedcolumnname)
|
||||
* **joinCount**
|
||||
* Type: `integer`
|
||||
|
||||
@ -221,14 +221,14 @@ Type: `object`
|
||||
* **Properties**
|
||||
* **startDate**
|
||||
* Date can be only from today going back to last 29 days.
|
||||
* $ref: [../../type/basic.json\#/definitions/date](../types/basic.md#date)
|
||||
* $ref: [../../type/basic.json#/definitions/date](../types/basic.md#date)
|
||||
* **dayCount**
|
||||
* Type: `integer`
|
||||
* Default: `1`
|
||||
* **columnJoins**
|
||||
* Type: `array`
|
||||
* **Items**
|
||||
* $ref: [\#/definitions/columnJoins](table.md#columnjoins)
|
||||
* $ref: [#/definitions/columnJoins](table.md#columnjoins)
|
||||
|
||||
### tableData
|
||||
|
||||
@ -237,10 +237,10 @@ Type: `object`
|
||||
* This schema does not accept additional properties.
|
||||
* **Properties**
|
||||
* **columns**
|
||||
* List of local column names \(not fully qualified column names\) of the table.
|
||||
* List of local column names (not fully qualified column names) of the table.
|
||||
* Type: `array`
|
||||
* **Items**
|
||||
* $ref: [\#/definitions/columnName](table.md#columnname)
|
||||
* $ref: [#/definitions/columnName](table.md#columnname)
|
||||
* **rows**
|
||||
* Data for multiple rows of the table.
|
||||
* Type: `array`
|
||||
@ -292,7 +292,7 @@ Type: `object`
|
||||
* **Properties**
|
||||
* **profileDate**
|
||||
* Data one which profile is taken.
|
||||
* $ref: [../../type/basic.json\#/definitions/date](../types/basic.md#date)
|
||||
* $ref: [../../type/basic.json#/definitions/date](../types/basic.md#date)
|
||||
* **columnCount**
|
||||
* No.of columns in the table.
|
||||
* Type: `number`
|
||||
@ -303,7 +303,6 @@ Type: `object`
|
||||
* List of local column profiles of the table.
|
||||
* Type: `array`
|
||||
* **Items**
|
||||
* $ref: [\#/definitions/columnProfile](table.md#columnprofile)
|
||||
* $ref: [#/definitions/columnProfile](table.md#columnprofile)
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the Tag Category entity. A Tag Category contains tags called Primary Tags. Primary Tags can further have children Tags called Secondary Tags. Only two levels of tags are supported currently.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/tags/tagCategory.json](https://open-metadata.org/schema/entity/tags/tagCategory.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/tags/tagCategory.json](https://open-metadata.org/schema/entity/tags/tagCategory.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -11,15 +11,15 @@ This schema does not accept additional properties.
|
||||
## Properties
|
||||
|
||||
* **name** `required`
|
||||
* $ref: [\#/definitions/tagName](tagcategory.md#tagname)
|
||||
* $ref: [#/definitions/tagName](tagcategory.md#tagname)
|
||||
* **description** `required`
|
||||
* Description of the tag category.
|
||||
* Type: `string`
|
||||
* **categoryType** `required`
|
||||
* $ref: [\#/definitions/tagCategoryType](tagcategory.md#tagcategorytype)
|
||||
* $ref: [#/definitions/tagCategoryType](tagcategory.md#tagcategorytype)
|
||||
* **href**
|
||||
* Link to the resource corresponding to the tag category.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
* **usageCount**
|
||||
* Count how many times the tags from this tag category are used.
|
||||
* Type: `integer`
|
||||
@ -27,7 +27,7 @@ This schema does not accept additional properties.
|
||||
* Tags under this category.
|
||||
* Type: `array`
|
||||
* **Items**
|
||||
* $ref: [\#/definitions/tag](tagcategory.md#tag)
|
||||
* $ref: [#/definitions/tag](tagcategory.md#tag)
|
||||
|
||||
## Type definitions in this schema
|
||||
|
||||
@ -48,4 +48,3 @@ This schema does not accept additional properties.
|
||||
### tag
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,16 +2,16 @@
|
||||
|
||||
This schema defines the Team entity. A Team is a group of zero or more users. Teams can own zero or more data assets.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/teams/team.json](https://open-metadata.org/schema/entity/teams/team.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/teams/team.json](https://open-metadata.org/schema/entity/teams/team.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
## Properties
|
||||
|
||||
* **id** `required`
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name** `required`
|
||||
* $ref: [\#/definitions/teamName](team.md#teamname)
|
||||
* $ref: [#/definitions/teamName](team.md#teamname)
|
||||
* **displayName**
|
||||
* Name used for display purposes. Example 'Data Science team'.
|
||||
* Type: `string`
|
||||
@ -20,7 +20,7 @@ Type: `object`
|
||||
* Type: `string`
|
||||
* **href** `required`
|
||||
* Link to the resource corresponding to this entity.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
* **profile**
|
||||
* Team profile information.
|
||||
* $ref: [../../type/profile.json](../types/profile.md)
|
||||
@ -29,10 +29,10 @@ Type: `object`
|
||||
* Type: `boolean`
|
||||
* **users**
|
||||
* Users that are part of the team.
|
||||
* $ref: [../../type/entityReference.json\#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* **owns**
|
||||
* List of entities owned by the team.
|
||||
* $ref: [../../type/entityReference.json\#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
|
||||
## Type definitions in this schema
|
||||
|
||||
@ -43,4 +43,3 @@ Type: `object`
|
||||
* Length: between 1 and 64
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the Thread entity. A Thread is a collection of posts made by the users. The first post that starts a thread is **about** a data asset **from** a user. Other users can respond to this post by creating new posts in the thread. Note that bot users can also interact with a thread. A post can contains links that mention Users or other Data Assets.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/feed/thread.json](https://open-metadata.org/schema/entity/feed/thread.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/feed/thread.json](https://open-metadata.org/schema/entity/feed/thread.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -10,23 +10,23 @@ Type: `object`
|
||||
|
||||
* **id** `required`
|
||||
* Unique identifier that identifies an entity instance.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **href**
|
||||
* Link to the resource corresponding to this entity.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
* **threadTs**
|
||||
* Timestamp of the when the first post created the thread.
|
||||
* $ref: [../../type/basic.json\#/definitions/dateTime](../types/basic.md#datetime)
|
||||
* $ref: [../../type/basic.json#/definitions/dateTime](../types/basic.md#datetime)
|
||||
* **about** `required`
|
||||
* Data asset about which this thread is created for with format <\#E/{enties}/{entityName}/{field}/{fieldValue}.
|
||||
* $ref: [../../type/basic.json\#/definitions/entityLink](../types/basic.md#entitylink)
|
||||
* Data asset about which this thread is created for with format <#E/{enties}/{entityName}/{field}/{fieldValue}.
|
||||
* $ref: [../../type/basic.json#/definitions/entityLink](../types/basic.md#entitylink)
|
||||
* **addressedTo**
|
||||
* User or team this thread is addressed to in format <\#E/{enties}/{entityName}/{field}/{fieldValue}.
|
||||
* $ref: [../../type/basic.json\#/definitions/entityLink](../types/basic.md#entitylink)
|
||||
* User or team this thread is addressed to in format <#E/{enties}/{entityName}/{field}/{fieldValue}.
|
||||
* $ref: [../../type/basic.json#/definitions/entityLink](../types/basic.md#entitylink)
|
||||
* **posts** `required`
|
||||
* Type: `array`
|
||||
* **Items**
|
||||
* $ref: [\#/definitions/post](thread.md#post)
|
||||
* $ref: [#/definitions/post](thread.md#post)
|
||||
|
||||
## Type definitions in this schema
|
||||
|
||||
@ -43,8 +43,7 @@ Type: `object`
|
||||
* Type: `string`
|
||||
* String format must be a "date-time"
|
||||
* **from** `required`
|
||||
* The ID of the User \(regular user or a bot\) posting the message.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* The ID of the User (regular user or a bot) posting the message.
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the Topic entity. A topic is a feed into which messages are published by publishers and read from by consumers in a messaging service.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/data/topic.json](https://open-metadata.org/schema/entity/data/topic.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/data/topic.json](https://open-metadata.org/schema/entity/data/topic.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -10,10 +10,10 @@ Type: `object`
|
||||
|
||||
* **id** `required`
|
||||
* Unique identifier that identifies this topic instance.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name** `required`
|
||||
* Name that identifies the topic.
|
||||
* $ref: [\#/definitions/topicName](topic.md#topicname)
|
||||
* $ref: [#/definitions/topicName](topic.md#topicname)
|
||||
* **fullyQualifiedName**
|
||||
* Name that uniquely identifies a topic in the format 'messagingServiceName.topicName'.
|
||||
* Type: `string`
|
||||
@ -32,17 +32,17 @@ Type: `object`
|
||||
* Type: `string`
|
||||
* **schemaType**
|
||||
* Schema used for message serialization.
|
||||
* $ref: [\#/definitions/schemaType](topic.md#schematype)
|
||||
* $ref: [#/definitions/schemaType](topic.md#schematype)
|
||||
* **cleanupPolicies**
|
||||
* Topic clean up policies. For Kafka - `cleanup.policy` configuration.
|
||||
* Type: `array`
|
||||
* **Items**
|
||||
* $ref: [\#/definitions/cleanupPolicy](topic.md#cleanuppolicy)
|
||||
* $ref: [#/definitions/cleanupPolicy](topic.md#cleanuppolicy)
|
||||
* **retentionTime**
|
||||
* Retention time in milliseconds. For Kafka - `retention.ms` configuration.
|
||||
* Type: `number`
|
||||
* **replicationFactor**
|
||||
* Replication Factor in integer \(more than 1\).
|
||||
* Replication Factor in integer (more than 1).
|
||||
* Type: `integer`
|
||||
* **maximumMessageSize**
|
||||
* Maximum message size in bytes. For Kafka - `max.message.bytes` configuration.
|
||||
@ -59,7 +59,7 @@ Type: `object`
|
||||
* $ref: [../../type/entityReference.json](../types/entityreference.md)
|
||||
* **followers**
|
||||
* Followers of this table.
|
||||
* $ref: [../../type/entityReference.json\#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* **tags**
|
||||
* Tags for this table.
|
||||
* Type: `array`
|
||||
@ -67,7 +67,7 @@ Type: `object`
|
||||
* $ref: [../../type/tagLabel.json](../types/taglabel.md)
|
||||
* **href**
|
||||
* Link to the resource corresponding to this entity.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
|
||||
## Type definitions in this schema
|
||||
|
||||
@ -95,4 +95,3 @@ Type: `object`
|
||||
2. _"compact"_
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
This schema defines the User entity. A user can be part of 0 or more teams. A special type of user called Bot is used for automation. A user can be an owner of zero or more data assets. A user can also follow zero or more data assets.
|
||||
|
||||
**$id:** [https://open-metadata.org/schema/entity/teams/user.json](https://open-metadata.org/schema/entity/teams/user.json)
|
||||
**$id: **[https://open-metadata.org/schema/entity/teams/user.json](https://open-metadata.org/schema/entity/teams/user.json)
|
||||
|
||||
Type: `object`
|
||||
|
||||
@ -12,18 +12,18 @@ This schema does not accept additional properties.
|
||||
|
||||
* **id** `required`
|
||||
* Unique identifier that identifies a user entity instance.
|
||||
* $ref: [../../type/basic.json\#/definitions/uuid](../types/basic.md#uuid)
|
||||
* $ref: [../../type/basic.json#/definitions/uuid](../types/basic.md#uuid)
|
||||
* **name** `required`
|
||||
* $ref: [\#/definitions/userName](user.md#username)
|
||||
* $ref: [#/definitions/userName](user.md#username)
|
||||
* **displayName**
|
||||
* Name used for display purposes. Example 'FirstName LastName'.
|
||||
* Type: `string`
|
||||
* **email** `required`
|
||||
* Email address of the user.
|
||||
* $ref: [../../type/basic.json\#/definitions/email](../types/basic.md#email)
|
||||
* $ref: [../../type/basic.json#/definitions/email](../types/basic.md#email)
|
||||
* **href** `required`
|
||||
* Link to the resource corresponding to this entity.
|
||||
* $ref: [../../type/basic.json\#/definitions/href](../types/basic.md#href)
|
||||
* $ref: [../../type/basic.json#/definitions/href](../types/basic.md#href)
|
||||
* **timezone**
|
||||
* The Timezone of the user.
|
||||
* Type: `string`
|
||||
@ -42,13 +42,13 @@ This schema does not accept additional properties.
|
||||
* $ref: [../../type/profile.json](../types/profile.md)
|
||||
* **teams**
|
||||
* Teams that the user belongs to.
|
||||
* $ref: [../../type/entityReference.json\#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* **owns**
|
||||
* List of entities owned by the user.
|
||||
* $ref: [../../type/entityReference.json\#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* **follows**
|
||||
* List of entities followed by the user.
|
||||
* $ref: [../../type/entityReference.json\#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
* $ref: [../../type/entityReference.json#/definitions/entityReferenceList](../types/entityreference.md#entityreferencelist)
|
||||
|
||||
## Type definitions in this schema
|
||||
|
||||
@ -59,4 +59,3 @@ This schema does not accept additional properties.
|
||||
* Length: between 1 and 64
|
||||
|
||||
_This document was updated on: Thursday, September 16, 2021_
|
||||
|
||||
|
||||