yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-09-30 11:26:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

removed the content from scripts

Browse Source
This commit is contained in:
parthp2107 2021-08-06 16:10:23 +05:30
parent f765bfe4a3
commit b71c75ba40
4 changed files with 49 additions and 209 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

52
docs/openmetadata-apis/schemas/entities/README.md
View File

@ -8,12 +8,12 @@ An Entity is a special type that has an identity and represents an object that i
| Abstract | Extensible | | Abstract | Extensible |
| :--- | :--- | | :--- | :--- |
| **id** | Mandatory attribute of type UUID that identifies the entity instance | | **id** | Mandatory attribute of type `UUID` that identifies the entity instance |
| **name** | Name of the entity \(example database name\). For some entities, the name may uniquely identify an entity. | | **name** | Name of the entity \(example database name\). For some entities, the name may uniquely identify an entity. |
| **fullyQualifiedName** | Human-readable name that uniquely identifies an entity that is formed using all the names in the hierarchy above the given entity. Example - Attributes of an entity may also have to uniquely identify a field. For example, a column of a table has attribute set to | | **fullyQualifiedName** | Human-readable name that uniquely identifies an entity that is formed using all the names in the hierarchy above the given entity. Example - `databaseService.database.table.` Attributes of an entity may also have `FQN` to uniquely identify a field. For example, a column of a table has `fqn` attribute set to `databaseService.database.table.columnName.` |
| **displayName** | Optional name used for display purposes. For example, the name could be john.smith@domain.com and displayName could be John Smith. | | **displayName** | Optional name used for display purposes. For example, the name could be`john.smith@domain.com` and `displayName` could be `John Smith.` |
| **description** | Description of the entity instance. Not all entities need a description. For example, a User entity might not need a description and just the name of the user might suffice. A entity needs to provide details of what is stored in the database when to use it and other information on how to use it. | | **description** | Description of the entity instance. Not all entities need a description. For example, a User entity might not need a description and just the name of the user might suffice. A `Database` entity needs `description` to provide details of what is stored in the database when to use it and other information on how to use it. |
| **Owner** | Optional attribute used to capture the ownership information. Not all entities have ownership information \(for example User, Team, and Organization\). | | **Owner** | Optional attribute used to capture the ownership information. Not all entities have ownership information \(for example `User, Team`, and `Organization`\). |
| **href** | An attribute generated on the fly as part of API response to provide the URL link to the entity returned. | | **href** | An attribute generated on the fly as part of API response to provide the URL link to the entity returned. |
### **Relationships** ### **Relationships**
@ -24,16 +24,49 @@ An Entity is a special type that has an identity and represents an object that i
* One to Many: a Database contains multiple Tables. * One to Many: a Database contains multiple Tables.
* Many-to-Many: A User belongs to multiple Teams. A team has multiple Users. * Many-to-Many: A User belongs to multiple Teams. A team has multiple Users.
All relationships are captured using the EntityReference type. All relationships are captured using the `EntityReference` type.
Following is an example of a JSON schema of the User entity with attributes id, displayName, and email. User entity has one-to-many relationships to another entity Team \(user is member of multiple teams\).: is a special shell builtin string: not found Following is an example of a JSON schema of the User entity with attributes id, displayName, and email. User entity has one-to-many relationships to another entity Team (user is member of multiple teams).
```javascript
{
"title": "User entity",
"type": "object",
"properties" : {
"id": {
"description": "Unique identifier for instance of a User",
"$ref": "#/definitions/uuid"
},
"displayName": {
"description": "Name used for display purposes. Example 'John Smith'",
"type" : "string"
},
"email": {
"description": "User's Email",
"type": "string"
},
"teams" : {
"description": "Teams that this user belongs to",
"type": "array",
"items" :{
"$ref": "#/definitions/entityReference"
}
}
}
}
```
## Metadata system entities ## Metadata system entities
Metadata system has the following core entities: 1. **Data Entities** - These entities represent data, such as databases, tables, and topics, and assets created using data, such as Dashboards, Reports, Metrics, and ML Features. It also includes entities such as Pipelines that are used for creating data assets. 2. **Services** - Services represent platforms and services used for storing and processing data. It includes Online Data Stores, Data Warehouses, ETL tools, Dashboard services, etc. 3. **Users & Teams** - These entities represent users within an organization and teams that they are organized under. 4. **Activities** - These entities are related to feeds, posts, and notifications for collaboration between users. 5. **Glossary and Tags** - Entities for defining business glossary that includes hierarchical tags. Metadata system has the following core entities:
1. **Data Entities** - These entities represent data, such as databases, tables, and topics, and assets created using data, such as Dashboards, Reports, Metrics, and ML Features. It also includes entities such as Pipelines that are used for creating data assets.
2. **Services** - Services represent platforms and services used for storing and processing data. It includes Online Data Stores, Data Warehouses, ETL tools, Dashboard services, etc.
3. **Users & Teams** - These entities represent users within an organization and teams that they are organized under.
4. **Activities** - These entities are related to feeds, posts, and notifications for collaboration between users.
5. **Glossary and Tags** - Entities for defining business glossary that includes hierarchical tags.
## List of Schema Entities ## List of Schema Entities
{% page-ref page="bots.md" %} {% page-ref page="bots.md" %}
{% page-ref page="dashboard.md" %} {% page-ref page="dashboard.md" %}
@ -57,4 +90,3 @@ Metadata system has the following core entities: 1. **Data Entities** - These en
{% page-ref page="team.md" %} {% page-ref page="team.md" %}
{% page-ref page="user.md" %} {% page-ref page="user.md" %}

6
docs/openmetadata-apis/schemas/types/README.md
View File

@ -1,9 +1,8 @@
# Types # Types
JSON schema supports many native types - null, boolean, object, array, number and string. In addition, to develop clear and consistent vocabulary, domain-specific reusable types are defined ranging from simple types, such as UUID, timestamp, and email to more complex object types, such as Tags, Ownership and Usage. JSON schema supports many native types - `null`, `boolean`, `object`, `array`, `number` and `string`. In addition, to develop clear and consistent vocabulary, domain-specific reusable types are defined ranging from simple types, such as `UUID`, `timestamp`, and `email` to more complex object types, such as `Tags`, `Ownership` and `Usage`.
## List of Schema Types ## List of Schema Types
{% page-ref page="basic.md" %} {% page-ref page="basic.md" %}
{% page-ref page="collectiondescriptor.md" %} {% page-ref page="collectiondescriptor.md" %}
@ -12,6 +11,8 @@ JSON schema supports many native types - null, boolean, object, array, number an
{% page-ref page="entityreference.md" %} {% page-ref page="entityreference.md" %}
{% page-ref page="entityusage.md" %}
{% page-ref page="jdbcconnection.md" %} {% page-ref page="jdbcconnection.md" %}
{% page-ref page="profile.md" %} {% page-ref page="profile.md" %}
@ -21,4 +22,3 @@ JSON schema supports many native types - null, boolean, object, array, number an
{% page-ref page="taglabel.md" %} {% page-ref page="taglabel.md" %}
{% page-ref page="usagedetails.md" %} {% page-ref page="usagedetails.md" %}

32
scripts/editContent.sh
View File

@ -1,35 +1,3 @@
echo "----------------------Changing file content----------------------"
path=docs/openmetadata-apis/schemas/README.md
sed -i d $path
echo "# Schemas
## Schema Modeling
OpenMetadata takes the **schema-first** approach to model metadata as a Single Source of Truth with clear vocabulary for the system. First, the Entities and Types in the system are identified and schemas are defined for them. Code is then generated from these schemas and is used in implementing the system. We use [JSON schema](https://json-schema.org/) as the Schema Definition Language as it offers several advantages:
* Easy to describe the structure and semantics of metadata models with readable documentation that is both human and machine consumable.
* Common types can be developed once and can be reused as building blocks in other schemas and become the basis of vocabulary development.
* Models can include structural validation, such as required/optional fields, default values, allowed values, regex that not only serve as automated testing and validation but also as documentation of API contract.
* A rich set of tools are availble that supports JSON schema support for generating code and validation from JSON schema in various languages, reducing the manual boilerplate coding.
* Supports rich formats to convert schema types into native standard types during code generation, such as URI, date, and time.
## Reference
1. [JSON schema](https://json-schema.org/) specification version [Draft-07 to 2019-099](https://json-schema.org/draft/2019-09/release-notes.html)
2. [JSON schema 2 POJO](https://www.jsonschema2pojo.org/) tool used for Java code generation from JSON schema
3. [Data model code generator](https://github.com/koxudaxi/datamodel-code-generator) for generating python code from JSON schema
## Schemas
{% page-ref page="\"entities\"" %}
{% page-ref page="\"types\"" %}
## Version Note
The schemas linked above follow the JSON Schema Spec version: http://json-schema.org/draft-07/schema#" >> $path
echo "----------------------Changing paths----------------------" echo "----------------------Changing paths----------------------"
path=docs/openmetadata-apis/schemas/*/*.md path=docs/openmetadata-apis/schemas/*/*.md
sed -i -e 's#../../out/entity/bots.json#https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/schema/entity/bots.json#g' \ sed -i -e 's#../../out/entity/bots.json#https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/schema/entity/bots.json#g' \

164
scripts/generateDirectories.sh
View File

@ -1,163 +1,4 @@
filename="entities" echo "------------------Merging Files------------------"
path=docs/openmetadata-apis/schemas/$filename
if [ ! -d $path ]
then
mkdir $path
echo "File not found. File created $filename"
else
echo "File exists $filename"
fi
filename="types"
path=docs/openmetadata-apis/schemas/$filename
if [ ! -d $path ]
then
mkdir $path
echo "File not found. File created $filename"
else
echo "File exists $filename"
fi
filename="README.md"
path=docs/openmetadata-apis/schemas/entities/$filename
if [ ! -d $path ]
then
echo >> $path
echo "File not found. File created $filename"
else
echo "File exists $filename"
fi
sed -i d $path
echo "# Entities
An Entity is a special type that has an identity and represents an object that is either real or conceptual. An entity can be related to another entity through relationships. An Entity has two types of **Fields** - **Attributes** and **Relationships**:
### **Attributes**
**Attributes** represent an Entity’s data. Entities MUST include an attribute called **ID** that uniquely identifies an instance of an entity. It might optionally include a human-readable **fullyQualitifedName** attribute that uniquely identifies the entity. An attribute of an entity MUST not be another Entity and should be captured through a relationship. Entities typically SHOULD have the following common attributes:
| Abstract | Extensible |
| :--- | :--- |
| **id** | Mandatory attribute of type UUID that identifies the entity instance |
| **name** | Name of the entity \(example database name\). For some entities, the name may uniquely identify an entity. |
| **fullyQualifiedName** | Human-readable name that uniquely identifies an entity that is formed using all the names in the hierarchy above the given entity. Example - `databaseService.database.table.` Attributes of an entity may also have `FQN` to uniquely identify a field. For example, a column of a table has `fqn` attribute set to `databaseService.database.table.columnName.` |
| **displayName** | Optional name used for display purposes. For example, the name could be john.smith@domain.com and displayName could be John Smith. |
| **description** | Description of the entity instance. Not all entities need a description. For example, a User entity might not need a description and just the name of the user might suffice. A `Database` entity needs `description` to provide details of what is stored in the database when to use it and other information on how to use it. |
| **Owner** | Optional attribute used to capture the ownership information. Not all entities have ownership information \(for example User, Team, and Organization). |
| **href** | An attribute generated on the fly as part of API response to provide the URL link to the entity returned. |
### **Relationships**
**Relationships** capture information about the association of an Entity with another Entity. Relationships can have cardinality - **One-to-one**, **One-to-many**, **Many-to-one**, and **Many-to-many**. Example of relationships:
* One-to-one: A Table is owned by a User
* One to Many: a Database contains multiple Tables.
* Many-to-Many: A User belongs to multiple Teams. A team has multiple Users.
All relationships are captured using the EntityReference type.
Following is an example of a JSON schema of the User entity with attributes id, displayName, and email. User entity has one-to-many relationships to another entity Team \(user is member of multiple teams\).
```javascript
{
"title": "User entity",
"type": "object",
"properties" : {
"id": {
"description": "Unique identifier for instance of a User",
"$ref": "#/definitions/uuid"
},
"displayName": {
"description": "Name used for display purposes. Example 'John Smith'",
"type" : "string"
},
"email": {
"description": "User's Email",
"type": "string"
},
"teams" : {
"description": "Teams that this user belongs to",
"type": "array",
"items" :{
"$ref": "#/definitions/entityReference"
}
}
}
}
```
## Metadata system entities
Metadata system has the following core entities:
1. **Data Entities** - These entities represent data, such as databases, tables, and topics, and assets created using data, such as Dashboards, Reports, Metrics, and ML Features. It also includes entities such as Pipelines that are used for creating data assets.
2. **Services** - Services represent platforms and services used for storing and processing data. It includes Online Data Stores, Data Warehouses, ETL tools, Dashboard services, etc.
3. **Users & Teams** - These entities represent users within an organization and teams that they are organized under.
4. **Activities** - These entities are related to feeds, posts, and notifications for collaboration between users.
5. **Glossary and Tags** - Entities for defining business glossary that includes hierarchical tags.
## List of Schema Entities
{% page-ref page="\"bots.md\"" %}
{% page-ref page="\"dashboard.md\"" %}
{% page-ref page="\"database.md\"" %}
{% page-ref page="\"databaseservice.md\"" %}
{% page-ref page="\"thread.md\"" %}
{% page-ref page="\"metrics.md\"" %}
{% page-ref page="\"pipeline.md\"" %}
{% page-ref page="\"report.md\"" %}
{% page-ref page="\"table.md\"" %}
{% page-ref page="\"tagcategory.md\"" %}
{% page-ref page="\"team.md\"" %}
{% page-ref page="\"user.md\"" %}" >> $path
filename="README.md"
path=docs/openmetadata-apis/schemas/types/$filename
if [ ! -d $path ]
then
echo >> $path
echo "File not found. File created $filename"
else
echo "File exists $filename"
fi
sed -i d $path
echo "# Types
JSON schema supports many native types - null, boolean, object, array, number and string. In addition, to develop clear and consistent vocabulary, domain-specific reusable types are defined ranging from simple types, such as UUID, timestamp, and email to more complex object types, such as Tags, Ownership and Usage.
## List of Schema Types
{% page-ref page="\"basic.md\"" %}
{% page-ref page="\"collectiondescriptor.md\"" %}
{% page-ref page="\"dailycount.md\"" %}
{% page-ref page="\"entityreference.md\"" %}
{% page-ref page="\"entityusage.md\"" %}
{% page-ref page="\"jdbcconnection.md\"" %}
{% page-ref page="\"profile.md\"" %}
{% page-ref page="\"schedule.md\"" %}
{% page-ref page="\"taglabel.md\"" %}
{% page-ref page="\"usagedetails.md\"" %}" >> $path
cd docs/openmetadata-apis/schemas/SchemaMarkdown cd docs/openmetadata-apis/schemas/SchemaMarkdown
cat bots-* >> bots.md cat bots-* >> bots.md
cat dashboard-* >> dashboard.md cat dashboard-* >> dashboard.md
@ -181,10 +22,9 @@ cat profile-* >> profile.md
cat schedule-* >> schedule.md cat schedule-* >> schedule.md
cat taglabel-* >> taglabel.md cat taglabel-* >> taglabel.md
cat usagedetails-* >> usagedetails.md cat usagedetails-* >> usagedetails.md
echo "------------------Files Merged------------------"
echo "------------------Moving files------------------" echo "------------------Moving files------------------"
cp /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas/SchemaMarkdown/README.md /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas
cp /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas/SchemaMarkdown/bots.md /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas/entities cp /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas/SchemaMarkdown/bots.md /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas/entities
cp /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas/SchemaMarkdown/dashboard.md /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas/entities cp /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas/SchemaMarkdown/dashboard.md /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas/entities
cp /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas/SchemaMarkdown/database.md /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas/entities cp /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas/SchemaMarkdown/database.md /home/runner/work/OpenMetadata/OpenMetadata/docs/openmetadata-apis/schemas/entities