From 612c08361234f4e654535d9394aeeb7802cb4ef9 Mon Sep 17 00:00:00 2001 From: Sriharsha Chintalapani Date: Wed, 28 Sep 2022 14:05:51 -0700 Subject: [PATCH] Fix Doc links (#7734) * Fix Broken links * Fix symlink Co-authored-by: Pere Miquel Brull --- ingestion/examples/workflows | 1 + .../how-to-add-a-new-implementation/index.md | 6 +++--- .../content/developers/architecture/code-layout.md | 12 ++++++------ .../build-code-and-run-tests/ingestion-framework.md | 2 +- .../build-code-and-run-tests/prerequisites.md | 4 ++-- .../openmetadata/connectors/credentials/index.md | 2 +- .../connectors/dashboard/powerbi/airflow.md | 2 +- .../openmetadata/connectors/dashboard/powerbi/cli.md | 2 +- .../connectors/database/azuresql/airflow.md | 4 ++-- .../connectors/database/bigquery/airflow.md | 4 ++-- .../openmetadata/connectors/database/bigquery/cli.md | 4 ++-- .../connectors/database/deltalake/airflow.md | 2 +- .../connectors/database/deltalake/cli.md | 2 +- .../openmetadata/connectors/database/index.md | 2 +- .../connectors/database/mariadb/airflow.md | 4 ++-- .../openmetadata/connectors/database/mariadb/cli.md | 4 ++-- .../connectors/database/postgres/index.md | 10 +++++----- .../openmetadata/connectors/pipeline/glue/index.md | 4 ++-- .../ingestion/workflows/data-quality/index.md | 2 +- .../openmetadata/integrations/great-expectations.md | 6 +++--- .../content/sdk/python/entities/ml-model.md | 4 ++-- 21 files changed, 42 insertions(+), 41 deletions(-) create mode 120000 ingestion/examples/workflows diff --git a/ingestion/examples/workflows b/ingestion/examples/workflows new file mode 120000 index 00000000000..14ea8b54b23 --- /dev/null +++ b/ingestion/examples/workflows @@ -0,0 +1 @@ +../src/metadata/examples/workflows \ No newline at end of file diff --git a/openmetadata-docs/content/deployment/secrets-manager/how-to-add-a-new-implementation/index.md b/openmetadata-docs/content/deployment/secrets-manager/how-to-add-a-new-implementation/index.md index ea514bd92b8..659c359ab2a 100644 --- a/openmetadata-docs/content/deployment/secrets-manager/how-to-add-a-new-implementation/index.md +++ b/openmetadata-docs/content/deployment/secrets-manager/how-to-add-a-new-implementation/index.md @@ -28,7 +28,7 @@ You can find [this](https://github.com/open-metadata/OpenMetadata/blob/main/open ## 2. Update OM Server code -Once we have updated the JSON Schema, we can start implementing our Secrets Manager, extending the `ThirdPartySecretsManager.java` abstract class located [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/catalog/secrets/ThirdPartySecretsManager.java). For example: +Once we have updated the JSON Schema, we can start implementing our Secrets Manager, extending the `ThirdPartySecretsManager.java` abstract class located [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/secrets/ThirdPartySecretsManager.java). For example: ```java public abstract class AwesomeSecretsManager extends ThirdPartySecretsManager { @@ -50,7 +50,7 @@ public abstract class AwesomeSecretsManager extends ThirdPartySecretsManager { } ``` -After this, we can update `SecretsManagerFactory.java` which is a factory class. We can find this file [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/catalog/secrets/SecretsManagerFactory.java). +After this, we can update `SecretsManagerFactory.java` which is a factory class. We can find this file [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/secrets/SecretsManagerFactory.java). ```java ... @@ -87,4 +87,4 @@ Similar to what we did in step 2, we have to add our implementation to the facto ```

-If you need support while implementing your Secret Manager client, do not hesitate to reach out to us on [Slack](https://slack.open-metadata.org/). \ No newline at end of file +If you need support while implementing your Secret Manager client, do not hesitate to reach out to us on [Slack](https://slack.open-metadata.org/). diff --git a/openmetadata-docs/content/developers/architecture/code-layout.md b/openmetadata-docs/content/developers/architecture/code-layout.md index e013c209af4..52b3c9878b9 100644 --- a/openmetadata-docs/content/developers/architecture/code-layout.md +++ b/openmetadata-docs/content/developers/architecture/code-layout.md @@ -31,7 +31,7 @@ All OpenMetadata supported types are defined under [`OpenMetadata/openmetadata-s The API request objects are defined under [`OpenMetadata/openmetadata-spec/src/main/resources/json/schema/api`](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/api). ## API -OpenMetadata uses the [Dropwizard](https://www.dropwizard.io/) Java framework to build REST APIs. You can locate defined APIs in the directory [`OpenMetadata/openmetadata-service/src/main/java/org/openmetadata/catalog/resources`](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-service/src/main/java/org/openmetadata/catalog/resources). OpenMetadata uses [Swagger](https://swagger.io/) to generate API documentation following OpenAPI standards. +OpenMetadata uses the [Dropwizard](https://www.dropwizard.io/) Java framework to build REST APIs. You can locate defined APIs in the directory [`OpenMetadata/openmetadata-service/src/main/java/org/openmetadata/service/resources`](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-service/src/main/java/org/openmetadata/service/resources). OpenMetadata uses [Swagger](https://swagger.io/) to generate API documentation following OpenAPI standards. ## System and Components @@ -40,20 +40,20 @@ OpenMetadata uses the [Dropwizard](https://www.dropwizard.io/) Java framework to ### Events OpenMetadata captures changes to entities as `events` and stores them in the OpenMetadata server database. OpenMetadata also indexes change events in Elasticsearch to make them searchable. -The event handlers are defined under [`OpenMetadata/openmetadata-service/src/main/java/org/openmetadata/catalog/events`](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-service/src/main/java/org/openmetadata/catalog/events) and are applied globally to any outgoing response using the `ContainerResponseFilter`. +The event handlers are defined under [`OpenMetadata/openmetadata-service/src/main/java/org/openmetadata/service/events`](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-service/src/main/java/org/openmetadata/service/events) and are applied globally to any outgoing response using the `ContainerResponseFilter`. ### Database -OpenMetadata uses MySQL for the metadata catalog. The catalog code is located in the directory [`OpenMetadata/openmetadata-service/src/main/java/org/openmetadata/catalog/jdbi3`](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-service/src/main/java/org/openmetadata/catalog/jdbi3). +OpenMetadata uses MySQL for the metadata catalog. The catalog code is located in the directory [`OpenMetadata/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3`](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3). The database entity tables are created using the command [`OpenMetadata/bootstrap/bootstrap_storage.sh`](https://github.com/open-metadata/OpenMetadata/blob/main/bootstrap/bootstrap_storage.sh). [Flyway](https://flywaydb.org/) is used for managing the database table versions. ### Elasticsearch -OpenMetadata uses Elasticsearch to store the Entity change events and makes it searchable by search index. The [`OpenMetadata/openmetadata-service/src/main/java/org/openmetadata/catalog/elasticsearch/ElasticSearchEventHandler.java`](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/catalog/elasticsearch/ElasticSearchEventHandler.java) is responsible for capturing the change events and updating Elasticsearch. +OpenMetadata uses Elasticsearch to store the Entity change events and makes it searchable by search index. The [`OpenMetadata/openmetadata-service/src/main/java/org/openmetadata/catalog/elasticsearch/ElasticSearchEventHandler.java`](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/elasticsearch/ElasticSearchEventHandler.java) is responsible for capturing the change events and updating Elasticsearch. Elasticsearch indices are created when the [`OpenMetadata/ingestion/pipelines/metadata_to_es.json`](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/pipelines/metadata_to_es.json) ingestion connector is run. ### Authentication/Authorization -OpenMetadata uses Google OAuth for authentication. All incoming requests are filtered by validating the JWT token using the Google OAuth provider. Access control is provided by [`Authorizer`](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/catalog/security/Authorizer.java). +OpenMetadata uses Google OAuth for authentication. All incoming requests are filtered by validating the JWT token using the Google OAuth provider. Access control is provided by [`Authorizer`](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/security/Authorizer.java). See the configuration file `OpenMetadata` [`/conf/openmetadata.yaml`](https://github.com/open-metadata/OpenMetadata/blob/main/conf/openmetadata.yaml) for the authentication and authorization configurations. @@ -89,4 +89,4 @@ See the directory [`OpenMetadata/ingestion/examples/airflow/dags`](https://githu **JsonSchema python typings** -You can generate Python types for OpenMetadata models defined using Json Schema using the make generate command of the [`Makefile`](https://github.com/open-metadata/OpenMetadata/blob/main/Makefile/README.md). Generated files are located in the directory `OpenMetadata/ingestion/src/metadata/generated` \ No newline at end of file +You can generate Python types for OpenMetadata models defined using Json Schema using the make generate command of the [`Makefile`](https://github.com/open-metadata/OpenMetadata/blob/main/Makefile/README.md). Generated files are located in the directory `OpenMetadata/ingestion/src/metadata/generated` diff --git a/openmetadata-docs/content/developers/contribute/build-code-and-run-tests/ingestion-framework.md b/openmetadata-docs/content/developers/contribute/build-code-and-run-tests/ingestion-framework.md index bf88ea47458..9c5dbc86a23 100644 --- a/openmetadata-docs/content/developers/contribute/build-code-and-run-tests/ingestion-framework.md +++ b/openmetadata-docs/content/developers/contribute/build-code-and-run-tests/ingestion-framework.md @@ -37,7 +37,7 @@ Once you have generated the sources, you should be able to run the tests and the ### Quality tools When working on the Ingestion Framework, you might want to take into consideration the following style-check tooling: -- [pylint](https://www.pylint.org/) is a Static Code Analysis tool to catch errors, align coding standards and help us follow conventions and apply improvements. +- [pylint](https://pylint.pycqa.org/en/latest/) is a Static Code Analysis tool to catch errors, align coding standards and help us follow conventions and apply improvements. - [black](https://black.readthedocs.io/en/stable/) can be used to both autoformat the code and validate that the codebase is compliant. - [isort](https://pycqa.github.io/isort/) helps us not lose time trying to find the proper combination of importing from `stdlib`, requirements, project files… diff --git a/openmetadata-docs/content/developers/contribute/build-code-and-run-tests/prerequisites.md b/openmetadata-docs/content/developers/contribute/build-code-and-run-tests/prerequisites.md index 15d88a1a92b..0162d5aa14f 100644 --- a/openmetadata-docs/content/developers/contribute/build-code-and-run-tests/prerequisites.md +++ b/openmetadata-docs/content/developers/contribute/build-code-and-run-tests/prerequisites.md @@ -14,7 +14,7 @@ OpenMetadata being a full stack project, we use the following for development: - [Python 3.7 or higher](https://www.python.org/downloads/) - [Node >=10.0.0](https://nodejs.org/en/download/) - [Yarn ^1.22.0](https://classic.yarnpkg.com/lang/en/docs/install/) -- [Rpm (Optional, only to run RPM profile with maven)](https://brewinstall.org/install-rpm-on-mac-with-brew/) +- [Rpm (Optional, only to run RPM profile with maven)](https://macappstore.org/rpm/) - Antlr 4.9.2 - `sudo make install_antlr_cli` - Here is a snapshot of a working environment on a Macbook. @@ -76,4 +76,4 @@ We use Java for developing OpenMetadata backend server. Following are the key te - [jsonschema2pojo](https://www.jsonschema2pojo.org/) for Java code generation - [Dropwizard](https://www.dropwizard.io/en/latest/) for the web service application -- [JDBI3](http://jdbi.org/) for database access \ No newline at end of file +- [JDBI3](http://jdbi.org/) for database access diff --git a/openmetadata-docs/content/openmetadata/connectors/credentials/index.md b/openmetadata-docs/content/openmetadata/connectors/credentials/index.md index 85f13249da1..068e93b377d 100644 --- a/openmetadata-docs/content/openmetadata/connectors/credentials/index.md +++ b/openmetadata-docs/content/openmetadata/connectors/credentials/index.md @@ -63,4 +63,4 @@ Once the account is created, you can see the fields in the exported JSON file fr IAM & Admin > Service Accounts > Keys ``` -You can validate the whole Google service account setup [here](deployment/security/google). +You can validate the whole Google service account setup [here](/deployment/security/google). diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/airflow.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/airflow.md index b964f18dd6b..cfaf9bbc64c 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/airflow.md @@ -31,7 +31,7 @@ pip3 install "openmetadata-ingestion[powerbi]" ## Metadata Ingestion All connectors are defined as JSON Schemas. -[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/powerbiConnection.json) +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/powerBIConnection.json) you can find the structure to create a connection to PowerBI. In order to create and run a Metadata Ingestion workflow, we will follow diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/cli.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/cli.md index ba6a463a7e3..d6b6af38491 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/cli.md @@ -31,7 +31,7 @@ pip3 install "openmetadata-ingestion[powerbi]" ## Metadata Ingestion All connectors are defined as JSON Schemas. -[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/powerbiConnection.json) +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/powerBIConnection.json) you can find the structure to create a connection to PowerBI. In order to create and run a Metadata Ingestion workflow, we will follow diff --git a/openmetadata-docs/content/openmetadata/connectors/database/azuresql/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/azuresql/airflow.md index 8303da9a604..e0fe1dc65f1 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/azuresql/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/azuresql/airflow.md @@ -33,7 +33,7 @@ pip3 install "openmetadata-ingestion[azuresql]" ## Metadata Ingestion All connectors are defined as JSON Schemas. -[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/azuresqlConnection.json) +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/azureSQLConnection.json) you can find the structure to create a connection to AzureSQL. In order to create and run a Metadata Ingestion workflow, we will follow @@ -434,7 +434,7 @@ workflowConfig: #### Source Configuration -- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/azuresqlConnection.json). +- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/azureSQLConnection.json). - The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json). Note that the filter patterns support regex as includes or excludes. E.g., diff --git a/openmetadata-docs/content/openmetadata/connectors/database/bigquery/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/bigquery/airflow.md index 49f93350095..2319d81a0f9 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/bigquery/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/bigquery/airflow.md @@ -59,7 +59,7 @@ pip3 install "openmetadata-ingestion[bigquery-usage]" ## Metadata Ingestion All connectors are defined as JSON Schemas. -[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/bigqueryConnection.json) +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/bigQueryConnection.json) you can find the structure to create a connection to BigQuery. In order to create and run a Metadata Ingestion workflow, we will follow @@ -604,7 +604,7 @@ workflowConfig: #### Source Configuration -- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/bigqueryConnection.json). +- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/bigQueryConnection.json). - The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json). Note that the filter patterns support regex as includes or excludes. E.g., diff --git a/openmetadata-docs/content/openmetadata/connectors/database/bigquery/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/bigquery/cli.md index 487b272d1b3..a1746db240d 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/bigquery/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/bigquery/cli.md @@ -59,7 +59,7 @@ pip3 install "openmetadata-ingestion[bigquery-usage]" ## Metadata Ingestion All connectors are defined as JSON Schemas. -[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/bigqueryConnection.json) +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/bigQueryConnection.json) you can find the structure to create a connection to BigQuery. In order to create and run a Metadata Ingestion workflow, we will follow @@ -561,7 +561,7 @@ workflowConfig: #### Source Configuration -- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/bigqueryConnection.json). +- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/bigQueryConnection.json). - The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json). Note that the filter patterns support regex as includes or excludes. E.g., diff --git a/openmetadata-docs/content/openmetadata/connectors/database/deltalake/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/deltalake/airflow.md index 1199c0fdb6f..8ff7d3e06c3 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/deltalake/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/deltalake/airflow.md @@ -32,7 +32,7 @@ pip3 install "openmetadata-ingestion[deltalake]" ## Metadata Ingestion All connectors are defined as JSON Schemas. -[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/deltalakeConnection.json) +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/deltaLakeConnection.json) you can find the structure to create a connection to Deltalake. In order to create and run a Metadata Ingestion workflow, we will follow diff --git a/openmetadata-docs/content/openmetadata/connectors/database/deltalake/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/deltalake/cli.md index 1c6532595c2..5d89ebadf26 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/deltalake/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/deltalake/cli.md @@ -32,7 +32,7 @@ pip3 install "openmetadata-ingestion[deltalake]" ## Metadata Ingestion All connectors are defined as JSON Schemas. -[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/deltalakeConnection.json) +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/deltaLakeConnection.json) you can find the structure to create a connection to Deltalake. In order to create and run a Metadata Ingestion workflow, we will follow diff --git a/openmetadata-docs/content/openmetadata/connectors/database/index.md b/openmetadata-docs/content/openmetadata/connectors/database/index.md index 1c8947fbdba..29b339f3a11 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/index.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/index.md @@ -13,7 +13,7 @@ slug: /openmetadata/connectors/database - [DB2](/openmetadata/connectors/database/db2) - [DeltaLake](/openmetadata/connectors/database/deltalake) - [Druid](/openmetadata/connectors/database/druid) -- [DynamoDB](/openmetadata/connectors/database/dynamocb) +- [DynamoDB](/openmetadata/connectors/database/dynamodb) - [Glue](/openmetadata/connectors/database/glue) - [Hive](/openmetadata/connectors/database/hive) - [MariaDB](/openmetadata/connectors/database/mariadb) diff --git a/openmetadata-docs/content/openmetadata/connectors/database/mariadb/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/mariadb/airflow.md index 87018595ee5..859bc5fa133 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/mariadb/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/mariadb/airflow.md @@ -33,7 +33,7 @@ pip3 install "openmetadata-ingestion[mariadb]" ## Metadata Ingestion All connectors are defined as JSON Schemas. -[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/mariadbConnection.json) +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/mariaDBConnection.json) you can find the structure to create a connection to MariaDB. In order to create and run a Metadata Ingestion workflow, we will follow @@ -429,7 +429,7 @@ workflowConfig: #### Source Configuration -- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/mariadbConnection.json). +- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/mariaDBConnection.json). - The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json). Note that the filter patterns support regex as includes or excludes. E.g., diff --git a/openmetadata-docs/content/openmetadata/connectors/database/mariadb/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/mariadb/cli.md index 0e5dda26515..ec40f4e5ecf 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/mariadb/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/mariadb/cli.md @@ -33,7 +33,7 @@ pip3 install "openmetadata-ingestion[mariadb]" ## Metadata Ingestion All connectors are defined as JSON Schemas. -[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/mariadbConnection.json) +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/mariaDBConnection.json) you can find the structure to create a connection to MariaDB. In order to create and run a Metadata Ingestion workflow, we will follow @@ -382,7 +382,7 @@ workflowConfig: #### Source Configuration -- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/mariadbConnection.json). +- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/mariaDBConnection.json). - The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json). Note that the filter patterns support regex as includes or excludes. E.g., diff --git a/openmetadata-docs/content/openmetadata/connectors/database/postgres/index.md b/openmetadata-docs/content/openmetadata/connectors/database/postgres/index.md index a6dc765ea93..fbf3de7b1ee 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/postgres/index.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/postgres/index.md @@ -21,14 +21,14 @@ the following docs to connect using Airflow SDK or with the CLI. icon="air" title="Ingest with Airflow" text="Configure the ingestion using Airflow SDK" - link="/openmetadata/connectors/database/postgresql/airflow" + link="/openmetadata/connectors/database/postgres/airflow" size="half" /> @@ -75,7 +75,7 @@ Select PostgreSQL as the service type and click Next.

Select Service @@ -95,7 +95,7 @@ from.
Add New Service @@ -111,7 +111,7 @@ desired.
Configure service connection diff --git a/openmetadata-docs/content/openmetadata/connectors/pipeline/glue/index.md b/openmetadata-docs/content/openmetadata/connectors/pipeline/glue/index.md index 82af1c1fabf..58fec5c634c 100644 --- a/openmetadata-docs/content/openmetadata/connectors/pipeline/glue/index.md +++ b/openmetadata-docs/content/openmetadata/connectors/pipeline/glue/index.md @@ -93,7 +93,7 @@ from.
Add New Service @@ -109,7 +109,7 @@ desired.
Configure service connection diff --git a/openmetadata-docs/content/openmetadata/ingestion/workflows/data-quality/index.md b/openmetadata-docs/content/openmetadata/ingestion/workflows/data-quality/index.md index 8edfbf180fb..57d99f0d7eb 100644 --- a/openmetadata-docs/content/openmetadata/ingestion/workflows/data-quality/index.md +++ b/openmetadata-docs/content/openmetadata/ingestion/workflows/data-quality/index.md @@ -120,7 +120,7 @@ processor: value: [value] - ... ``` -The processor type should be set to ` "orm-test-runner"`. For accepted test definition names and parameter value names refer to the [tests page](/content/openmetadata/ingestion/workflows/data-quality/tests.md). +The processor type should be set to ` "orm-test-runner"`. For accepted test definition names and parameter value names refer to the [tests page](/openmetadata/ingestion/workflows/data-quality/tests). `sink` and `workflowConfig` will have the same settings than the ingestion and profiler workflow. diff --git a/openmetadata-docs/content/openmetadata/integrations/great-expectations.md b/openmetadata-docs/content/openmetadata/integrations/great-expectations.md index 3cf32f4ea81..4448632d3b1 100644 --- a/openmetadata-docs/content/openmetadata/integrations/great-expectations.md +++ b/openmetadata-docs/content/openmetadata/integrations/great-expectations.md @@ -11,7 +11,7 @@ Configure Great Expectations to integrate with OpenMetadata and ingest your test ### OpenMetadata Requirements You will to have OpenMetadata version 0.10 or later. -To deploy OpenMetadata, follow the procedure [Try OpenMetadata in Docker](quick-start/local-deployment) or follow the [Prefect Integration](/openmetadata/integrations/prefect) guide. +To deploy OpenMetadata, follow the procedure [Try OpenMetadata in Docker](/quick-start/local-deployment) or follow the [Prefect Integration](/openmetadata/integrations/prefect) guide. Before ingesting your tests results from Great Expectations you will need to have your table metadata ingested into OpenMetadata. Follow the instruction in the [Connectors](/openmetadata/connectors) section to learn more. @@ -108,6 +108,6 @@ alt="Run Great Expectations checkpoint" /> ### List of Great Expectations Supported Test -We currently only support a certain number of Great Expectations tests. The full list can be found in the [Tests](/openmetadata/data-quality/tests) section. +We currently only support a certain number of Great Expectations tests. The full list can be found in the [Tests](/openmetadata/ingestion/workflows/data-quality/tests) section. -If a test is not supported, there is no need to worry about the execution of your Great Expectations test. We will simply skip the tests that are not supported and continue the execution of your test suite. \ No newline at end of file +If a test is not supported, there is no need to worry about the execution of your Great Expectations test. We will simply skip the tests that are not supported and continue the execution of your test suite. diff --git a/openmetadata-docs/content/sdk/python/entities/ml-model.md b/openmetadata-docs/content/sdk/python/entities/ml-model.md index d614592495e..8da602b5918 100644 --- a/openmetadata-docs/content/sdk/python/entities/ml-model.md +++ b/openmetadata-docs/content/sdk/python/entities/ml-model.md @@ -34,7 +34,7 @@ There are some common questions with lifecycle tools, such as _Where (URL) is th This information, together with the rest of the features brought by OpenMetadata, lets us also manage topics such as schema changes in the sources or feature drifts, with the corresponding alerting systems. -While we can already extract certain pieces of information automatically via our Connectors (e.g., [Mlflow](/connectors/mlmodel/mlflow)), there are attributes that we'll need to fill in by ourselves. Thanks to the [Solution Design](/developers/architecture/design) of OpenMetadata and the [Python SDK](/sdk/python), this is going to be a rather easy task that will unlock the full power of your **organization's metadata**. +While we can already extract certain pieces of information automatically via our Connectors (e.g., [Mlflow](/connectors/ml-model/mlflow)), there are attributes that we'll need to fill in by ourselves. Thanks to the [Solution Design](/main-concepts/high-level-design) of OpenMetadata and the [Python SDK](/sdk/python), this is going to be a rather easy task that will unlock the full power of your **organization's metadata**. ## Properties Now that we have a clearer view of what we are trying to achieve, let's jump into a deeper view on the `MlModel` Entity [definition](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/data/mlmodel.json): @@ -142,4 +142,4 @@ In this doc, we have seen the role that OpenMetadata serves from a Machine Learn We have shown how to use the Python API to enrich the models' metadata and add the lineage information with its related Entities and how the versioning will respond to changes. -For further information on the properties, do not hesitate to review the [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/data/mlmodel.json) and for examples and usages with the Python API, you can take a look at our [integration tests](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/tests/integration/ometa/test_ometa_model_api.py). \ No newline at end of file +For further information on the properties, do not hesitate to review the [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/data/mlmodel.json) and for examples and usages with the Python API, you can take a look at our [integration tests](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/tests/integration/ometa/test_ometa_model_api.py).