diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 91e25240fa7..81054e1a1af 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -12,6 +12,7 @@ * [BigQuery Usage](openmetadata/connectors/bigquery-usage.md) * [Data Model](connectors/data-model/README.md) * [DBT](connectors/data-model/dbt.md) + * [MariaDB](connectors/data-model/mariadb.md) * [ElasticSearch](openmetadata/connectors/elastic-search.md) * [Glue Catalog](connectors/glue-catalog.md) * [Hive](openmetadata/connectors/hive.md) diff --git a/docs/connectors/data-model/README.md b/docs/connectors/data-model/README.md index fbf0bf263c6..3a0cf9facfb 100644 --- a/docs/connectors/data-model/README.md +++ b/docs/connectors/data-model/README.md @@ -3,3 +3,8 @@ {% content-ref url="dbt.md" %} [dbt.md](dbt.md) {% endcontent-ref %} + +{% content-ref url="mariadb.md" %} +[mariadb.md](mariadb.md) +{% endcontent-ref %} + diff --git a/docs/connectors/data-model/dbt.md b/docs/connectors/data-model/dbt.md index 8f5d1472896..5c1d15fd482 100644 --- a/docs/connectors/data-model/dbt.md +++ b/docs/connectors/data-model/dbt.md @@ -58,7 +58,7 @@ metadata ingest -c ./examples/workflows/dbt.json Below is the configuration to publish DBT data into the OpenMetadata service. -Add `metadata-rest` sink along with `metadata-server` config +Add optionally `pii` processor and `metadata-rest` sink along with `metadata-server` config {% code title="dbt.json" %} ```javascript diff --git a/docs/connectors/data-model/mariadb.md b/docs/connectors/data-model/mariadb.md new file mode 100644 index 00000000000..8c6f992f906 --- /dev/null +++ b/docs/connectors/data-model/mariadb.md @@ -0,0 +1,94 @@ +--- +description: This guide will help install MariaDB connector and run manually +--- + +# MariaDB + +{% hint style="info" %} +**Prerequisites** + +OpenMetadata is built using Java, DropWizard, Jetty, and MySQL. + +1. Python 3.7 or above +{% endhint %} + +### Install from PyPI + +{% tabs %} +{% tab title="Install Using PyPI" %} +```bash +pip install 'openmetadata-ingestion[mysql]' +``` +{% endtab %} +{% endtabs %} + +### Run Manually + +```bash +metadata ingest -c ./examples/workflows/mariadb.json +``` + +### Configuration + +{% code title="mariadb.json" %} +```javascript +{ + "source": { + "type": "mariadb", + "config": { + "username": "openmetadata_user", + "password": "openmetadata_password", + "database": "openmetadata_db", + "service_name": "local_mysql", + "filter_pattern": { + "excludes": ["mysql.*", "information_schema.*", "performance_schema.*", "sys.*"] + } + } + }, + ... +``` +{% endcode %} + +1. **username** - pass the MariaDB username. +2. **password** - password for the username +3. **service\_name** - Service Name for this MariaDB cluster. If you added MariaDB cluster through OpenMetadata UI, make sure the service name matches the same. +4. **filter\_pattern** - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata +5. **data\_profiler\_enabled** - Enable data-profiling (Optional). It will provide you the newly ingested data. +6. **data\_profiler\_offset** - Specify offset. +7. **data\_profiler\_limit** - Specify limit. + +## Publish to OpenMetadata + +Below is the configuration to publish MariaDB data into the OpenMetadata service. + +Add optionally `pii` processor and `metadata-rest` sink along with `metadata-server` config + +{% code title="mariadb.json" %} +```javascript +{ + "source": { + "type": "mariadb", + "config": { + "username": "openmetadata_user", + "password": "openmetadata_password", + "database": "openmetadata_db", + "service_name": "local_mysql", + "filter_pattern": { + "excludes": ["mysql.*", "information_schema.*", "performance_schema.*", "sys.*"] + } + } + }, + "sink": { + "type": "metadata-rest", + "config": {} + }, + "metadata_server": { + "type": "metadata-server", + "config": { + "api_endpoint": "http://localhost:8585/api", + "auth_provider_type": "no-auth" + } + } +} +``` +{% endcode %} diff --git a/docs/connectors/glue-catalog.md b/docs/connectors/glue-catalog.md index b02b260dabf..3997f3c1df8 100644 --- a/docs/connectors/glue-catalog.md +++ b/docs/connectors/glue-catalog.md @@ -60,7 +60,7 @@ metadata ingest -c ./examples/workflows/glue.json Below is the configuration to publish Glue data into the OpenMetadata service. -Add `metadata-rest` sink along with `metadata-server` config +Add optionally `pii` processor and `metadata-rest` sink along with `metadata-server` config {% code title="glue.json" %} ```javascript diff --git a/docs/install/metadata-ingestion/connectors/README.md b/docs/install/metadata-ingestion/connectors/README.md index 3b973040bbc..94a7ca715a1 100644 --- a/docs/install/metadata-ingestion/connectors/README.md +++ b/docs/install/metadata-ingestion/connectors/README.md @@ -9,9 +9,9 @@ OpenMetadata supports connectors to some popular services. We will continue as a * [BigQuery Usage](../../../openmetadata/connectors/bigquery-usage.md) * [Data Model](../../../connectors/data-model/) * [DBT](../../../connectors/data-model/dbt.md) + * [MariaDB](../../../connectors/data-model/mariadb.md) * [ElasticSearch](../../../openmetadata/connectors/elastic-search.md) -* [Glue Catalog](../../../connectors/glue-catalog.md) -* [MariaDB](../../../connectors/mariadb.md) +* Glue Catalog * [MSSQL](../../../openmetadata/connectors/mssql.md) * [MySQL](../../../openmetadata/connectors/mysql.md) * [Hive](../../../openmetadata/connectors/hive.md) diff --git a/docs/install/run-openmetadata.md b/docs/install/run-openmetadata.md index d5d2eea34a2..8bd7f8985c5 100644 --- a/docs/install/run-openmetadata.md +++ b/docs/install/run-openmetadata.md @@ -51,7 +51,8 @@ Preferences -> Resources -> Advanced Create a new directory for OpenMetadata and navigate into that directory. ``` -mkdir openmetadata-docker; cd openmetadata-docker +mkdir openmetadata-docker +cd openmetadata-docker ``` ### 2. Create a Python virtual environment