mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2026-01-06 12:36:56 +00:00
MINOR: add Mongodb profiler docs (#15240)
* docs(mongodb): added profiler docs
This commit is contained in:
Imri Paran
GitHub
parent
1efbeb1204
commit
7f111c7ed8
@ -12,7 +12,7 @@ slug: /connectors/database/mongodb
|
||||
| Stage | BETA |
|
||||
| Metadata | {% icon iconName="check" /%} |
|
||||
| Query Usage | {% icon iconName="cross" /%} |
|
||||
| Data Profiler | {% icon iconName="cross" /%} |
|
||||
| Data Profiler | {% icon iconName="check" /%} |
|
||||
| Data Quality | {% icon iconName="cross" /%} |
|
||||
| Stored Procedures | {% icon iconName="cross" /%} |
|
||||
| Owners | {% icon iconName="cross" /%} |
|
||||
@ -35,6 +35,7 @@ Configure and schedule MongoDB metadata workflows from the OpenMetadata UI:
|
||||
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
- [Data Profiler](#data-profiler)
|
||||
|
||||
{% partial file="/v1.3/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/mongodb/yaml"} /%}
|
||||
|
||||
@ -79,3 +80,18 @@ To fetch the metadata from MongoDB to OpenMetadata, the MongoDB user must have a
|
||||
{% partial file="/v1.3/connectors/troubleshooting.md" /%}
|
||||
|
||||
{% partial file="/v1.3/connectors/database/related.md" /%}
|
||||
|
||||
## Data Profiler
|
||||
|
||||
{%inlineCallout icon="description" bold="OpenMetadata 1.3.1 or later" href="/deployment"%}
|
||||
To deploy OpenMetadata, check the Deployment guides.
|
||||
{%/inlineCallout%}
|
||||
|
||||
[Profiler deployment](/connectors/ingestion/workflows/profiler)
|
||||
|
||||
### Limitations
|
||||
|
||||
The MongodDB data profiler current supports only the following features:
|
||||
|
||||
1. **Row count**: The number of rows in the collection. Sampling or custom query is not supported.
|
||||
2. **Sample data:** If a custom query is defined it will be used for sample data.
|
||||
|
||||
@ -8,11 +8,11 @@ slug: /connectors/database/mongodb/yaml
|
||||
{% multiTablesWrapper %}
|
||||
|
||||
| Feature | Status |
|
||||
| :----------------- | :--------------------------- |
|
||||
| :----------------- |:-----------------------------|
|
||||
| Stage | BETA |
|
||||
| Metadata | {% icon iconName="check" /%} |
|
||||
| Query Usage | {% icon iconName="cross" /%} |
|
||||
| Data Profiler | {% icon iconName="cross" /%} |
|
||||
| Data Profiler | {% icon iconName="check" /%} |
|
||||
| Data Quality | {% icon iconName="cross" /%} |
|
||||
| Stored Procedures | {% icon iconName="cross" /%} |
|
||||
| Owners | {% icon iconName="cross" /%} |
|
||||
@ -34,6 +34,7 @@ Configure and schedule MongoDB metadata workflows from the OpenMetadata UI:
|
||||
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
- [Data Profiler](#data-profiler)
|
||||
|
||||
{% partial file="/v1.3/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/mongodb/yaml"} /%}
|
||||
|
||||
@ -154,6 +155,279 @@ source:
|
||||
|
||||
{% partial file="/v1.3/connectors/yaml/ingestion-cli.md" /%}
|
||||
|
||||
## Data Profiler
|
||||
|
||||
The Data Profiler workflow will be using the `orm-profiler` processor.
|
||||
|
||||
After running a Metadata Ingestion workflow, we can run Data Profiler workflow.
|
||||
While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server.
|
||||
|
||||
### Limitations
|
||||
|
||||
The MongodDB data profiler current supports only the following features:
|
||||
|
||||
1. **Row count**: The number of rows in the collection. Sampling or custom query is not supported.
|
||||
2. **Sample data:** If a custom query is defined it will be used for sample data.
|
||||
|
||||
### 1. Define the YAML Config
|
||||
|
||||
This is a sample config for the profiler:
|
||||
|
||||
{% codePreview %}
|
||||
|
||||
{% codeInfoContainer %}
|
||||
|
||||
{% codeInfo srNumber=13 %}
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json).
|
||||
|
||||
**generateSampleData**: Option to turn on/off generating sample data.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=16 %}
|
||||
|
||||
**processPiiSensitive**: Optional configuration to automatically tag columns that might contain sensitive information.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=18 %}
|
||||
|
||||
**timeoutSeconds**: Profiler Timeout in Seconds
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=20 %}
|
||||
|
||||
**schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=21 %}
|
||||
|
||||
**tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=22 %}
|
||||
|
||||
#### Processor Configuration
|
||||
|
||||
Choose the `orm-profiler`. Its config can also be updated to define tests from the YAML itself instead of the UI:
|
||||
|
||||
**tableConfig**: `tableConfig` allows you to set up some configuration at the table level.
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=23 %}
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
{% /codeInfo %}
|
||||
|
||||
|
||||
{% codeInfo srNumber=24 %}
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% /codeInfoContainer %}
|
||||
|
||||
{% codeBlock fileName="filename.yaml" %}
|
||||
|
||||
|
||||
```yaml
|
||||
source:
|
||||
type: monogodb
|
||||
serviceName: local_mongodb
|
||||
sourceConfig:
|
||||
config:
|
||||
type: Profiler
|
||||
```
|
||||
|
||||
```yaml {% srNumber=13 %}
|
||||
generateSampleData: true
|
||||
```
|
||||
```yaml {% srNumber=16 %}
|
||||
processPiiSensitive: false
|
||||
```
|
||||
```yaml {% srNumber=18 %}
|
||||
# timeoutSeconds: 43200
|
||||
```
|
||||
```yaml {% srNumber=20 %}
|
||||
# schemaFilterPattern:
|
||||
# includes:
|
||||
# - schema1
|
||||
# - schema2
|
||||
# excludes:
|
||||
# - schema3
|
||||
# - schema4
|
||||
```
|
||||
```yaml {% srNumber=21 %}
|
||||
# tableFilterPattern:
|
||||
# includes:
|
||||
# - table1
|
||||
# - table2
|
||||
# excludes:
|
||||
# - table3
|
||||
# - table4
|
||||
```
|
||||
|
||||
```yaml {% srNumber=22 %}
|
||||
processor:
|
||||
type: orm-profiler
|
||||
config: {} # Remove braces if adding properties
|
||||
# tableConfig:
|
||||
# - fullyQualifiedName: <table fqn>
|
||||
# profileQuery: <query to use for fetching the sample data>
|
||||
```
|
||||
|
||||
```yaml {% srNumber=23 %}
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
```
|
||||
|
||||
```yaml {% srNumber=24 %}
|
||||
workflowConfig:
|
||||
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
|
||||
openMetadataServerConfig:
|
||||
hostPort: <OpenMetadata host and port>
|
||||
authProvider: <OpenMetadata auth provider>
|
||||
```
|
||||
|
||||
{% /codeBlock %}
|
||||
|
||||
{% /codePreview %}
|
||||
|
||||
- You can learn more about how to configure and run the Profiler Workflow to extract Profiler data and execute the Data Quality from [here](/connectors/ingestion/workflows/profiler)
|
||||
|
||||
|
||||
|
||||
### 2. Prepare the Profiler DAG
|
||||
|
||||
Here, we follow a similar approach as with the metadata and usage pipelines, although we will use a different Workflow class:
|
||||
|
||||
{% codePreview %}
|
||||
|
||||
{% codeInfoContainer %}
|
||||
|
||||
{% codeInfo srNumber=25 %}
|
||||
|
||||
#### Import necessary modules
|
||||
|
||||
The `ProfilerWorkflow` class that is being imported is a part of a metadata orm_profiler framework, which defines a process of extracting Profiler data.
|
||||
|
||||
Here we are also importing all the basic requirements to parse YAMLs, handle dates and build our DAG.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=26 %}
|
||||
|
||||
**Default arguments for all tasks in the Airflow DAG.**
|
||||
- Default arguments dictionary contains default arguments for tasks in the DAG, including the owner's name, email address, number of retries, retry delay, and execution timeout.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
|
||||
{% codeInfo srNumber=27 %}
|
||||
|
||||
- **config**: Specifies config for the profiler as we prepare above.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=28 %}
|
||||
|
||||
- **metadata_ingestion_workflow()**: This code defines a function `metadata_ingestion_workflow()` that loads a YAML configuration, creates a `ProfilerWorkflow` object, executes the workflow, checks its status, prints the status to the console, and stops the workflow.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=29 %}
|
||||
|
||||
- **DAG**: creates a DAG using the Airflow framework, and tune the DAG configurations to whatever fits with your requirements
|
||||
- For more Airflow DAGs creation details visit [here](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#declaring-a-dag).
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% /codeInfoContainer %}
|
||||
|
||||
{% codeBlock fileName="filename.py" %}
|
||||
|
||||
```python {% srNumber=26 %}
|
||||
import yaml
|
||||
from datetime import timedelta
|
||||
from airflow import DAG
|
||||
from metadata.profiler.api.workflow import ProfilerWorkflow
|
||||
|
||||
try:
|
||||
from airflow.operators.python import PythonOperator
|
||||
except ModuleNotFoundError:
|
||||
from airflow.operators.python_operator import PythonOperator
|
||||
|
||||
from airflow.utils.dates import days_ago
|
||||
|
||||
|
||||
```
|
||||
```python {% srNumber=27 %}
|
||||
default_args = {
|
||||
"owner": "user_name",
|
||||
"email_on_failure": False,
|
||||
"retries": 3,
|
||||
"retry_delay": timedelta(seconds=10),
|
||||
"execution_timeout": timedelta(minutes=60),
|
||||
}
|
||||
|
||||
|
||||
```
|
||||
|
||||
```python {% srNumber=28 %}
|
||||
config = """
|
||||
<your YAML configuration>
|
||||
"""
|
||||
|
||||
|
||||
```
|
||||
|
||||
```python {% srNumber=29 %}
|
||||
def metadata_ingestion_workflow():
|
||||
workflow_config = yaml.safe_load(config)
|
||||
workflow = ProfilerWorkflow.create(workflow_config)
|
||||
workflow.execute()
|
||||
workflow.raise_from_status()
|
||||
workflow.print_status()
|
||||
workflow.stop()
|
||||
|
||||
|
||||
```
|
||||
|
||||
```python {% srNumber=30 %}
|
||||
with DAG(
|
||||
"profiler_example",
|
||||
default_args=default_args,
|
||||
description="An example DAG which runs a OpenMetadata ingestion workflow",
|
||||
start_date=days_ago(1),
|
||||
is_paused_upon_creation=False,
|
||||
catchup=False,
|
||||
) as dag:
|
||||
ingest_task = PythonOperator(
|
||||
task_id="profile_and_test_using_recipe",
|
||||
python_callable=metadata_ingestion_workflow,
|
||||
)
|
||||
|
||||
|
||||
```
|
||||
|
||||
{% /codeBlock %}
|
||||
|
||||
{% /codePreview %}
|
||||
|
||||
## dbt Integration
|
||||
|
||||
{% tilesContainer %}
|
||||
@ -165,3 +439,4 @@ description="Learn more about how to ingest dbt models' definitions and their li
|
||||
link="/connectors/ingestion/workflows/dbt" /%}
|
||||
|
||||
{% /tilesContainer %}
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user