MINOR: Data Insight Custom Dashboard Docs (#17389)

* MINOR: Data Insight Custom Dashboard Docs

* custom-dashboard.md

openmetadata-docs/content/v1.5.x-SNAPSHOT/how-to-guides/data-insights/airflow-sdk.md

@@ -1,117 +0,0 @@
----
-title: Run Data Insights using Airflow SDK
-slug: /how-to-guides/data-insights/airflow-sdk
----
-
-# Run Data Insights using Airflow SDK
-
-## 1. Define the YAML Config
-
-This is a sample config for Data Insights:
-
-```yaml
-source:
-  type: dataInsight
-  serviceName: OpenMetadata
-  sourceConfig:
-    config:
-      type: MetadataToElasticSearch
-processor:
-  type: data-insight-processor
-  config: {}
-sink:
-  type: elasticsearch
-  config:
-    es_host: localhost
-    es_port: 9200
-    recreate_indexes: false
-workflowConfig:
-  loggerLevel: DEBUG
-  openMetadataServerConfig:
-    hostPort: '<OpenMetadata host and port>'
-    authProvider: openmetadata
-    securityConfig:
-      jwtToken: '{bot_jwt_token}'
-```
-
-### Source Configuration - Source Config
-
-- To send the metadata to OpenMetadata, it needs to be specified as `type: MetadataToElasticSearch`.
-
-### Processor Configuration
-
-- To send the metadata to OpenMetadata, it needs to be specified as `type: data-insight-processor`.
-
-### Workflow Configuration
-
-The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
-
-For a simple, local installation using our docker containers, this looks like:
-
-```yaml
-workflowConfig:
-  openMetadataServerConfig:
-    hostPort: 'http://localhost:8585/api'
-    authProvider: openmetadata
-    securityConfig:
-      jwtToken: '{bot_jwt_token}'
-```
-
-We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client).
-You can find the different implementation of the ingestion below.
-
-## 2. Prepare the Data Insights DAG
-
-Create a Python file in your Airflow DAGs directory with the following contents:
-
-```python
-import pathlib
-import yaml
-from datetime import timedelta
-from airflow import DAG
-from metadata.workflow.data_insight import DataInsightWorkflow
- 
-
-try:
-    from airflow.operators.python import PythonOperator
-except ModuleNotFoundError:
-    from airflow.operators.python_operator import PythonOperator
-
-from metadata.config.common import load_config_file
-from airflow.utils.dates import days_ago
-
-default_args = {
-    "owner": "user_name",
-    "email": ["username@org.com"],
-    "email_on_failure": False,
-    "retries": 3,
-    "retry_delay": timedelta(minutes=5),
-    "execution_timeout": timedelta(minutes=60)
-}
-
-config = """
-<your YAML configuration>
-"""
-
-def metadata_ingestion_workflow():
-    workflow_config = yaml.safe_load(config)
-    workflow = DataInsightWorkflow.create(workflow_config)
-    workflow.execute()
-    workflow.raise_from_status()
-    workflow.print_status()
-    workflow.stop()
-
-with DAG(
-    "sample_data",
-    default_args=default_args,
-    description="An example DAG which runs a OpenMetadata ingestion workflow",
-    start_date=days_ago(1),
-    is_paused_upon_creation=False,
-    schedule_interval='*/5 * * * *',
-    catchup=False,
-) as dag:
-    ingest_task = PythonOperator(
-        task_id="ingest_using_recipe",
-        python_callable=metadata_ingestion_workflow,
-    )
-```

openmetadata-docs/content/v1.5.x-SNAPSHOT/how-to-guides/data-insights/custom-dashboard.md

@@ -0,0 +1,156 @@
+---
+title: Custom Data Insight Dashboards
+slug: /how-to-guides/data-insights/custom-dashboard
+---
+
+# Custom Data Insight Dashboards (Collate Only)
+
+The default Data Insight charts in OpenMetadata focus on description, ownership, and tiering. However, documentation needs often vary across organizations. To address this, OpenMetadata now supports the creation of custom Data Insight dashboards and charts. This documentation will guide you through the process of creating these custom dashboards and charts within your OpenMetadata instance, empowering you to tailor insights to your organization's unique documentation goals for enhanced data governance.
+
+
+## Step 1: Create a Dashboard
+
+Navigate to the `Insights` page from the side menu of application and click on `Create Dashboard` button from top left, provide the name, description and owner of the dashboard and finally click on `Create` button to save the dashboard.
+
+{% image
+    src="/images/v1.5/features/data-insight/custom-dashboard/navigate-to-insights.png"
+    alt="Navigate to Insights"
+    caption="Navigate to Insights"
+ /%}
+
+
+ {% image
+    src="/images/v1.5/features/data-insight/custom-dashboard/add-dashboard.png"
+    alt="Click on Add Dashboard"
+    caption="Click on Add Dashboard"
+ /%}
+
+{% image
+    src="/images/v1.5/features/data-insight/custom-dashboard/create-custom-dashboard-form.png"
+    alt="Fill Create Custom Dashboard Form"
+    caption="Fill Create Custom Dashboard Form"
+ /%}
+
+
+## Step 2: Create a Chart
+
+Once you create a dashboard, you can click on `Create Chart` button to add a new chart to this dashboard
+
+{% image
+    src="/images/v1.5/features/data-insight/custom-dashboard/create-chart.png"
+    alt="Create Chart"
+    caption="Create Chart"
+ /%}
+
+Now you will see a create chart form with a preview of the chart based on the configuration provided in the form. This preview will be a plot of data from previous 7 days. By default it shows the number/count of records of data available to compute. This number or record represent the total number of `Table`, `Stored Procedure`, `Database Schema`, `Database`, `Chart`, `Dashboard`, `Dashboard Data Model`, `Pipeline`, `Topic`, `Container`, `Search Index`, `ML Model`, `Data Product`, `Glossary Term` & `Tag` available within your OpenMetadata Instance.
+
+{% image
+    src="/images/v1.5/features/data-insight/custom-dashboard/create-chart-form-1.png"
+    alt="Create Chart Form Part 1"
+    caption="Create Chart Form Part 1"
+ /%}
+
+ {% image
+    src="/images/v1.5/features/data-insight/custom-dashboard/create-chart-form-2.png"
+    alt="Create Chart Form Part 2"
+    caption="Create Chart Form Part 2"
+ /%}
+
+You can customize the default settings to create the desired chart, to do that lets understand what are the available fields and how to customize it.
+
+
+- **Name**: Provide a relevant name to the chart that we are about to create.
+- **Chart Type**: You can create a `Line`, `Area`, `Bar` chart and a `Summary Card` which is a special chart/card to represent a single value.
+- **Description**: Provide a details description to this chart so that it is easier to understand the output of the chart.
+- **GroupBy**: Group by field is used to breakdown the results by a specific field, for example choose 
+`entityType` to breakdown the results by entity type.
+- **Y Axis Label**: Provide a custom label to y-axis.
+- **Advanced Filters**: Filter down results by applying advanced filters, the experience of advanced filters is similar to the experience of explore page.
+- **Method**: We support computation of chart based on 2 methods `Function` & `Formula`.
+
+
+### Chart By Function:
+
+- **Function**: In this method you can choose the function from the dropdown menu of functions, the supported functions are `Count`, `Sum`, `Avg`, `Max` & `Min`.
+**Note**: The functions `Sum`, `Avg`, `Max` & `Min` are only supported with field of numerical type, if you attempt to apply this function on any non numerical field then you will not see any output.
+
+- **Field**: Select the filed from the given dropdown of list on which you want to apply the function, for example if you select function as `Count` and field as `id` which is also the default selected option, then it will give you the count of records as described above. If you select function as `Avg` and field as `size` then it will give you avg size of all the containers ingested in OpenMetadata, because only container entities has `size` as a field.
+
+### Chart By Formula:
+
+- **Formula**: In case you want to write a custom logic to plot chart then chart by formula is the way to go. With formula you can write a complex mathematical expression including functions, fields and filters and the final result of this expression will get plotted on the chart.
+
+
+{% image
+    src="/images/v1.5/features/data-insight/custom-dashboard/formula-breakdown.png"
+    alt="Formula Breakdown"
+    caption="Formula Breakdown"
+ /%}
+
+A formula contains 3 element.
+
+**Function**: The supported functions are `count`, `sum`, `avg`, `min` & `max`, and as explained above The functions `Sum`, `Avg`, `Max` & `Min` are only supported with field of numerical type, if you attempt to apply this function on any non numerical field then you will not see any output.
+
+**Key/Field**: To define a key within a formula, use the format `k='<field-name>'`. The `k=` part indicates that you're specifying a key field, and the field name goes inside single quotes.
+
+You can choose the field name from the fields dropdown from the chart by function method. For example, if you want to use the `Owner Name` field, you would write `k='owner.name'` in the formula. If you want to use `Owner DisplayName` you would write `k='owner.displayName'`.
+
+Remember, spaces in field names are replaced with a period (.), and the key should be in lower camel case (meaning the first word is lowercase, and each subsequent word starts with an uppercase letter).
+
+**Query**: The query is to filter down the result to plot, use the format `q='<filed-name>: <value>'`. The `q=` part signifies a query, adn the condition goes inside the single quotes.
+
+The value can be a literal value or it can be `*` to signify that any value exists for the give field.
+
+**Examples**:
+
+```
+// count of records
+count()
+
+// count of records where owner exists 
+count(q='owner: *')
+
+// count of records where owner exists and tier Tier1 is applied
+count(q='owner: * AND tier: "Tier.Tier1"')
+
+// count of records where owner is admin and tier Tier1 is applied
+count(q='owner: "admin" AND tier: "Tier.Tier1"')
+
+// count of records where owner is admin and tier Tier1 is applied
+count(q='owner: "admin" AND tier: "Tier.Tier1"')
+
+// count of tables where owner is admin and tier Tier1 is applied
+count(q='owner: "admin" AND tier: "Tier.Tier1" AND entityType: "table"')
+
+// count of tables where owner is admin and any tier except "NoTier" (which is used to signify that no tier is applied)
+count(q='owner: "admin" AND NOT(tier: "NoTier") AND entityType: "table"')
+
+// Percentage of tables having complete description, hasDescription is special field which is set to 1 when description is complete
+(count(q='entityType: "table" and hasDescription: 1')/count())*100
+
+// Avg size of containers
+avg(k='size',q='entityType: "container"')
+
+// Max size of containers
+max(k='size',q='entityType: "container"')
+```
+
+
+## Step 3: Save the Chart and Layout
+
+Once you have configured your chart correctly, click on the save button to save the chart and add it to dashboard.
+
+{% image
+    src="/images/v1.5/features/data-insight/custom-dashboard/create-chart-form-preview.png"
+    alt="Create Chart Final Preview"
+    caption="Create Chart Final Preview"
+ /%}
+
+
+You can add multiple charts to a dashboard and rearrange the layout within a dashboard.
+
+{% image
+    src="/images/v1.5/features/data-insight/custom-dashboard/save-and-rearrange-chart.png"
+    alt="Save and Rearrange Chart"
+    caption="Save and Rearrange Chart"
+ /%}

openmetadata-docs/content/v1.5.x-SNAPSHOT/how-to-guides/data-insights/elasticsearch-reindex.md

@@ -1,90 +0,0 @@
----
-title: Run Elasticsearch Reindex using Airflow SDK
-slug: /how-to-guides/data-insights/elasticsearch-reindex
----
-
-# Run Elasticsearch Reindex using Airflow SDK
-
-## 1. Define the YAML Config
-
-This is a sample config for Elasticsearch Reindex:
-
-```yaml
-source:
-source:
-  type: metadata_elasticsearch
-  serviceName: openMetadata
-  serviceConnection:
-    config:
-      type: MetadataES
-  sourceConfig:
-    config: {}
-sink:
-  type: elasticsearch
-  config:
-    es_host: localhost
-    es_port: 9200
-    recreate_indexes: true
-workflowConfig:
-  openMetadataServerConfig:
-    hostPort: http://localhost:8585/api
-    authProvider: openmetadata
-    securityConfig:
-      jwtToken: "eyJraWQiOiJHYjM4OWEtOWY3Ni1nZGpzLWE5MmotMDI0MmJrOTQzNTYiLCJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsImlzQm90IjpmYWxzZSwiaXNzIjoib3Blbi1tZXRhZGF0YS5vcmciLCJpYXQiOjE2NjM5Mzg0NjIsImVtYWlsIjoiYWRtaW5Ab3Blbm1ldGFkYXRhLm9yZyJ9.tS8um_5DKu7HgzGBzS1VTA5uUjKWOCU0B_j08WXBiEC0mr0zNREkqVfwFDD-d24HlNEbrqioLsBuFRiwIWKc1m_ZlVQbG7P36RUxhuv2vbSp80FKyNM-Tj93FDzq91jsyNmsQhyNv_fNr3TXfzzSPjHt8Go0FMMP66weoKMgW2PbXlhVKwEuXUHyakLLzewm9UMeQaEiRzhiTMU3UkLXcKbYEJJvfNFcLwSl9W8JCO_l0Yj3ud-qt_nQYEZwqW6u5nfdQllN133iikV4fM5QZsMCnm8Rq1mvLR0y9bmJiD7fwM1tmJ791TUWqmKaTnP49U493VanKpUAfzIiOiIbhg"
-```
-
-### 2. Prepare the Ingestion DAG
-
-Create a Python file in your Airflow DAGs directory with the following contents:
-
-```python
-import pathlib
-import yaml
-from datetime import timedelta
-from airflow import DAG
-
-try:
-    from airflow.operators.python import PythonOperator
-except ModuleNotFoundError:
-    from airflow.operators.python_operator import PythonOperator
-
-from metadata.config.common import load_config_file
-from metadata.workflow.metadata import MetadataWorkflow
- 
-from airflow.utils.dates import days_ago
-
-default_args = {
-    "owner": "user_name",
-    "email": ["username@org.com"],
-    "email_on_failure": False,
-    "retries": 3,
-    "retry_delay": timedelta(minutes=5),
-    "execution_timeout": timedelta(minutes=60)
-}
-
-config = """
-<your YAML configuration>
-"""
-
-def metadata_ingestion_workflow():
-    workflow_config = yaml.safe_load(config)
-    workflow = MetadataWorkflow.create(workflow_config)
-    workflow.execute()
-    workflow.raise_from_status()
-    workflow.print_status()
-    workflow.stop()
-
-with DAG(
-    "sample_data",
-    default_args=default_args,
-    description="An example DAG which runs a OpenMetadata ingestion workflow",
-    start_date=days_ago(1),
-    is_paused_upon_creation=False,
-    schedule_interval='*/5 * * * *',
-    catchup=False,
-) as dag:
-    ingest_task = PythonOperator(
-        task_id="ingest_using_recipe",
-        python_callable=metadata_ingestion_workflow,
-    )
-```

openmetadata-docs/content/v1.5.x-SNAPSHOT/how-to-guides/data-insights/metadata-cli.md

@@ -1,69 +0,0 @@
----
-title: Run Data Insights using Metadata CLI
-slug: /how-to-guides/data-insights/metadata-cli
----
-
-# Run Data Insights using Metadata CLI
-
-## 1. Define the YAML Config
-
-This is a sample config for Data Insights:
-
-```yaml
-source:
-  type: dataInsight
-  serviceName: OpenMetadata
-  sourceConfig:
-    config:
-      type: MetadataToElasticSearch
-processor:
-  type: data-insight-processor
-  config: {}
-sink:
-  type: elasticsearch
-  config:
-    es_host: localhost
-    es_port: 9200
-    recreate_indexes: false
-workflowConfig:
-  loggerLevel: DEBUG
-  openMetadataServerConfig:
-    hostPort: '<OpenMetadata host and port>'
-    authProvider: openmetadata
-    securityConfig:
-      jwtToken: '{bot_jwt_token}'
-```
-
-### Source Configuration - Source Config
-
-- To send the metadata to OpenMetadata, it needs to be specified as `type: MetadataToElasticSearch`.
-
-### Processor Configuration
-
-- To send the metadata to OpenMetadata, it needs to be specified as `type: data-insight-processor`.
-
-### Workflow Configuration
-
-The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
-
-For a simple, local installation using our docker containers, this looks like:
-
-```yaml
-workflowConfig:
-  openMetadataServerConfig:
-    hostPort: 'http://localhost:8585/api'
-    authProvider: openmetadata
-    securityConfig:
-      jwtToken: '{bot_jwt_token}'
-```
-
-We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client).
-You can find the different implementation of the ingestion below.
-
-## 2. Run with the CLI
-
-First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run:
-
-```bash
-metadata insight -c <path-to-yaml>
-```

openmetadata-docs/content/v1.5.x-SNAPSHOT/menu.md

@@ -813,6 +813,8 @@ site_menu:
     url: /how-to-guides/data-insights/email-report
   - category: How-to Guides / Data Insights / How to Transform the Data Culture of Your Company
     url: /how-to-guides/data-insights/data-culture
+  - category: How-to Guides / Data Insights / Custom Data Insight Dashboards
+    url: /how-to-guides/data-insights/custom-dashboard
 
   - category: How-to Guides / Data Governance
     url: /how-to-guides/data-governance