Update: Trino, Presto Logic and Doc (#9859)

* Update: Trino, Presto Logic and Doc

* Update: Trino, Presto Logic and Doc

* Update permissions in Doc

* Update openmetadata-docs/content/connectors/database/trino/index.md

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

ingestion/src/metadata/examples/workflows/presto.yaml

@@ -5,10 +5,10 @@ source:
     config:
       type: Presto
       hostPort: localhost:8080
-      catalog: tpcds
+      catalog: catalog_name
       username: admin
       password: password
-      databaseSchema: tpcds
+      databaseSchema: schema_name
   sourceConfig:
     config:
       generateSampleData: false

ingestion/src/metadata/examples/workflows/trino.yaml

@@ -6,8 +6,8 @@ source:
       type: Trino
       hostPort: localhost:8080
       username: user
-      catalog: tpcds
-      databaseSchema: tiny
+      catalog: catalog_name
+      databaseSchema: schema_name
       connectionOptions: {}
       connectionArguments: {}
   sourceConfig:

ingestion/src/metadata/ingestion/source/database/presto/metadata.py

@@ -132,6 +132,7 @@ class PrestoSource(CommonDbSourceService):
         else:
             results = self.connection.execute("SHOW CATALOGS")
             for res in results:
+                if res:
                     new_catalog = res[0]
                     database_fqn = fqn.build(
                         self.metadata,

ingestion/src/metadata/ingestion/source/database/trino/metadata.py

@@ -181,6 +181,7 @@ class TrinoSource(CommonDbSourceService):
         else:
             results = self.connection.execute("SHOW CATALOGS")
             for res in results:
+                if res:
                     new_catalog = res[0]
                     database_fqn = fqn.build(
                         self.metadata,

openmetadata-docs/content/connectors/database/presto/airflow.md

@@ -142,6 +142,7 @@ workflowConfig:
 - **password**: Password to connect to Presto.
 - **hostPort**: Enter the fully qualified hostname and port number for your Presto deployment in the Host and Port field.
 - **catalog**: Presto offers a catalog feature where all the databases are stored. (Providing the Catalog is not mandatory from 0.12.2 or greater versions)
+- **DatabaseSchema**: DatabaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
 - **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Presto during the connection. These details must be added as Key-Value pairs.
 - **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Presto during the connection. These details must be added as Key-Value pairs.
   - In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"`

openmetadata-docs/content/connectors/database/presto/cli.md

@@ -142,6 +142,7 @@ workflowConfig:
 - **password**: Password to connect to Presto.
 - **hostPort**: Enter the fully qualified hostname and port number for your Presto deployment in the Host and Port field.
 - **catalog**: Presto offers a catalog feature where all the databases are stored. (Providing the Catalog is not mandatory from 0.12.2 or greater versions)
+- **DatabaseSchema**: DatabaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
 - **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Presto during the connection. These details must be added as Key-Value pairs.
 - **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Presto during the connection. These details must be added as Key-Value pairs. 
   - In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"`

openmetadata-docs/content/connectors/database/presto/index.md

@@ -136,6 +136,7 @@ the changes.
 - **Password**: Password to connect to Presto.
 - **Host and Port**: Enter the fully qualified hostname and port number for your Presto deployment in the Host and Port field.
 - **Catalog**: Presto offers a catalog feature where all the databases are stored. (Providing the Catalog is not mandatory from 0.12.2 or greater versions)
+- **DatabaseSchema**: DatabaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
 - **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Presto during the connection. These details must be added as Key-Value pairs.
 - **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Presto during the connection. These details must be added as Key-Value pairs. 
   - In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"`

openmetadata-docs/content/connectors/database/trino/airflow.md

@@ -30,6 +30,17 @@ To run the Trino ingestion, you will need to install:
 pip3 install "openmetadata-ingestion[trino]"
 ```
 
+<Note>
+
+To Inesget metadata from the Trino User Must have select privileges to this tables.
+- `information_schema.schemata`
+- `information_schema.columns`
+- `information_schema.tables`
+- `information_schema.views`
+- `system.metadata.table_comments`
+
+</Note>
+
 ## Metadata Ingestion
 
 All connectors are defined as JSON Schemas.
@@ -145,6 +156,7 @@ workflowConfig:
 - **password**: Password to connect to Trino.
 - **hostPort**: Enter the fully qualified hostname and port number for your Trino deployment in the Host and Port field.
 - **catalog**: Trino offers a catalog feature where all the databases are stored. (Providing the Catalog is not mandatory from 0.12.2 or greater versions)
+- **DatabaseSchema**: DatabaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
 - **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Trino during the connection. These details must be added as Key-Value pairs.
 - **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Trino during the connection. These details must be added as Key-Value pairs.
   - In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"`

openmetadata-docs/content/connectors/database/trino/cli.md

@@ -30,6 +30,17 @@ To run the Trino ingestion, you will need to install:
 pip3 install "openmetadata-ingestion[trino]"
 ```
 
+<Note>
+
+To Inesget metadata from the Trino User Must have select privileges to this tables.
+- `information_schema.schemata`
+- `information_schema.columns`
+- `information_schema.tables`
+- `information_schema.views`
+- `system.metadata.table_comments`
+
+</Note>
+
 ## Metadata Ingestion
 
 All connectors are defined as JSON Schemas.
@@ -145,6 +156,7 @@ workflowConfig:
 - **password**: Password to connect to Trino.
 - **hostPort**: Enter the fully qualified hostname and port number for your Trino deployment in the Host and Port field.
 - **catalog**: Trino offers a catalog feature where all the databases are stored. (Providing the Catalog is not mandatory from 0.12.2 or greater versions)
+- **DatabaseSchema**: DatabaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
 - **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Trino during the connection. These details must be added as Key-Value pairs.
 - **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Trino during the connection. These details must be added as Key-Value pairs. 
   - In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"`

openmetadata-docs/content/connectors/database/trino/index.md

@@ -43,6 +43,17 @@ To deploy OpenMetadata, check the <a href="/deployment">Deployment</a> guides.
 To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
 custom Airflow plugins to handle the workflow deployment.
 
+<Note>
+
+To ingest metadata from the Trino source, the user must have select privileges for the following tables.
+- `information_schema.schemata`
+- `information_schema.columns`
+- `information_schema.tables`
+- `information_schema.views`
+- `system.metadata.table_comments`
+
+</Note>
+
 ## Metadata Ingestion
 
 ### 1. Visit the Services Page
@@ -136,6 +147,7 @@ the changes.
 - **Password**: Password to connect to Trino.
 - **Host and Port**: Enter the fully qualified hostname and port number for your Trino deployment in the Host and Port field.
 - **Catalog**: Trino offers a catalog feature where all the databases are stored. (Providing the Catalog is not mandatory from 0.12.2 or greater versions)
+- **DatabaseSchema**: DatabaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
 - **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Trino during the connection. These details must be added as Key-Value pairs.
 - **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Trino during the connection. These details must be added as Key-Value pairs. 
   - In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"`