datahub/metadata-ingestion/source_docs/presto_on_hive.md

Presto on Hive

For context on getting started with ingestion, check out our metadata ingestion guide.

Setup

To install this plugin, run pip install 'acryl-datahub[presto-on-hive]'.

Capabilities

This plugin extracts the following:

• Metadata for Presto views and Hive tables (external / managed)
• Column types associated with each table / view
• Detailed table / view property info

Quickstart recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: presto-on-hive
  config:
    # Hive metastore DB connection 
    host_port: localhost:5432
    database: metastore
   
    # specify the schema where metastore tables reside
    schema_pattern:
      allow:
        - "^public"

    # credentials
    username: user # optional
    password: pass # optional

    #scheme: 'postgresql+psycopg2' # set this if metastore db is using postgres
    #scheme: 'mysql+pymysql' # set this if metastore db is using mysql, default if unset

    # set this to have advanced filters on what to ingest
    #views_where_clause_suffix: AND d."name" in ('db1') 
    #tables_where_clause_suffix: AND d."name" in ('db1')
    
sink:
  # sink configs

Config details

Note that a . is used to denote nested fields in the YAML recipe.

Field	Required	Default	Description
username			Database username.
password			Database password.
host_port	✅		Host URL and port to connect to.
database			Database to ingest.
env		"PROD"	Environment to use in namespace when constructing URNs.
options.<option>			Any options specified here will be passed to SQLAlchemy's create_engine as kwargs. See https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine for details.
schema_pattern.allow			List of regex patterns for schemas to include in ingestion.
schema_pattern.ignoreCase		True	Whether to ignore case sensitivity during pattern matching.
include_views		True	Whether Presto views should be ingested.
include_tables		True	Whether Hive tables should be ingested.
views_where_clause_suffix		""	Where clause to specify what Presto views should be ingested.
tables_where_clause_suffix		""	Where clause to specify what Hive tables should be ingested.
schemas_where_clause_suffix		""	Where clause to specify what Hive schemas should be ingested.

Compatibility

Coming soon!

Questions

If you've got any questions on configuring this source, feel free to ping us on our Slack!