OpenMetadata/docs/connectors/glue-catalog.md

description
This guide will help install Glue connector and run manually

Glue Catalog

{% hint style="info" %} Prerequisites

OpenMetadata is built using Java, DropWizard, Jetty, and MySQL.

1. Python 3.7 or above {% endhint %}

Install from PyPI

{% tabs %} {% tab title="Install Using PyPI" %}

pip install 'openmetadata-ingestion[glue]'

{% endtab %} {% endtabs %}

Run Manually

metadata ingest -c ./examples/workflows/glue.json

Configuration

{% code title="glue.json" %}

{
  "source": {
    "type": "glue",
    "config": {
      "aws_access_key_id": "aws_access_key_id",
      "aws_secret_access_key": "aws_secret_access_key",
      "db_service_name": "local_glue_db",
      "pipeline_service_name": "local_glue_pipeline",
      "region_name": "region_name",
      "endpoint_url": "endpoint_url",
      "service_name": "local_glue"
    }
  },
...

{% endcode %}

1. aws_access_key_id - Access Key for AWS.
2. aws_secret_access_key - Secret Key for AWS.
3. db_service_name - Service Name for this Glue Database cluster.
4. pipeline_service_name - Service Name for this Glue Pipeline cluster.
5. region_name - AWS account region.
6. endpoint_url - Service Endpoints from AWS.

Publish to OpenMetadata

Below is the configuration to publish Glue data into the OpenMetadata service.

Add optionally pii processor and metadata-rest sink along with metadata-server config

{% code title="glue.json" %}

{
  "source": {
    "type": "glue",
    "config": {
      "aws_access_key_id": "aws_access_key_id",
      "aws_secret_access_key": "aws_secret_access_key",
      "db_service_name": "local_glue_db",
      "pipeline_service_name": "local_glue_pipeline",
      "region_name": "region_name",
      "endpoint_url": "endpoint_url",
      "service_name": "local_glue"
    }
  },
  "sink": {
    "type": "metadata-rest",
    "config": {}
  },
  "metadata_server": {
    "type": "metadata-server",
    "config": {
      "api_endpoint": "http://localhost:8585/api",
      "auth_provider_type": "no-auth"
    }
  }
}

{% endcode %}