Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

118 lines
3.9 KiB
Markdown
Raw Normal View History

---
title: Connectors
slug: /connectors
---
# Connectors
OpenMetadata can extract metadata from the following list of connectors below.
## Ingestion Deployment
To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
custom Airflow plugins to handle the workflow deployment. If you want to install it manually in an already existing
Airflow host, you can follow [this](/deployment/ingestion/openmetadata) guide.
If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check
the following docs to run the Ingestion Framework in any orchestrator externally.
{% tilesContainer %}
{% tile
title="Run Connectors from the OpenMetadata UI"
description="Learn how to manage your deployment to run connectors from the UI"
link="/deployment/ingestion/openmetadata"
/ %}
{% tile
title="External Schedulers"
description="Get more information about running the Ingestion Framework Externally"
link="/deployment/ingestion"
/ %}
{% /tilesContainer %}
## Database / DataWarehouse Services
- [Athena](/connectors/database/athena)
- [AzureSQL](/connectors/database/azuresql)
- [BigQuery](/connectors/database/bigquery)
- [Clickhouse](/connectors/database/clickhouse)
- [Couchbase](/connectors/database/couchbase)
- [Data lake](/connectors/database/datalake)
- [Databricks SQL](/connectors/database/databricks)
- [DB2](/connectors/database/db2)
- [Delta Lake](/connectors/database/deltalake)
- [Domo Database](/connectors/database/domo-database)
- [Druid](/connectors/database/druid)
- [DynamoDB](/connectors/database/dynamodb)
- [Glue](/connectors/database/glue)
- [Greenplum](/connectors/database/greenplum)
- [Hive](/connectors/database/hive)
Fixes #5448: Implement initial Iceberg Connector using PyIceberg (#14825) * Create the iceberg connection schema * Link the IcebergConnection configuration with the forms on the UI * Add the pyiceberg dependency on the ingestion package * Create the get_connection and test_connection functions * First iteration on the iceberg ingestion logic * Add A more comprehensive implementation of the Iceberg Source * Add UnitTests * Update icebergConnection definition * Update the iceberg souce code based on new schema * Updated icebergConnecgtion schema for simplicity and to be able to configure Converters * Updated setup dependencies to be more flexible * Updated get_owner_ref logic * Fix formatting * Changed the icebergConnection json schema structure to enable the ClassConverters * Add the IcebergCatalog and IcebergFileSystem ClassConverters * Refactor the code to take into account the new jsonSchema structure * Fix formatting * Add Documentation for the Iceberg Connector * Fix Menu order for Iceberg * ui: add Iceberg service icon and constant * Fix DynamoDb Catalog issue due to how PyIceberg instantes it * Changed uri title to URI * Fix ClassConverter for Iceberg * Fix GetSecretValue for password types * Fix formatting * Fix formatting * Add Iceberg Connector Images for the docs * Add pylint disable for Hacky super() call * Add Iceberg.md for the UI docs * Fix pylint complaint * Fix pylint complaint * Fix UnitTests * fix type error and unit tests * update pipeline type checks * Fix Sonar Cloud complaints --------- Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2024-01-29 06:32:58 +01:00
- [Iceberg](/connectors/database/iceberg)
- [Impala](/connectors/database/impala)
- [MariaDB](/connectors/database/mariadb)
- [MongoDB](/connectors/database/mongodb)
- [MSSQL](/connectors/database/mssql)
- [MySQL](/connectors/database/mysql)
- [Oracle](/connectors/database/oracle)
- [PinotDB](/connectors/database/pinotdb)
- [Postgres](/connectors/database/postgres)
- [Presto](/connectors/database/presto)
- [Redshift](/connectors/database/redshift)
- [Salesforce](/connectors/database/salesforce)
- [SAP Hana](/connectors/database/sap-hana)
- [SAS](/connectors/database/sas)
- [SingleStore](/connectors/database/singlestore)
- [Snowflake](/connectors/database/snowflake)
- [SQLite](/connectors/database/sqlite)
- [Trino](/connectors/database/trino)
- [Unity Catalog](/connectors/database/unity-catalog)
- [Vertica](/connectors/database/vertica)
## Dashboard Services
- [Domo Dashboard](/connectors/dashboard/domo-dashboard)
- [Looker](/connectors/dashboard/looker)
- [Metabase](/connectors/dashboard/metabase)
- [Mode](/connectors/dashboard/mode)
- [PowerBI](/connectors/dashboard/powerbi)
- [Qlik Sense](/connectors/dashboard/qliksense)
- [QuickSight](/connectors/dashboard/quicksight)
- [Redash](/connectors/dashboard/redash)
- [Superset](/connectors/dashboard/superset)
- [Tableau](/connectors/dashboard/tableau)
## Messaging Services
- [Kafka](/connectors/messaging/kafka)
- [Kinesis](/connectors/messaging/kinesis)
- [Redpanda](/connectors/messaging/redpanda)
## Pipeline Services
- [Airbyte](/connectors/pipeline/airbyte)
- [Airflow](/connectors/pipeline/airflow)
- [Dagster](/connectors/pipeline/dagster)
- [Databricks Pipeline](/connectors/pipeline/databricks-pipeline)
- [Domo Pipeline](/connectors/pipeline/domo-pipeline)
- [Fivetran](/connectors/pipeline/fivetran)
- [Glue](/connectors/pipeline/glue-pipeline)
- [NiFi](/connectors/pipeline/nifi)
- [Spline](/connectors/pipeline/spline)
## ML Model Services
- [MLflow](/connectors/ml-model/mlflow)
- [Sagemaker](/connectors/ml-model/sagemaker)
## Storage Services
- [S3](/connectors/storage/s3)
## Metadata Services
- [Amundsen](/connectors/metadata/amundsen)
- [Atlas](/connectors/metadata/atlas)
## Search Services
Fixes #5448: Implement initial Iceberg Connector using PyIceberg (#14825) * Create the iceberg connection schema * Link the IcebergConnection configuration with the forms on the UI * Add the pyiceberg dependency on the ingestion package * Create the get_connection and test_connection functions * First iteration on the iceberg ingestion logic * Add A more comprehensive implementation of the Iceberg Source * Add UnitTests * Update icebergConnection definition * Update the iceberg souce code based on new schema * Updated icebergConnecgtion schema for simplicity and to be able to configure Converters * Updated setup dependencies to be more flexible * Updated get_owner_ref logic * Fix formatting * Changed the icebergConnection json schema structure to enable the ClassConverters * Add the IcebergCatalog and IcebergFileSystem ClassConverters * Refactor the code to take into account the new jsonSchema structure * Fix formatting * Add Documentation for the Iceberg Connector * Fix Menu order for Iceberg * ui: add Iceberg service icon and constant * Fix DynamoDb Catalog issue due to how PyIceberg instantes it * Changed uri title to URI * Fix ClassConverter for Iceberg * Fix GetSecretValue for password types * Fix formatting * Fix formatting * Add Iceberg Connector Images for the docs * Add pylint disable for Hacky super() call * Add Iceberg.md for the UI docs * Fix pylint complaint * Fix pylint complaint * Fix UnitTests * fix type error and unit tests * update pipeline type checks * Fix Sonar Cloud complaints --------- Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2024-01-29 06:32:58 +01:00
- [Elasticsearch](/connectors/search/elasticsearch)