373cafcda2
* Create the iceberg connection schema * Link the IcebergConnection configuration with the forms on the UI * Add the pyiceberg dependency on the ingestion package * Create the get_connection and test_connection functions * First iteration on the iceberg ingestion logic * Add A more comprehensive implementation of the Iceberg Source * Add UnitTests * Update icebergConnection definition * Update the iceberg souce code based on new schema * Updated icebergConnecgtion schema for simplicity and to be able to configure Converters * Updated setup dependencies to be more flexible * Updated get_owner_ref logic * Fix formatting * Changed the icebergConnection json schema structure to enable the ClassConverters * Add the IcebergCatalog and IcebergFileSystem ClassConverters * Refactor the code to take into account the new jsonSchema structure * Fix formatting * Add Documentation for the Iceberg Connector * Fix Menu order for Iceberg * ui: add Iceberg service icon and constant * Fix DynamoDb Catalog issue due to how PyIceberg instantes it * Changed uri title to URI * Fix ClassConverter for Iceberg * Fix GetSecretValue for password types * Fix formatting * Fix formatting * Add Iceberg Connector Images for the docs * Add pylint disable for Hacky super() call * Add Iceberg.md for the UI docs * Fix pylint complaint * Fix pylint complaint * Fix UnitTests * fix type error and unit tests * update pipeline type checks * Fix Sonar Cloud complaints --------- Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
3.9 KiB
| title | slug |
|---|---|
| Connectors | /connectors |
Connectors
OpenMetadata can extract metadata from the following list of connectors below.
Ingestion Deployment
To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If you want to install it manually in an already existing Airflow host, you can follow this guide.
If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check the following docs to run the Ingestion Framework in any orchestrator externally.
{% tilesContainer %} {% tile title="Run Connectors from the OpenMetadata UI" description="Learn how to manage your deployment to run connectors from the UI" link="/deployment/ingestion/openmetadata" / %} {% tile title="External Schedulers" description="Get more information about running the Ingestion Framework Externally" link="/deployment/ingestion" / %} {% /tilesContainer %}
Database / DataWarehouse Services
- Athena
- AzureSQL
- BigQuery
- Clickhouse
- Couchbase
- Data lake
- Databricks SQL
- DB2
- Delta Lake
- Domo Database
- Druid
- DynamoDB
- Glue
- Greenplum
- Hive
- Iceberg
- Impala
- MariaDB
- MongoDB
- MSSQL
- MySQL
- Oracle
- PinotDB
- Postgres
- Presto
- Redshift
- Salesforce
- SAP Hana
- SAS
- SingleStore
- Snowflake
- SQLite
- Trino
- Unity Catalog
- Vertica