38 Commits

Author SHA1 Message Date
IceS2
373cafcda2
Fixes #5448: Implement initial Iceberg Connector using PyIceberg (#14825)
* Create the iceberg connection schema

* Link the IcebergConnection configuration with the forms on the UI

* Add the pyiceberg dependency on the ingestion package

* Create the get_connection and test_connection functions

* First iteration on the iceberg ingestion logic

* Add A more comprehensive implementation of the Iceberg Source

* Add UnitTests

* Update icebergConnection definition

* Update the iceberg souce code based on new schema

* Updated icebergConnecgtion schema for simplicity and to be able to configure Converters

* Updated setup dependencies to be more flexible

* Updated get_owner_ref logic

* Fix formatting

* Changed the icebergConnection json schema structure to enable the ClassConverters

* Add the IcebergCatalog and IcebergFileSystem ClassConverters

* Refactor the code to take into account the new jsonSchema structure

* Fix formatting

* Add Documentation for the Iceberg Connector

* Fix Menu order for Iceberg

* ui: add Iceberg service icon and constant

* Fix DynamoDb Catalog issue due to how PyIceberg instantes it

* Changed uri title to URI

* Fix ClassConverter for Iceberg

* Fix GetSecretValue for password types

* Fix formatting

* Fix formatting

* Add Iceberg Connector Images for the docs

* Add pylint disable for Hacky super() call

* Add Iceberg.md for the UI docs

* Fix pylint complaint

* Fix pylint complaint

* Fix UnitTests

* fix type error and unit tests

* update pipeline type checks

* Fix Sonar Cloud complaints

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2024-01-29 06:32:58 +01:00
Teddy
0f80ee2cbc
MINOR - Go client documentation (#14628)
* feat: primitive OpenMetadata Go client

* chore: make command for Go client

* fix: add retry wait logic

* doc: added go documentation

* fix: moved openmetadata-go-client to https://github.com/open-metadata/openmetadata-sdk

* fix: added new line back in Makefile file
2024-01-28 10:28:00 -08:00
NiharDoshi99
c1d62186df
MINOR - metadata tag extraction for Databricks (#14874)
* metadata tag extraction for databaricks

* fix python test

* changes as per comment

* fix python test

* fix python checkstyle
2024-01-26 07:09:24 +01:00
Mayur Singal
59d97357c6
MINOR: Fix hive yaml docs (#14842) 2024-01-25 15:35:33 +05:30
Shiyang Xiao
9f5a70bd71
MINOR - update docs & added unit test for SAS Connector (#14743)
Co-authored-by: Shiyang Xiao <Shiyang.Xiao@sas.com>
2024-01-23 14:55:29 -08:00
Mayur Singal
2c068dd876
Minor: Docs add compute metric config in docs (#14798) 2024-01-19 23:02:13 +05:30
Shilpa Vernekar
d7f4172416
DOCS: Knowledge Center Article and other changes (#14795) 2024-01-19 10:44:05 +00:00
Pere Miquel Brull
05038cc99c
MINOR - Add notes on external ingestion docs (#14756)
* MINOR - Add notes on external ingestion docs

* GCS Composer
2024-01-18 11:07:34 +01:00
IceS2
69bf469a76
MINOR - Update Developers documentation to make it easier to follow (#14754)
* chore(docs): Changed the structure of the file to facilitate following it

* fix(docs): Add missing cd command before starting the server from script
2024-01-18 08:50:50 +01:00
Pere Miquel Brull
ddf7a0d30f
MINOR - Run ingestion externally docs (#14751)
* add docs

* MINOR - Run Ingestion Externally docs
2024-01-17 08:21:34 +01:00
NiharDoshi99
c120ae98a4
#14727 change all all_ query to dba_ query (#14748) 2024-01-17 12:38:42 +05:30
Mayur Singal
9694e19599
Minor: Docs - MSSQL Usage & Lineage Requirements (#14746) 2024-01-17 11:41:50 +05:30
Onkar Ravgan
64a4e1afce
Fix 12180, 14158: Added LF tags to Athena (#14718)
* Added LF tags to athena

* fixed pytests

* Added docs
2024-01-16 14:24:31 +05:30
Mayur Singal
d1a1003a9d
Minor: Docs - Backup & Restore Troubleshooting steps (#14710) 2024-01-15 10:11:23 -08:00
Mayur Singal
d6b90ff459
Minor: Added feature table to pipelines & messaging connectors (#14707) 2024-01-15 15:26:11 +05:30
Shilpa Vernekar
7eeaa05279
Docs: Change Glossary Video Link (#14703) 2024-01-12 22:03:55 +05:30
Mayur Singal
190212c8ac
Fix #11556: Add support for Db2 for IBM i (#14680) 2024-01-11 12:35:52 +05:30
Shilpa Vernekar
befba918d8
Docs: Minor change - Glossary Import Video added (#14675) 2024-01-10 18:58:31 +00:00
Mayur Singal
616e3bd89c
Minor: Fix postgres version on docs (#14642) 2024-01-10 22:53:05 +05:30
Mayur Singal
9288620c29
Minor: Fix Feature Tables of Connectors (#14658) 2024-01-10 12:48:42 +05:30
Sriharsha Chintalapani
182a19f3cc
Add docs on where to get the API token (#14633) 2024-01-09 07:55:40 -08:00
Mohit Yadav
dbd9f1381a
[DOCS] Update Troublesbooting docs for SSL to new env and keys (#14654)
* Update Troublesbooting docs for SSL to new env and keys in openmetadata.yaml

* Update PipelineServiceClientConfig
2024-01-09 07:54:56 -08:00
Mayur Singal
2832bbcc8c
Fix #14362: Spark Pipeline Connector & Docs (#14626) 2024-01-09 12:40:43 +05:30
Onkar Ravgan
2c539f912d
Added alation connector docs (#14614) 2024-01-08 12:28:25 +05:30
Sriharsha Chintalapani
a92fedf544
DB2 connector docs, call out db2 package installation (#14605)
* DB2 connector docs, call out db2 package installation

* DB2 connector docs, call out db2 package installation
2024-01-06 19:25:58 -08:00
Mayur Singal
a789fc86d6
Fix #13053: Remove Connection URI config MongoDB (#14584)
* Fix #13053: Remove Connection URI cofig MongoDB

* pyformat & test fixes
2024-01-05 10:51:12 -08:00
Aniket Katkar
4b1befacaf
added sample content for the api and sdk pages. (#14543) 2024-01-04 21:26:56 +05:30
Shilpa Vernekar
0dfa1e2de1
Broken links corrected (#14489) 2023-12-22 13:32:47 -08:00
Akash Jain
3a51f0b7c2
fix: SSL Docs for OPENMETADATA_OPTS (#14465)
* fix: SSL Docs for `OPENMETADATA_OPTS`

* fix(docs): update the files in 1.3.0 docs section as well
2023-12-21 10:42:06 +05:30
Ayush Shah
c3fac5e97e
Fix Adls connection typo (#14449) 2023-12-20 14:25:54 +01:00
Shilpa Vernekar
1dc79bfd3c
Docs: Links corrected (#14459) 2023-12-20 18:20:33 +05:30
Onkar Ravgan
7d032977d7
Fix 14084: Removed Airflow Mssql backend connection (#14420)
* Removed Airflow Mssql backend conn

* added breaking changes docs
2023-12-20 12:28:28 +05:30
Shilpa Vernekar
cce059d98f
Docs: Link Corrected (#14452) 2023-12-20 06:14:43 +00:00
Ayush Shah
03f884b376
MINOR: Fix Typos (#14448) 2023-12-20 10:30:23 +05:30
Pere Miquel Brull
a83a5ba3a3
MINOR - Skip delta tests for 3.11 (#14398)
* MINOR - Bump delta for 3.11

* Update flags

* MINOR - Bump delta for 3.11

* Update tests regex

* Update version

* Deprecations

* Format

* Version

* Try delta spark

* Skip delta tests for 3.11

* Update ingestion/tests/unit/topology/pipeline/test_airflow.py
2023-12-18 17:01:57 +01:00
Pere Miquel Brull
d4d8f3655f
#14043 - Fix metadata backup memory profile (#14378)
* Update looker docs

* Fix db dump
2023-12-14 15:46:35 +01:00
Shilpa Vernekar
91899233f9
Docs: Updated v1.3 & changed to How-to Guides (#14380) 2023-12-13 09:28:06 -08:00
Pere Miquel Brull
9d5c3a490f
MINOR - Prepare 1.3 docs directories (#14357) 2023-12-13 18:33:08 +05:30