14 Commits

Author SHA1 Message Date
Onkar Ravgan
370abda43f
dbt docs revamp (#15787) 2024-04-03 20:42:30 +05:30
Shilpa Vernekar
f28214e0c0
Missing docs added (#15763) 2024-04-03 09:03:08 +00:00
Ayush Shah
1c2fbdd9f4
MINOR: Add / Fix GCS and ADLS - docs, bugs (#15502)
Add GCS and ADLS docs
2024-03-12 21:13:24 +05:30
mgorsk1
98850ab5cc
feat: OpenLineage integration (#15317)
* 🎉 Init OpenLineage connector

Co-authored-by: dechoma <dominik.choma@gmail.com>

* MLH - make linter happy

* review fixes

* 🐛 Fix path for ol event in tests

* 🐛 Fix path for ol event in tests

* Update ingestion/setup.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/models.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* review fixes 2

* linter

* review

* review

* make linter happy

* fix test_yield_pipeline_lineage_details test

* make linter happy

* fix tests

* fix tests 2

---------

Co-authored-by: dechoma <dominik.choma@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-03-12 08:39:25 +01:00
Imri Paran
7036b3cbf7
MINOR: BigTable connector docs (#15145)
* feat(connector): add BigTable

* bigtable work

1. docstrings
2. tests
3. created a Row BaseModel
4. implemented a ClassConverter

* docs moved to separate PR

* format files

* docs(bigtable): added docs

* fix after merge
2024-02-19 08:11:29 +01:00
Mayur Singal
e6c2c36f11
MINOR: Add databricks configuration docs for spark agent (#15182) 2024-02-14 21:16:34 +05:30
Pere Miquel Brull
eb501c1996
MINOR - Breaking change for alerts docs (#15026) 2024-02-05 11:54:59 +01:00
IceS2
373cafcda2
Fixes #5448: Implement initial Iceberg Connector using PyIceberg (#14825)
* Create the iceberg connection schema

* Link the IcebergConnection configuration with the forms on the UI

* Add the pyiceberg dependency on the ingestion package

* Create the get_connection and test_connection functions

* First iteration on the iceberg ingestion logic

* Add A more comprehensive implementation of the Iceberg Source

* Add UnitTests

* Update icebergConnection definition

* Update the iceberg souce code based on new schema

* Updated icebergConnecgtion schema for simplicity and to be able to configure Converters

* Updated setup dependencies to be more flexible

* Updated get_owner_ref logic

* Fix formatting

* Changed the icebergConnection json schema structure to enable the ClassConverters

* Add the IcebergCatalog and IcebergFileSystem ClassConverters

* Refactor the code to take into account the new jsonSchema structure

* Fix formatting

* Add Documentation for the Iceberg Connector

* Fix Menu order for Iceberg

* ui: add Iceberg service icon and constant

* Fix DynamoDb Catalog issue due to how PyIceberg instantes it

* Changed uri title to URI

* Fix ClassConverter for Iceberg

* Fix GetSecretValue for password types

* Fix formatting

* Fix formatting

* Add Iceberg Connector Images for the docs

* Add pylint disable for Hacky super() call

* Add Iceberg.md for the UI docs

* Fix pylint complaint

* Fix pylint complaint

* Fix UnitTests

* fix type error and unit tests

* update pipeline type checks

* Fix Sonar Cloud complaints

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2024-01-29 06:32:58 +01:00
Shiyang Xiao
9f5a70bd71
MINOR - update docs & added unit test for SAS Connector (#14743)
Co-authored-by: Shiyang Xiao <Shiyang.Xiao@sas.com>
2024-01-23 14:55:29 -08:00
Shilpa Vernekar
d7f4172416
DOCS: Knowledge Center Article and other changes (#14795) 2024-01-19 10:44:05 +00:00
Mayur Singal
2832bbcc8c
Fix #14362: Spark Pipeline Connector & Docs (#14626) 2024-01-09 12:40:43 +05:30
Onkar Ravgan
e4ea06834f
Added redploy docs (#14526) 2023-12-29 10:04:18 +01:00
Shilpa Vernekar
91899233f9
Docs: Updated v1.3 & changed to How-to Guides (#14380) 2023-12-13 09:28:06 -08:00
Pere Miquel Brull
9d5c3a490f
MINOR - Prepare 1.3 docs directories (#14357) 2023-12-13 18:33:08 +05:30