389 Commits

Author SHA1 Message Date
Mayur Singal
a9fc51ec8b
MINOR: Change sqllineage import to collate_sqllineage (#14870) 2024-02-05 19:44:08 +05:30
IceS2
373cafcda2
Fixes #5448: Implement initial Iceberg Connector using PyIceberg (#14825)
* Create the iceberg connection schema

* Link the IcebergConnection configuration with the forms on the UI

* Add the pyiceberg dependency on the ingestion package

* Create the get_connection and test_connection functions

* First iteration on the iceberg ingestion logic

* Add A more comprehensive implementation of the Iceberg Source

* Add UnitTests

* Update icebergConnection definition

* Update the iceberg souce code based on new schema

* Updated icebergConnecgtion schema for simplicity and to be able to configure Converters

* Updated setup dependencies to be more flexible

* Updated get_owner_ref logic

* Fix formatting

* Changed the icebergConnection json schema structure to enable the ClassConverters

* Add the IcebergCatalog and IcebergFileSystem ClassConverters

* Refactor the code to take into account the new jsonSchema structure

* Fix formatting

* Add Documentation for the Iceberg Connector

* Fix Menu order for Iceberg

* ui: add Iceberg service icon and constant

* Fix DynamoDb Catalog issue due to how PyIceberg instantes it

* Changed uri title to URI

* Fix ClassConverter for Iceberg

* Fix GetSecretValue for password types

* Fix formatting

* Fix formatting

* Add Iceberg Connector Images for the docs

* Add pylint disable for Hacky super() call

* Add Iceberg.md for the UI docs

* Fix pylint complaint

* Fix pylint complaint

* Fix UnitTests

* fix type error and unit tests

* update pipeline type checks

* Fix Sonar Cloud complaints

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2024-01-29 06:32:58 +01:00
Teddy
c90a86b8ad
chore: remove typing-extension dependency (#14757) 2024-01-17 09:58:10 +00:00
Vijay Ravi
abe716d7fa
MINOR: Update jsonpatch package version (#14740)
* MINOR: Update jsonpatch package version

* Format

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-01-17 07:17:11 +01:00
Shiyang Xiao
241f3c68d7
Fixes #14413: Add SAS connector (#14415)
* feat: SAS Viya connector

* refactor SASCatalog to SAS

* add SAS logo to UI and connection documentation

* doc changes

* modify ingestion logic

* revert original changes

* added support for dataflow & perfect logic for reports/datatables

* add filter doc

* more updates to perfect ingestion for each asset type

* fix a bug with table lineage not created properly

* Delete ingestion/pipelines/sasCatalog.yaml

* precomit fix

* Conversion to database connector

* minor fixes

* make custom properties type generic

* Add SAS javaEnum

* add dummy variable for sas.yaml

---------

Co-authored-by: lizmc <liz.mcintosh@sas.com>
Co-authored-by: Shiyang Xiao <Shiyang.Xiao@sas.com>
2024-01-11 06:46:57 -08:00
Ayush Shah
9c6d202555
Add Sample data, modify regex pattern (#14467) 2024-01-11 14:23:33 +05:30
Mayur Singal
190212c8ac
Fix #11556: Add support for Db2 for IBM i (#14680) 2024-01-11 12:35:52 +05:30
Onkar Ravgan
ecdb7b9f41
Fixes 14109 and 14325: Optimised Tableau Connector (#14548)
* Optimised tableau conn

* Added comment
2024-01-08 06:33:05 +01:00
Pere Miquel Brull
0e92a975e3
#14425 - Create ingestion-base-slim image (#14426)
* #14425 - Create ingestion-base-slim image

* Format

* Bump airflow

* Bump constraints
2023-12-19 11:09:38 +01:00
Pere Miquel Brull
eaacc693bd
#12027 - Add support for Python 3.11 (#14385)
* Fix datamodel codegen and bump versions

* Add 3.11 tests

* Update hive

* pandas

* pandas
2023-12-14 15:46:58 +01:00
Pere Miquel Brull
7fcdf08ca4
#11626 & #14131 - Lineage with other Entities & attr-based xlets (#14191)
* Add OMEntity model

* Test OMEntity

* Update repr

* Fix __str__

* Add entity ref map

* Test serializer for backend

* Fix tests

* Fix serializer

* Test runner

* Add runner tests

* Update docs

* Format
2023-12-01 06:29:44 +01:00
chyueyi
b6b337e09a
feat: add support for doris datasource (#14087)
* feat: add support for doris datasource

* fix: fix python style check

* fix: add pydoris dependency

* fix: add pydoris dependency

* fix: py_format_check

* fix: parse error when doris view column is VARCHAR(*), check data length if not digit then return 1

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-11-28 13:57:52 +05:30
VolkovGeoPhy
c95de19a09
great-expectations~=0.18.0 (#14056) 2023-11-23 12:33:07 +01:00
Pere Miquel Brull
c7e758eccc
Fix pyproject - TypeError: 'list' object is not a mapping (#14064)
* Fix pyproject - TypeError: 'list' object is not a mapping

* Add dynamic optional-dependencies

* Add dynamic optional-dependencies

* Bump datamodel

* Bump datamodel
2023-11-22 08:38:47 +01:00
Pere Miquel Brull
caaf0e7a1d
Fix #12436 - Migrate to pyproject.toml (#14025)
* test

* Use pyproject.toml

* Fix pylint

* makefile

* makefile

* Fix pylint

* isort

* pyproject

* Airflow apis pyproject

* Remove ingestion core

* isort

* Fix makefile help
2023-11-22 07:10:37 +01:00
Mohit Yadav
3f8a931e39
Bump Pom Version to 1.3.0-SNAPSHOT (#14008)
* Bump Pom Version to 1.3.0-SNAPSHOT

* chore: Fix Makefile recipe

* fix: Prepare Main Branch for Next Feature Release

* fix: Syntax issue

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-11-17 11:33:47 +05:30
Ayush Shah
f94e2dbb47
Fix Hive Bytes issue, add athena yaml, fix bigquerymultiple project id token issue (#13640) 2023-10-18 23:48:21 +05:30
Onkar Ravgan
115cd3506d
Enable pymssql python library (#13489)
* enabled dep

* review comments
2023-10-10 12:51:52 +02:00
Pere Miquel Brull
f6a87ee02a
Fix #12082 - Bump PyAthena version (#13464) 2023-10-09 20:47:19 +02:00
Mayur Singal
f879656f0a
Fix #12047: Clean commonregex package from setup (#13439) 2023-10-05 13:41:31 +05:30
Teddy
c4a3de6a85
fix: handle tableConfig for profiler CLI (#13437)
* fix: handle tableConfig for profiler CLI

* fix: empty commit for CI
2023-10-05 10:02:57 +02:00
Nguyen Huu Loc
ef1974edd6
Support LookML multi repos (#13140)
* Draft: Support LookML multi repos

* [Looker] manually create Dashboard datamodel

* [Looker] Support remote import & lineage for looker view

* Rollback parser.py

* refactor code

* Update code

* Remove logs & add comments

* Remove Middle & Nothing

* - Fix yield datamodel error
- Remove logs

* Support clone repo from Bitbucket

* Fix typo

* Optimize imports

* Fix pylint

---------

Co-authored-by: Loc Nguyen <loc.nguyenhuu@xendit.co>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-10-04 15:16:21 +02:00
Ayush Shah
462b2f9445
Fix Latest Pylint 3.0.0 issues (#13413)
* Fix Latest Pylint issues

* add compatible bound to pylint version
2023-10-03 07:43:09 +02:00
Pere Miquel Brull
d915254fac
Prepare Storage Connector for ADLS & Docs (#13376)
* Prepare Storage Connector for ADLS & Docs

* Format

* Fix test
2023-10-02 12:15:09 +02:00
Teddy
6ca71ae323
Issue 12679 - Handle Entity Object Instantiation Error + Refactor Workflow (#13384)
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: ran linting

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-02 12:05:30 +02:00
Pere Miquel Brull
2c3ff8dc08
Prep v1.1.7 migrations to address test cases & suites (#13345)
* Prep v1.1.7 migrations to address test cases

* get or create executable suite

* Format

* Fix tests

* Add postgres
2023-09-27 11:49:21 +02:00
vanshika18
6f92c3be1f
fix #11882 upgrade MLflow to 2.x (#13299) 2023-09-26 11:04:15 +05:30
VolkovGeoPhy
68fcd551fe
GX up to 0.17+ (#12038)
* GX up to 0.17+

* ~=

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-19 18:33:31 +02:00
Teddy
a424c37a34
[Proposal] Playwright CI (#13248)
* feat: playwright CI tests

* feat: ran python linting

* feat: updated workflow to run in PR

* feat: updated server start step

* feat: fixed playwright run command

* feat: fix matrix value

* feat: removed browser option in test run

* feat: updated typo in make file

* feat: move playwritgh installation

* feat: fail CI on error

* feat: save tracing on failure

* feat: enforce expect on .fill

* feat: change tracing to video

* feat: clean up code and update CI workflow for main branch
2023-09-19 14:44:36 +02:00
Ayush Shah
947565d8a3
check importlib setup (#13200) 2023-09-15 12:03:47 +05:30
Mayur Singal
791b2f4c38
Use Collate SQLLineage Package for lineage (#13173) 2023-09-13 16:14:03 +05:30
Mayur Singal
4e633877b3
Fix ElasticSearch Test Connection & Deploy (#13061) 2023-09-08 12:40:48 +05:30
vanshika18
3cc15e6d0b
Fixed #8418 added new connector couchbase (#12836) 2023-08-29 11:16:32 +05:30
Ayush Shah
8776069940
Fix Qliksense import (#12849) 2023-08-11 12:28:05 +05:30
Pere Miquel Brull
e97d4befb1
Fix #12770 - Cleanup DL structure & Readers & Python 3.8 (#12776) 2023-08-09 16:07:16 +05:30
Ayush Shah
2e259a9899
Python Dependency Conflict fixes (#12667)
* Fixes Dependency conflict

* add fix to dep
2023-07-31 14:44:25 +02:00
Onkar Ravgan
795498c54a
Fixed athena import and lineage/usage validation (#12663)
* Athena fixes

* fixed to lower version
2023-07-31 12:37:48 +02:00
Mayur Singal
0c6435ae9b
Fix #2957: Add support for qlik sense enterprise connector (#12450) 2023-07-19 21:08:05 +05:30
Ayush Shah
940ab3d183
Fix Dependency issue - mssql (#12499) 2023-07-19 07:27:38 +00:00
Onkar Ravgan
a3e11ac59c
disabled pymmsql dependancy (#12481) 2023-07-18 19:07:19 +05:30
Pere Miquel Brull
a3bff293a4
Fix #12190 - Bring support for Airflow 2.6 & Use Airflow 2.6.3 as the new ingestion base (#12398)
* Bump Airflow version

* Support Airflow 2.6

* Lint

* Bump airflow version

* Fix airflow 2.6 lineage

* Fix airflow 2.6 lineage
2023-07-13 06:41:59 +02:00
Anuj359
7f55b808e5
fix(chore): Prepare Main Branch with next release (1.2.0) (#12364)
* Updating Dockerfile with multistage

* Updating reviewed changes

* Docker Development changes

* Docker workflow changes

* Arguments update

* Script path update

* Arguments update

* Resolving the reviewed suggestions

* Preparing for next release and updating docker sort value changes

* Adding .dev0 in the python related release version

* Adding .dev0 in the python related release version
2023-07-12 16:55:18 +05:30
Megumi AIKAWA
9aede5e821
Fixes 11924 : Add pgspider support (#12159)
* Add PGSpider service connector

* add sqlalchemy-pgspider to setup.py

* Delete unnecessary code and move files.

* Change to work as a Postgres connector sub-module

* Rebuild PGSpiderLineage class into a set of funcs

---------

Co-authored-by: Pham Ngoc Son <son1.phamngoc@toshiba.co.jp>
2023-07-05 09:18:59 +02:00
Onkar Ravgan
78d7faf04a
Added Azure Blob Storage to dbt config (#11839) 2023-06-22 10:58:38 +05:30
Pere Miquel Brull
4a8554c313
Fix Redshift performance and memory issues for ingestion-base (#12002)
* base op

* Update redshift deps and ingestion-base image

* Remove vim

* Remove vim
2023-06-19 12:09:09 +02:00
Mayur Singal
7fa963eec3
Fix #1076: Add mongodb support (#11943) 2023-06-15 11:14:22 +05:30
Mayur Singal
9e6e00caf9
Fix #9282: Support Databricks Unity Catalog (#11896) 2023-06-07 11:50:31 +05:30
Pere Miquel Brull
11c07ee8ab
Fix #11516 - SAP Hana Connector (#11777)
* SAP Hana skeleton

* Add SAP Hana Connector

* Fix ingestion and docs

* Prep SAP Hana Profiler

* Linting

* Update index.md

* Revert: Update index.md

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-31 16:00:31 +02:00
Chirag Madlani
7adc291364
fix(ui): circular deps for entityReference.json (#11760)
* fix(ui): circular deps for entityReference.json

* Fix circular Dependency python

* Cap Delta Spark version

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-26 18:02:21 +05:30
Onkar Ravgan
3fbddc2a03
upgrade kafka dep (#11721) 2023-05-23 09:59:12 -07:00