Teddy
c90a86b8ad
chore: remove typing-extension dependency ( #14757 )
2024-01-17 09:58:10 +00:00
Vijay Ravi
abe716d7fa
MINOR: Update jsonpatch package version ( #14740 )
...
* MINOR: Update jsonpatch package version
* Format
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-01-17 07:17:11 +01:00
Shiyang Xiao
241f3c68d7
Fixes #14413 : Add SAS connector ( #14415 )
...
* feat: SAS Viya connector
* refactor SASCatalog to SAS
* add SAS logo to UI and connection documentation
* doc changes
* modify ingestion logic
* revert original changes
* added support for dataflow & perfect logic for reports/datatables
* add filter doc
* more updates to perfect ingestion for each asset type
* fix a bug with table lineage not created properly
* Delete ingestion/pipelines/sasCatalog.yaml
* precomit fix
* Conversion to database connector
* minor fixes
* make custom properties type generic
* Add SAS javaEnum
* add dummy variable for sas.yaml
---------
Co-authored-by: lizmc <liz.mcintosh@sas.com>
Co-authored-by: Shiyang Xiao <Shiyang.Xiao@sas.com>
2024-01-11 06:46:57 -08:00
Ayush Shah
9c6d202555
Add Sample data, modify regex pattern ( #14467 )
2024-01-11 14:23:33 +05:30
Mayur Singal
190212c8ac
Fix #11556 : Add support for Db2 for IBM i ( #14680 )
2024-01-11 12:35:52 +05:30
Onkar Ravgan
ecdb7b9f41
Fixes 14109 and 14325: Optimised Tableau Connector ( #14548 )
...
* Optimised tableau conn
* Added comment
2024-01-08 06:33:05 +01:00
Pere Miquel Brull
0e92a975e3
#14425 - Create ingestion-base-slim image ( #14426 )
...
* #14425 - Create ingestion-base-slim image
* Format
* Bump airflow
* Bump constraints
2023-12-19 11:09:38 +01:00
Pere Miquel Brull
eaacc693bd
#12027 - Add support for Python 3.11 ( #14385 )
...
* Fix datamodel codegen and bump versions
* Add 3.11 tests
* Update hive
* pandas
* pandas
2023-12-14 15:46:58 +01:00
Pere Miquel Brull
7fcdf08ca4
#11626 & #14131 - Lineage with other Entities & attr-based xlets ( #14191 )
...
* Add OMEntity model
* Test OMEntity
* Update repr
* Fix __str__
* Add entity ref map
* Test serializer for backend
* Fix tests
* Fix serializer
* Test runner
* Add runner tests
* Update docs
* Format
2023-12-01 06:29:44 +01:00
chyueyi
b6b337e09a
feat: add support for doris datasource ( #14087 )
...
* feat: add support for doris datasource
* fix: fix python style check
* fix: add pydoris dependency
* fix: add pydoris dependency
* fix: py_format_check
* fix: parse error when doris view column is VARCHAR(*), check data length if not digit then return 1
---------
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-11-28 13:57:52 +05:30
VolkovGeoPhy
c95de19a09
great-expectations~=0.18.0 ( #14056 )
2023-11-23 12:33:07 +01:00
Pere Miquel Brull
c7e758eccc
Fix pyproject - TypeError: 'list' object is not a mapping ( #14064 )
...
* Fix pyproject - TypeError: 'list' object is not a mapping
* Add dynamic optional-dependencies
* Add dynamic optional-dependencies
* Bump datamodel
* Bump datamodel
2023-11-22 08:38:47 +01:00
Pere Miquel Brull
caaf0e7a1d
Fix #12436 - Migrate to pyproject.toml ( #14025 )
...
* test
* Use pyproject.toml
* Fix pylint
* makefile
* makefile
* Fix pylint
* isort
* pyproject
* Airflow apis pyproject
* Remove ingestion core
* isort
* Fix makefile help
2023-11-22 07:10:37 +01:00
Mohit Yadav
3f8a931e39
Bump Pom Version to 1.3.0-SNAPSHOT ( #14008 )
...
* Bump Pom Version to 1.3.0-SNAPSHOT
* chore: Fix Makefile recipe
* fix: Prepare Main Branch for Next Feature Release
* fix: Syntax issue
---------
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-11-17 11:33:47 +05:30
Ayush Shah
f94e2dbb47
Fix Hive Bytes issue, add athena yaml, fix bigquerymultiple project id token issue ( #13640 )
2023-10-18 23:48:21 +05:30
Onkar Ravgan
115cd3506d
Enable pymssql python library ( #13489 )
...
* enabled dep
* review comments
2023-10-10 12:51:52 +02:00
Pere Miquel Brull
f6a87ee02a
Fix #12082 - Bump PyAthena version ( #13464 )
2023-10-09 20:47:19 +02:00
Mayur Singal
f879656f0a
Fix #12047 : Clean commonregex package from setup ( #13439 )
2023-10-05 13:41:31 +05:30
Teddy
c4a3de6a85
fix: handle tableConfig for profiler CLI ( #13437 )
...
* fix: handle tableConfig for profiler CLI
* fix: empty commit for CI
2023-10-05 10:02:57 +02:00
Nguyen Huu Loc
ef1974edd6
Support LookML multi repos ( #13140 )
...
* Draft: Support LookML multi repos
* [Looker] manually create Dashboard datamodel
* [Looker] Support remote import & lineage for looker view
* Rollback parser.py
* refactor code
* Update code
* Remove logs & add comments
* Remove Middle & Nothing
* - Fix yield datamodel error
- Remove logs
* Support clone repo from Bitbucket
* Fix typo
* Optimize imports
* Fix pylint
---------
Co-authored-by: Loc Nguyen <loc.nguyenhuu@xendit.co>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-10-04 15:16:21 +02:00
Ayush Shah
462b2f9445
Fix Latest Pylint 3.0.0 issues ( #13413 )
...
* Fix Latest Pylint issues
* add compatible bound to pylint version
2023-10-03 07:43:09 +02:00
Pere Miquel Brull
d915254fac
Prepare Storage Connector for ADLS & Docs ( #13376 )
...
* Prepare Storage Connector for ADLS & Docs
* Format
* Fix test
2023-10-02 12:15:09 +02:00
Teddy
6ca71ae323
Issue 12679 - Handle Entity Object Instantiation Error + Refactor Workflow ( #13384 )
...
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes
* feat: __init__.py files creation
* feat: updated workflow import classes in code and doc
* feat: moved kpi runner from runner to processor folder
* fix: skip failure on list entities
* feat: deleted unused files
* feat: updated status reporter
* feat: ran linting
* feat: fix test error with typing and fqn
* feat: updated test dependencies
* feat: ran linting
* feat: move execution order up
* feat: updated cost analysis report to align with new workflow
* feat: fix entity already exists for pipeline entity status
* feat: ran python linting
* feat: move skip_on_failure to method
* feat: ran linting
---------
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-02 12:05:30 +02:00
Pere Miquel Brull
2c3ff8dc08
Prep v1.1.7 migrations to address test cases & suites ( #13345 )
...
* Prep v1.1.7 migrations to address test cases
* get or create executable suite
* Format
* Fix tests
* Add postgres
2023-09-27 11:49:21 +02:00
vanshika18
6f92c3be1f
fix #11882 upgrade MLflow to 2.x ( #13299 )
2023-09-26 11:04:15 +05:30
VolkovGeoPhy
68fcd551fe
GX up to 0.17+ ( #12038 )
...
* GX up to 0.17+
* ~=
---------
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-19 18:33:31 +02:00
Teddy
a424c37a34
[Proposal] Playwright CI ( #13248 )
...
* feat: playwright CI tests
* feat: ran python linting
* feat: updated workflow to run in PR
* feat: updated server start step
* feat: fixed playwright run command
* feat: fix matrix value
* feat: removed browser option in test run
* feat: updated typo in make file
* feat: move playwritgh installation
* feat: fail CI on error
* feat: save tracing on failure
* feat: enforce expect on .fill
* feat: change tracing to video
* feat: clean up code and update CI workflow for main branch
2023-09-19 14:44:36 +02:00
Ayush Shah
947565d8a3
check importlib setup ( #13200 )
2023-09-15 12:03:47 +05:30
Mayur Singal
791b2f4c38
Use Collate SQLLineage Package for lineage ( #13173 )
2023-09-13 16:14:03 +05:30
Mayur Singal
4e633877b3
Fix ElasticSearch Test Connection & Deploy ( #13061 )
2023-09-08 12:40:48 +05:30
vanshika18
3cc15e6d0b
Fixed #8418 added new connector couchbase ( #12836 )
2023-08-29 11:16:32 +05:30
Ayush Shah
8776069940
Fix Qliksense import ( #12849 )
2023-08-11 12:28:05 +05:30
Pere Miquel Brull
e97d4befb1
Fix #12770 - Cleanup DL structure & Readers & Python 3.8 ( #12776 )
2023-08-09 16:07:16 +05:30
Ayush Shah
2e259a9899
Python Dependency Conflict fixes ( #12667 )
...
* Fixes Dependency conflict
* add fix to dep
2023-07-31 14:44:25 +02:00
Onkar Ravgan
795498c54a
Fixed athena import and lineage/usage validation ( #12663 )
...
* Athena fixes
* fixed to lower version
2023-07-31 12:37:48 +02:00
Mayur Singal
0c6435ae9b
Fix #2957 : Add support for qlik sense enterprise connector ( #12450 )
2023-07-19 21:08:05 +05:30
Ayush Shah
940ab3d183
Fix Dependency issue - mssql ( #12499 )
2023-07-19 07:27:38 +00:00
Onkar Ravgan
a3e11ac59c
disabled pymmsql dependancy ( #12481 )
2023-07-18 19:07:19 +05:30
Pere Miquel Brull
a3bff293a4
Fix #12190 - Bring support for Airflow 2.6 & Use Airflow 2.6.3 as the new ingestion base ( #12398 )
...
* Bump Airflow version
* Support Airflow 2.6
* Lint
* Bump airflow version
* Fix airflow 2.6 lineage
* Fix airflow 2.6 lineage
2023-07-13 06:41:59 +02:00
Anuj359
7f55b808e5
fix(chore): Prepare Main Branch with next release (1.2.0) ( #12364 )
...
* Updating Dockerfile with multistage
* Updating reviewed changes
* Docker Development changes
* Docker workflow changes
* Arguments update
* Script path update
* Arguments update
* Resolving the reviewed suggestions
* Preparing for next release and updating docker sort value changes
* Adding .dev0 in the python related release version
* Adding .dev0 in the python related release version
2023-07-12 16:55:18 +05:30
Megumi AIKAWA
9aede5e821
Fixes 11924 : Add pgspider support ( #12159 )
...
* Add PGSpider service connector
* add sqlalchemy-pgspider to setup.py
* Delete unnecessary code and move files.
* Change to work as a Postgres connector sub-module
* Rebuild PGSpiderLineage class into a set of funcs
---------
Co-authored-by: Pham Ngoc Son <son1.phamngoc@toshiba.co.jp>
2023-07-05 09:18:59 +02:00
Onkar Ravgan
78d7faf04a
Added Azure Blob Storage to dbt config ( #11839 )
2023-06-22 10:58:38 +05:30
Pere Miquel Brull
4a8554c313
Fix Redshift performance and memory issues for ingestion-base ( #12002 )
...
* base op
* Update redshift deps and ingestion-base image
* Remove vim
* Remove vim
2023-06-19 12:09:09 +02:00
Mayur Singal
7fa963eec3
Fix #1076 : Add mongodb support ( #11943 )
2023-06-15 11:14:22 +05:30
Mayur Singal
9e6e00caf9
Fix #9282 : Support Databricks Unity Catalog ( #11896 )
2023-06-07 11:50:31 +05:30
Pere Miquel Brull
11c07ee8ab
Fix #11516 - SAP Hana Connector ( #11777 )
...
* SAP Hana skeleton
* Add SAP Hana Connector
* Fix ingestion and docs
* Prep SAP Hana Profiler
* Linting
* Update index.md
* Revert: Update index.md
---------
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-31 16:00:31 +02:00
Chirag Madlani
7adc291364
fix(ui): circular deps for entityReference.json ( #11760 )
...
* fix(ui): circular deps for entityReference.json
* Fix circular Dependency python
* Cap Delta Spark version
---------
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-26 18:02:21 +05:30
Onkar Ravgan
3fbddc2a03
upgrade kafka dep ( #11721 )
2023-05-23 09:59:12 -07:00
Sriharsha Chintalapani
6509a3670a
Fix #11664 : Refactor patch_mixin to use jsonpatch lib ( #11696 )
...
* Fix #11664 : Refactor patch_mixin to use jsonpatch lib
* Migrate to jsonpatch
* Fix nested cols
* Format
* Update patch_description
* Table constraints
* tag
* owner
* column tag
* column desc
* Format
* Format
* Fix log
* Update dbt patch
* Update column fqn
* Fix test
* Fix tests
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-05-23 15:47:11 +02:00
Pere Miquel Brull
1370325762
Fix typing_extension verson ( #11719 )
...
* fix extension
* fix extension
2023-05-23 10:09:31 +02:00
