723 Commits

Author SHA1 Message Date
Teddy
8e4388c35e
Issue 11345 (#12859)
* feat: added serviceName dimension to entity report

* feat: fix python test
2023-08-14 08:05:14 +02:00
Ayush Shah
5ea4de646c
Fix Postgres Profiler for incompatible versions (#12795) 2023-08-11 15:32:43 +05:30
Pere Miquel Brull
d8e288626c
Storage connection naming, ADLS service and namespace packaging (#12810)
* Update storage connection naming

* Update storage services and remove typing of generic service

* Remove inits for namespace package

* Add test connection

* Revert "Remove inits for namespace package"

This reverts commit 89f1ae9ea8a633593776f51caf2586d2110d4636.

* Add enum

* Remove init to allow namespace packaging in some modules

* Remove init to allow namespace packaging in some modules

* Move examples and fix test

* Format

* Linting
2023-08-10 07:32:58 +02:00
Pere Miquel Brull
e97d4befb1
Fix #12770 - Cleanup DL structure & Readers & Python 3.8 (#12776) 2023-08-09 16:07:16 +05:30
Pere Miquel Brull
a82c2f001c
Fix #9704 - Add inline TNS string (#12711)
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-08-07 11:04:13 +02:00
Onkar Ravgan
bf44c7996a
fixed nested records avro (#12633)
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-07-31 11:54:29 +05:30
vanshika18
72e4d0070f
Support MSSLQ handle multiple databases with pyodbc connection (#12526) 2023-07-26 10:43:24 +05:30
Ayush Shah
ab1ec50c2c
Fixes Mssql Ntext, text and Image (#12490) 2023-07-20 13:34:35 +05:30
Mayur Singal
0c6435ae9b
Fix #2957: Add support for qlik sense enterprise connector (#12450) 2023-07-19 21:08:05 +05:30
Ayush Shah
27a0c9e802
Fix Docker Import (#12455) 2023-07-17 12:50:11 +05:30
Teddy
42a426226e
Fixes Issue #11803 #12103 - Add BigQuery Struct Support (#12435)
* ref: implemented interface for profiler components + removed struct logic

* ref: ran python linting

* ref: added UML diagram to readme.md

* ref: empty commit for labeler check

* ref: remove multiple context manager for 3.7 3.8 compatibility

* ref: remove

* fix: mapper logic for BQ struct types

* feat: added BQ support for structs

* feat: clean code smell + handle null self.col.table value

* feat: ran python linting

* feat: updated test for profiler handler + disabled flaky test

* Update ingestion/tests/unit/profiler/pandas/test_sample.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

---------

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-07-14 09:12:46 +02:00
Teddy
54fbe250a1
fix: import error + BQ E2E CLI (#12420) 2023-07-13 13:35:37 +02:00
vanshika18
9174cea1ab
Add support for Superset datamodels (#12183) 2023-07-13 14:10:37 +05:30
Pere Miquel Brull
a3bff293a4
Fix #12190 - Bring support for Airflow 2.6 & Use Airflow 2.6.3 as the new ingestion base (#12398)
* Bump Airflow version

* Support Airflow 2.6

* Lint

* Bump airflow version

* Fix airflow 2.6 lineage

* Fix airflow 2.6 lineage
2023-07-13 06:41:59 +02:00
Teddy
b89cf64f14
Clean up profiler (#12369)
* ref: implemented interface for profiler components + removed struct logic

* ref: ran python linting

* ref: added UML diagram to readme.md

* ref: empty commit for labeler check

* ref: remove multiple context manager for 3.7 3.8 compatibility

* ref: remove
2023-07-12 17:02:32 +02:00
Pere Miquel Brull
6a1b1190a7
Fix #12313 - Airflow lineage parsing and schedules (#12339)
* Fix docs

* Fix lineage and intervals

* Format
2023-07-11 07:35:22 +02:00
Megumi AIKAWA
9aede5e821
Fixes 11924 : Add pgspider support (#12159)
* Add PGSpider service connector

* add sqlalchemy-pgspider to setup.py

* Delete unnecessary code and move files.

* Change to work as a Postgres connector sub-module

* Rebuild PGSpiderLineage class into a set of funcs

---------

Co-authored-by: Pham Ngoc Son <son1.phamngoc@toshiba.co.jp>
2023-07-05 09:18:59 +02:00
Pere Miquel Brull
b59ff24531
Fix looker view parser (#12254) 2023-07-03 07:25:08 +02:00
Teddy
93d132de5c
fix: removed SampleDataConnection class from package (#12229) 2023-07-01 11:21:21 -07:00
Pere Miquel Brull
1ecf5607c7
Looker - Fix file extension and blob import (#12232)
* Fix file extension and blob import

* Fix file extension and blob import
2023-06-29 16:14:17 +02:00
Mayur Singal
b44d4f1e5e
Fix SQLLineage Test (#12152) 2023-06-26 17:09:49 +05:30
Mayur Singal
a3fd6e9522
Fix #11659: Add support for filter patterns in dbt workflow (#12063) 2023-06-26 11:30:35 +05:30
Mayur Singal
7d1b123efe
Fix atlas lineage custom db name issue (#12108)
* Fix atlas lineage custom db name issue

* Fix test
2023-06-25 19:16:00 -07:00
Pere Miquel Brull
97e08ee25c
Fix #12106 - Fix looker chart sourceUrl & Array datatype handling (#12113)
* Fix looker chart url

* Handle array datatype

* Handle array datatype
2023-06-25 08:18:36 -07:00
Onkar Ravgan
d9d3f6895b
Fix 6874: Added Support for lineage from dbt ephemeral nodes (#12101)
* fixed dbt ephemeral nodes lin

* fixed dbt tests
2023-06-23 10:01:22 +02:00
Ayush Shah
cb6e42941a
Fix 12025: Clickhouse NaN issue (#12079) 2023-06-22 12:51:56 +05:30
Onkar Ravgan
5197682921
Fixed dagster bugs and Added Pydantic Models (#12048) 2023-06-22 10:59:09 +05:30
Teddy
1e86b6533c
Fixes #11743 - Remove SQLParse dependency for System Metrics (#12072)
* fix: removed sqlparse dependency for system metrics

* fix: update sample query

* fix: move system test os retrieval to `.get()`

* fix: move os.environ to `get`
2023-06-22 06:51:24 +02:00
Mayur Singal
c3cec54be9
Fix for #11807 Part 2: Add SourceUrl for table entity (#12013)
* Fix #11807 Part 2: Add SourceUrl for table entity

* address review comments: centralize sourceurl

* remove qlick

* pytest fix

* fix typo
2023-06-20 11:46:45 +02:00
Ayush Shah
83e9b6c310
Fixes 10395: Validation of yaml workflow configs (#11985) 2023-06-20 11:20:59 +05:30
Onkar Ravgan
f07c421264
Removed Empty Description Assignment to entities and added database name logic to tableau (#12031) 2023-06-19 19:19:42 +05:30
Teddy
76f5d3d571
Fixes #11994 - Update dbt and GE integration with new DQ flow (#12018)
* feat: updated GE integration to match new test workflow

* feat: updated unit tests to match new signature

* feat: added GE integration tests

* feat: ran python linting

* feat: updated dbt ingestion to match new TestSuite workflow

* feat: ran python linting

* feat: remove testSuite from Elasticsearch event test case update

* feat: ran java linting
2023-06-19 15:05:51 +02:00
Ayush Shah
f80eaf3a26
Fixes 11068: mysql & postgres iam auth (#11937) 2023-06-16 13:18:12 +05:30
Onkar Ravgan
d08c928801
Added project property to dashboards (#11986)
* Added projects to dashboards

* Added powerbi proj

* merge conflicts after source url

* fixed mongo pytest
2023-06-15 21:23:43 +05:30
Mayur Singal
82a0222257
SourceUrl changes for dashboard, pipeline & chart entities (#11991) 2023-06-15 14:44:48 +05:30
Mayur Singal
7fa963eec3
Fix #1076: Add mongodb support (#11943) 2023-06-15 11:14:22 +05:30
07Himank
62af9bb633
fixed issue for lineage description (#11500)
* fixed issue for lineage description

* fixed issue while ingesting

* fixed issue while ingesting

* added test case for Lingeage with description

* addressing comments .. enhancement

* addressing comments .. enhancement

* modified py test case and removed description from addLineage as we are not using it.

* add support for topic entity and description in lineage details

* fix pylint & test

* pytest fix

* fix column lineage null issue

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-06-12 11:17:32 +05:30
Mayur Singal
05dc42bdb8
Fix #11808: Handle lineage for single db sources in superset (#11933) 2023-06-09 12:43:06 +05:30
Onkar Ravgan
caabe89f9c
Centralize tags ingestion logic (#11880) 2023-06-09 10:45:53 +05:30
Teddy
4b9f213dbf
Fixes Issue #11863 - Add Status to DQ (#11893)
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.

* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity

* feat: add status object to test case results

* feat: ran python linting

* feat: fixed  update to
2023-06-06 10:09:16 +00:00
Teddy
721869428e
Revert "Fixe Issue #11863 - Add Status logic for test case results (#11881)" (#11892)
This reverts commit 06735fe8dbaac5b267c9a2cf744ca154f88a9247.
2023-06-06 09:56:12 +02:00
Teddy
06735fe8db
Fixe Issue #11863 - Add Status logic for test case results (#11881)
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.

* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity

* feat: add status object to test case results

* feat: ran python linting
2023-06-06 09:45:49 +02:00
Ayush Shah
65f370e4aa
Rename GCS to GCP (#11812) 2023-06-06 11:57:00 +05:30
Teddy
d0cffdcd66
Fixes Issue #11438 - Implement threshold and startegy for custom SQL (#11847)
* feat: Add threshold and strategy logic on the custom SQL object test

* feat: ran python linting

* feat: added safety checks for custom sql query

* feat: ran python linting
2023-06-02 09:41:31 +02:00
Teddy
c98a15ca19
Fixes #11705 - Update ingestion and backend to match new DQ flow (#11836)
* feat: refactor ingestion flow logic

* feat: ran python linting

* feat: update tests to match new workflow

* feat: ran python linting

* feat: update sample data test suite name

* feat: Added backend logic to support logical and executable test suites

* feat: clean up java and json code

* feat: added sample data for logical and executable test suites

* feat: remove executable from CreateTestSuite

* feat: ran python and java linting

* feat: added README info for data quality structure

* skipping cypress to keep main green

* fixed typescript type issue

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-06-01 23:19:13 -07:00
Pere Miquel Brull
fdeea71671
Fix Looker explore git link & Add BitBucket reader (#11837)
* Add looker test connection step

* Add looker test connection step

* Update Credentials

* Fix explore link and add bitbucket reader

* Format

* Fix test

* Fix spline linting

* Fix import
2023-06-02 07:19:32 +02:00
Mayur Singal
b57bbf833f
Fix #11572: Glue Support Partition Columns & Use Pydantic Models (#11776) 2023-05-31 12:03:34 +00:00
Chirag Madlani
7adc291364
fix(ui): circular deps for entityReference.json (#11760)
* fix(ui): circular deps for entityReference.json

* Fix circular Dependency python

* Cap Delta Spark version

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-26 18:02:21 +05:30
Sriharsha Chintalapani
6509a3670a
Fix #11664: Refactor patch_mixin to use jsonpatch lib (#11696)
* Fix #11664: Refactor patch_mixin to use jsonpatch lib

* Migrate to jsonpatch

* Fix nested cols

* Format

* Update patch_description

* Table constraints

* tag

* owner

* column tag

* column desc

* Format

* Format

* Fix log

* Update dbt patch

* Update column fqn

* Fix test

* Fix tests

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-05-23 15:47:11 +02:00
Teddy
8c50d1af52
Fixes #4565 - Fetch Metrics from System tables (#11645)
* feat: fetch metrics from system tables

* feat: add permission doc for fetching metrics from system tables

* feat: fix E2E tests to reflect full table row count after table metric update

* feat: ran linting

* feat: fix doc string engine name + function typing

* feat: ran python linting
2023-05-22 09:04:18 +02:00