Mayur Singal
029786d773
Fix #9207 : Add source in lineage details ( #13046 )
2023-09-04 11:05:56 +05:30
Pere Miquel Brull
6c0e9f5061
Part of #7272 - Centralize Workflows, Status, and Exception Management ( #13029 )
...
* Prep changes
* Prep changes
* prep changes
* Update imports
* Format
* Prep delete
* Prep delete
* Fix sink
* Prep test
* Commit
* passing either
* passing either
* Prep Either
* Metadata source with Either
* Update status
* Merge remote-tracking branch 'upstream/main' into issue-7272
* Format
* Linting
* Linting
* Linting
* Linting
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Comments
2023-08-30 15:49:42 +02:00
vanshika18
3cc15e6d0b
Fixed #8418 added new connector couchbase ( #12836 )
2023-08-29 11:16:32 +05:30
Pere Miquel Brull
2f3a50e95f
Fix #8885 - Misleading 404 logs & suffix cleanup ( #12916 )
...
* Fix #8885 - Misleading 404 debug logs
* Clean get_suffix complexity
* Prepare airflow config
* Add bootstrap comment
* Revert "Add bootstrap comment"
This reverts commit cc053513df6e25bcc5e742786adb91339461fdd2.
* Revert "Prepare airflow config"
This reverts commit f57645fe953e17d6153652daa1f42c14b97c942d.
* add removed method
* Add icons for docs
* Add missing routes
2023-08-21 11:35:36 +02:00
Mayur Singal
2f7a2193e0
Fix #12302 : Add ElasticSearch Ingestion Source ( #12892 )
2023-08-18 15:10:31 +05:30
Onkar Ravgan
5b47fd4acf
Added source url to entities ( #12901 )
...
* Added source url to entites
* added support to create and update sourceUrl
* fixed pytests
---------
Co-authored-by: 07Himank <himank07mehta@gmail.com>
2023-08-18 10:17:38 +02:00
Onkar Ravgan
3114ef9b5c
Add support for JWT auth in Trino ( #12823 )
...
* Added jwt field in trino
* fixed postgres migration
* fixed pytests
2023-08-17 07:22:58 +02:00
Pere Miquel Brull
a183fc67e2
Fix ADLS parquet reads ( #12840 )
...
* Fix ADLS parquet reads
* Generalize service methods
* Fix tests
2023-08-14 19:57:06 -07:00
Teddy
8e4388c35e
Issue 11345 ( #12859 )
...
* feat: added serviceName dimension to entity report
* feat: fix python test
2023-08-14 08:05:14 +02:00
Ayush Shah
5ea4de646c
Fix Postgres Profiler for incompatible versions ( #12795 )
2023-08-11 15:32:43 +05:30
Pere Miquel Brull
d8e288626c
Storage connection naming, ADLS service and namespace packaging ( #12810 )
...
* Update storage connection naming
* Update storage services and remove typing of generic service
* Remove inits for namespace package
* Add test connection
* Revert "Remove inits for namespace package"
This reverts commit 89f1ae9ea8a633593776f51caf2586d2110d4636.
* Add enum
* Remove init to allow namespace packaging in some modules
* Remove init to allow namespace packaging in some modules
* Move examples and fix test
* Format
* Linting
2023-08-10 07:32:58 +02:00
Pere Miquel Brull
e97d4befb1
Fix #12770 - Cleanup DL structure & Readers & Python 3.8 ( #12776 )
2023-08-09 16:07:16 +05:30
Pere Miquel Brull
a82c2f001c
Fix #9704 - Add inline TNS string ( #12711 )
...
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-08-07 11:04:13 +02:00
Onkar Ravgan
bf44c7996a
fixed nested records avro ( #12633 )
...
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-07-31 11:54:29 +05:30
vanshika18
72e4d0070f
Support MSSLQ handle multiple databases with pyodbc connection ( #12526 )
2023-07-26 10:43:24 +05:30
Ayush Shah
ab1ec50c2c
Fixes Mssql Ntext, text and Image ( #12490 )
2023-07-20 13:34:35 +05:30
Mayur Singal
0c6435ae9b
Fix #2957 : Add support for qlik sense enterprise connector ( #12450 )
2023-07-19 21:08:05 +05:30
Ayush Shah
27a0c9e802
Fix Docker Import ( #12455 )
2023-07-17 12:50:11 +05:30
Teddy
42a426226e
Fixes Issue #11803 #12103 - Add BigQuery Struct Support ( #12435 )
...
* ref: implemented interface for profiler components + removed struct logic
* ref: ran python linting
* ref: added UML diagram to readme.md
* ref: empty commit for labeler check
* ref: remove multiple context manager for 3.7 3.8 compatibility
* ref: remove
* fix: mapper logic for BQ struct types
* feat: added BQ support for structs
* feat: clean code smell + handle null self.col.table value
* feat: ran python linting
* feat: updated test for profiler handler + disabled flaky test
* Update ingestion/tests/unit/profiler/pandas/test_sample.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
---------
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-07-14 09:12:46 +02:00
Teddy
54fbe250a1
fix: import error + BQ E2E CLI ( #12420 )
2023-07-13 13:35:37 +02:00
vanshika18
9174cea1ab
Add support for Superset datamodels ( #12183 )
2023-07-13 14:10:37 +05:30
Pere Miquel Brull
a3bff293a4
Fix #12190 - Bring support for Airflow 2.6 & Use Airflow 2.6.3 as the new ingestion base ( #12398 )
...
* Bump Airflow version
* Support Airflow 2.6
* Lint
* Bump airflow version
* Fix airflow 2.6 lineage
* Fix airflow 2.6 lineage
2023-07-13 06:41:59 +02:00
Teddy
b89cf64f14
Clean up profiler ( #12369 )
...
* ref: implemented interface for profiler components + removed struct logic
* ref: ran python linting
* ref: added UML diagram to readme.md
* ref: empty commit for labeler check
* ref: remove multiple context manager for 3.7 3.8 compatibility
* ref: remove
2023-07-12 17:02:32 +02:00
Pere Miquel Brull
6a1b1190a7
Fix #12313 - Airflow lineage parsing and schedules ( #12339 )
...
* Fix docs
* Fix lineage and intervals
* Format
2023-07-11 07:35:22 +02:00
Megumi AIKAWA
9aede5e821
Fixes 11924 : Add pgspider support ( #12159 )
...
* Add PGSpider service connector
* add sqlalchemy-pgspider to setup.py
* Delete unnecessary code and move files.
* Change to work as a Postgres connector sub-module
* Rebuild PGSpiderLineage class into a set of funcs
---------
Co-authored-by: Pham Ngoc Son <son1.phamngoc@toshiba.co.jp>
2023-07-05 09:18:59 +02:00
Pere Miquel Brull
b59ff24531
Fix looker view parser ( #12254 )
2023-07-03 07:25:08 +02:00
Teddy
93d132de5c
fix: removed SampleDataConnection class from package ( #12229 )
2023-07-01 11:21:21 -07:00
Pere Miquel Brull
1ecf5607c7
Looker - Fix file extension and blob import ( #12232 )
...
* Fix file extension and blob import
* Fix file extension and blob import
2023-06-29 16:14:17 +02:00
Mayur Singal
b44d4f1e5e
Fix SQLLineage Test ( #12152 )
2023-06-26 17:09:49 +05:30
Mayur Singal
a3fd6e9522
Fix #11659 : Add support for filter patterns in dbt workflow ( #12063 )
2023-06-26 11:30:35 +05:30
Mayur Singal
7d1b123efe
Fix atlas lineage custom db name issue ( #12108 )
...
* Fix atlas lineage custom db name issue
* Fix test
2023-06-25 19:16:00 -07:00
Pere Miquel Brull
97e08ee25c
Fix #12106 - Fix looker chart sourceUrl & Array datatype handling ( #12113 )
...
* Fix looker chart url
* Handle array datatype
* Handle array datatype
2023-06-25 08:18:36 -07:00
Onkar Ravgan
d9d3f6895b
Fix 6874: Added Support for lineage from dbt ephemeral nodes ( #12101 )
...
* fixed dbt ephemeral nodes lin
* fixed dbt tests
2023-06-23 10:01:22 +02:00
Ayush Shah
cb6e42941a
Fix 12025: Clickhouse NaN issue ( #12079 )
2023-06-22 12:51:56 +05:30
Onkar Ravgan
5197682921
Fixed dagster bugs and Added Pydantic Models ( #12048 )
2023-06-22 10:59:09 +05:30
Teddy
1e86b6533c
Fixes #11743 - Remove SQLParse dependency for System Metrics ( #12072 )
...
* fix: removed sqlparse dependency for system metrics
* fix: update sample query
* fix: move system test os retrieval to `.get()`
* fix: move os.environ to `get`
2023-06-22 06:51:24 +02:00
Mayur Singal
c3cec54be9
Fix for #11807 Part 2: Add SourceUrl for table entity ( #12013 )
...
* Fix #11807 Part 2: Add SourceUrl for table entity
* address review comments: centralize sourceurl
* remove qlick
* pytest fix
* fix typo
2023-06-20 11:46:45 +02:00
Ayush Shah
83e9b6c310
Fixes 10395: Validation of yaml workflow configs ( #11985 )
2023-06-20 11:20:59 +05:30
Onkar Ravgan
f07c421264
Removed Empty Description Assignment to entities and added database name logic to tableau ( #12031 )
2023-06-19 19:19:42 +05:30
Teddy
76f5d3d571
Fixes #11994 - Update dbt and GE integration with new DQ flow ( #12018 )
...
* feat: updated GE integration to match new test workflow
* feat: updated unit tests to match new signature
* feat: added GE integration tests
* feat: ran python linting
* feat: updated dbt ingestion to match new TestSuite workflow
* feat: ran python linting
* feat: remove testSuite from Elasticsearch event test case update
* feat: ran java linting
2023-06-19 15:05:51 +02:00
Ayush Shah
f80eaf3a26
Fixes 11068: mysql & postgres iam auth ( #11937 )
2023-06-16 13:18:12 +05:30
Onkar Ravgan
d08c928801
Added project property to dashboards ( #11986 )
...
* Added projects to dashboards
* Added powerbi proj
* merge conflicts after source url
* fixed mongo pytest
2023-06-15 21:23:43 +05:30
Mayur Singal
82a0222257
SourceUrl changes for dashboard, pipeline & chart entities ( #11991 )
2023-06-15 14:44:48 +05:30
Mayur Singal
7fa963eec3
Fix #1076 : Add mongodb support ( #11943 )
2023-06-15 11:14:22 +05:30
07Himank
62af9bb633
fixed issue for lineage description ( #11500 )
...
* fixed issue for lineage description
* fixed issue while ingesting
* fixed issue while ingesting
* added test case for Lingeage with description
* addressing comments .. enhancement
* addressing comments .. enhancement
* modified py test case and removed description from addLineage as we are not using it.
* add support for topic entity and description in lineage details
* fix pylint & test
* pytest fix
* fix column lineage null issue
---------
Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-06-12 11:17:32 +05:30
Mayur Singal
05dc42bdb8
Fix #11808 : Handle lineage for single db sources in superset ( #11933 )
2023-06-09 12:43:06 +05:30
Onkar Ravgan
caabe89f9c
Centralize tags ingestion logic ( #11880 )
2023-06-09 10:45:53 +05:30
Teddy
4b9f213dbf
Fixes Issue #11863 - Add Status to DQ ( #11893 )
...
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.
* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity
* feat: add status object to test case results
* feat: ran python linting
* feat: fixed update to
2023-06-06 10:09:16 +00:00
Teddy
721869428e
Revert "Fixe Issue #11863 - Add Status logic for test case results ( #11881 )" ( #11892 )
...
This reverts commit 06735fe8dbaac5b267c9a2cf744ca154f88a9247.
2023-06-06 09:56:12 +02:00
Teddy
06735fe8db
Fixe Issue #11863 - Add Status logic for test case results ( #11881 )
...
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.
* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity
* feat: add status object to test case results
* feat: ran python linting
2023-06-06 09:45:49 +02:00
