Mayur Singal
9ef32eef99
Fix #6829 : Improve usage filtering ( #9613 )
2023-01-05 15:07:03 +00:00
Steve Russo
3a3f248edd
Issue 5592: Add Databricks Pipeline Connector ( #9554 )
...
* feat: update databricks_client
* feat: create pipelineconnection & connection
* test: add unit test
* feat: add examples file
* fix: remove unused imports
* style: fix python style
* fix: code review adjustments
* fix: remove unneeded schema ref
* fix: format requirements
* fix: change context calls
* test: add pipeline_status test
* format: pylint format change
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-01-05 14:14:12 +01:00
Pere Miquel Brull
1f49ea46b5
Fix #9149 - Postgres foreign tables & reorganize queries for sources ( #9588 )
...
Fix #9149 - Postgres foreign tables & reorganize queries for sources (#9588 )
2023-01-04 08:50:53 +01:00
8vicat
7e7528e13b
Improve SQL lineage parsing ( #9537 )
...
* CTEs support
* Supports inline comments
* Expands `*` in sources
2023-01-03 09:27:45 +01:00
NiharDoshi99
8f69386076
Fix: refactor get_column_type ( #9559 )
...
* Fix: refactor get_column_type
* Fix: added changes as per comments
* Fix: pylint
* Fix: minor changes
* Fix: minor changes
2023-01-03 10:28:38 +05:30
Pere Miquel Brull
7f21a7bced
Fix #8088 - Restructure source connections & clients ( #9545 )
2023-01-02 13:52:27 +01:00
Pere Miquel Brull
a4521fd664
Fix #6562 - Sources have their own package ( #9521 )
...
Fix #6562 - Sources have their own package (#9521 )
2022-12-27 15:00:22 +01:00
Suresh Srinivas
758c976cba
Fixes #9259 Change Tags APIs to conform with rest of the APIs ( #9260 )
2022-12-26 12:32:17 -08:00
Pere Miquel Brull
ec6ebb3694
Organise dynamic importers ( #9513 )
...
Organise dynamic importers (#9513 )
2022-12-23 12:17:06 +01:00
Teddy
9e01fe0636
feat(CLI): added migration logic cmd ( #9437 )
2022-12-21 07:55:18 +00:00
Mayur Singal
07aed10a7e
Snowflake E2E Tests ( #8781 )
2022-12-20 18:00:26 +05:30
Ayush Shah
2bf5eb9051
fix 7995: profileSample % and row number ( #9104 )
2022-12-20 14:55:11 +05:30
Milan Bariya
a3495c67f0
Databricks improve code and doc update ( #9394 )
...
* Databricks improve code and doc update
* Change base on comments
* Fix pytest fail
2022-12-19 18:53:10 +05:30
Pere Miquel Brull
3b7ae73473
Airflow e2e integration test ( #9363 )
...
* Prep airflow operator integration tests
* Add integration test to Makefile
2022-12-16 19:52:12 -08:00
Mayur Singal
5164a55341
Fix Mysql e2e test ( #9364 )
2022-12-16 19:39:02 -08:00
Teddy
3856d63f3a
Fixes #5522 by implementing timeout for the profiler ( #9340 )
...
* feat(Profiler): Added timeout logic in the profiler workflow
* feat(Profiler): Added additional input in workflow UI
* fix(Profiler): timeout text blob in UI
* fix(Profiler): Return profile_results outside the context manager
* Fix(Profiler): Added localizer for timeout label and message
* Update ingestion/src/metadata/interfaces/sqalchemy/sqa_profiler_interface.py
Co-authored-by: Nahuel <nahuel@getcollate.io>
Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-12-16 17:01:12 +01:00
NiharDoshi99
3190ac2516
Fix: making token as optional field ( #9314 )
2022-12-16 14:17:47 +00:00
Nahuel
819001182f
Fix#9251: DB2 connection config and ingestion update ( #9322 )
...
* DB2 connection config and ingestion update
* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py
Co-authored-by: Ayush Shah <ayush@getcollate.io>
* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py
Co-authored-by: Ayush Shah <ayush@getcollate.io>
* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v007__create_db_connection_info.sql
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-12-16 07:43:18 +01:00
Onkar Ravgan
b539b299ee
Integrated schema parsers ( #9305 )
...
* Integrated schema parsers
* Addressed review comments
* fixed pytests
2022-12-15 16:54:55 +05:30
Ayush Shah
a6ae9fd11a
Add Test Suite Implementation for Datalake ( #9235 )
2022-12-14 21:14:51 +05:30
Teddy
d1a739ec55
Fixes #9025 -- Added deletion of WebAnalytics events in dataInsight Workflow ( #9208 )
2022-12-13 11:43:29 +01:00
Nahuel
9a4e3a7a46
Fix#8424: Remove brackets from tables and schemas on lineage ( #9257 )
...
* Refactor LineageRunner use
* Address PR comments
* Address pylint errors
* Fix failing test
* Remove brackets from tables and schemas on lineage
2022-12-13 06:40:37 +01:00
Pere Miquel Brull
c75ba751b7
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification ( #9241 )
...
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241 )
2022-12-13 06:36:55 +01:00
Pere Miquel Brull
9a955036cf
Get sample data for e2e tests ( #9252 )
2022-12-12 15:08:01 +01:00
Pere Miquel Brull
0963eac48e
Fix #9182 - Airflow Lineage Operator & Airflow lineage state ( #9206 )
2022-12-10 19:54:41 +01:00
Ayush Shah
231b28fc87
Fix 7278: add test connection ( #9196 )
...
* Add Test Connection to Profiler
* remove Code Lint
* Fix AttributeError
* Fix Pytest
* Fix Bigquery Partition tests
* Fix Lint
2022-12-10 17:16:28 +01:00
Teddy
ac77f33b08
Fixes #7447 -- Add freshness metrics to profiler ( #9159 )
...
* refactor(profiler): integrated getter func.
Removed metric getter function from their own file.
Added metric getter to their own interface classs.
created dispatch by value methdo to dispatch metric getter func.
* feature(profiler): added systemProfiler schema
* feat(profiler): workflow fresh. & snflk impl.
* feat(profiler): freshness endpoint for put and get
* feat(profiler): added system met. for redshift
* feat(profiler): freshness met. for bigquery
* fix(profiler): keyword not found in func
* feat(profiler): Added sample data for freshness
* fix(profiler): fetch previous day for BQ
* fix(profiler): sonar + data fetching logic
* fix: typo in SystemMetric Class
* fix: linting
* fix: extracted out EntityList class into models.py
2022-12-07 14:33:30 +01:00
Sriharsha Chintalapani
25449001ca
Fix #9040 : Remove fields such as tableQueries, tableProfile, tests, sample data as part of table fields ( #9041 )
2022-12-06 21:07:04 -08:00
Ayush Shah
2efa720f59
Fix E2E: new envs ( #9143 )
2022-12-05 19:15:12 +05:30
Pere Miquel Brull
58f3c01470
Fix #7056 - Automatic lineage from ML Model to ML Feature Data Source ( #9113 )
2022-12-05 16:43:03 +05:30
Pere Miquel Brull
1b3ff505c2
Fix #8858 - Add chart description and add lineage flexibility ( #9124 )
...
Fix #8858 - Add chart description and add lineage flexibility (#9124 )
2022-12-02 16:22:09 +01:00
Nahuel
00375341f7
Fix#7690: Use ES to build Tables FQN when processing Lineage ( #9079 )
...
* Use ES to build Tables FQN when processing Lineage
* Fix pylent check
* Fix test failing
* Remove warning when calling ES with empty query
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-12-01 09:56:42 +01:00
Nahuel
76773e69de
Fix#6203: Refactor LineageRunner use ( #9073 )
...
* Refactor LineageRunner use
* Address PR comments
* Address pylint errors
* Fix failing test
2022-11-30 16:02:21 +01:00
Ayush Shah
2b2eadf611
Fix Credentials sent via env ( #9071 )
...
* Fix Credentials sent via env
* add new env var
* fix lint
* Add Comment
2022-11-30 15:37:24 +01:00
Teddy
3cad959e44
Fixes #6760 -- Implements REGEX for regex test ( #9033 )
...
feat(testCase): impelemented regex logic for test suite
2022-11-29 13:00:28 +01:00
Ayush Shah
293de4a894
E2E for Bigquery ( #9007 )
2022-11-29 16:22:08 +05:30
Pere Miquel Brull
eccbb087d3
Add table query test ( #9000 )
...
* Add table query test
* Add table query test
2022-11-25 06:56:12 +01:00
Teddy
8789bdb847
Ometa mixin for ingestion pipeline ( #8931 )
...
* Added sleep between data processing ingestion and KPI processing
* Added ometa mixin for ingestion pipeline and test suite
2022-11-23 09:46:56 +00:00
Milan Bariya
27997b2f77
Fix: Revert Athena tests and types change ( #8909 )
2022-11-21 12:35:23 +01:00
Nahuel
b14a3bc6ca
Review sensitive data not marked as password in the JSON schemas ( #8818 )
2022-11-18 11:35:08 +01:00
Pere Miquel Brull
b203a92565
Add config file env var loading tests ( #8799 )
...
* Add env var load test
* Add env var load test
2022-11-17 16:52:42 +00:00
Pere Miquel Brull
93381a7343
Add Pylint print checker and py-checkstyle as required ( #8849 )
...
* Always run the checkstyle
* Add print checker lint plugin
* Typos
* Add print checker
* Fix e2e
2022-11-17 16:44:26 +00:00
Teddy
4ab6dbc201
Data insight tests addition ( #8814 )
...
* Added additional tests around web analytics and data insight
* Added playwright dependency installation
* removed -m no-ui option
* Removed playwright tests
* Added event ingestion in setUp class
2022-11-17 10:44:52 +01:00
NiharDoshi99
03d4011a17
Fix: Changes in bigquey for project-id ( #8708 )
2022-11-17 14:26:37 +05:30
Pere Miquel Brull
d1cc3e880c
Fix e2e test and add slack alert ( #8817 )
...
* Fix mysql test
* Run alarm on failed test
* Force failure for testing
2022-11-16 17:43:48 +01:00
Teddy
4ccdce50e0
Fix data insight bugs ( #8761 )
...
* - made ES ingestion of data inisght idempotent for the daily execution
- added web analytic timedelta to 1 day to fecth data from the previous day
* Added time utils
* fixed timezone
2022-11-15 16:10:30 +01:00
Ayush Shah
5be0f8ee76
Dl Profiler ( #8694 )
...
* DQ commit
* Add DL Profiler
* Fix Ingestion and Profliing pylint checks
* Fix Tests
* PyFormat files
* Fix Tests
* Resolve Comments
* Fix Tests and Format Files
* Resolve Comments
* Fix Pylint and Code smells
* Resolve Comments
* Fix S3 parquet
* Fix Metrics Code Smell
2022-11-15 16:01:10 +01:00
Mohit Yadav
db0215f964
Mapping from kpi to dichart(One to one) ( #8707 )
...
* Mapping from kpi to dichart(One to one)
* Mapping from kpi to dichart(One to one)
* Removed second tests creating KPI
* Fix some ui improve as well
* review comment changed function name
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-11-14 19:44:01 +05:30
Pere Miquel Brull
34ba9d95c5
Ingestion Pipeline deployed, Athena tests and pydantic extras ( #8682 )
...
* Always run python tests
* Fix athena tests and types
* Update deployed prop in IngestionPipeline
* Fix #8554
* Format
* Use true as default deployed migration
* Remove repeated req
* Pydantic wiggle room
2022-11-13 11:59:43 +01:00
Onkar Ravgan
eee3f9ffec
Fix:#8553 Parse Avro/Protobuf/Json schemas ( #8654 )
...
* Added topic parsers
* Fixed pylint
* Addressed review comments
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-11-11 16:35:09 +05:30