12 Commits

Author SHA1 Message Date
Teddy
ce578e73d4
Fixes #5831 by implenting testSuite workflow logic (#6911)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells

* commited changed to profiler

* initial commit of the revamp workflow

* Fixed python formating

* cleaned up profiler submodule by removing test related files and functions

* Added airflow DAG logic for testSuite workflow

* Fixed code smells + added airflow ingestion tests + fixed comments
2022-08-25 10:01:28 +02:00
Teddy
78b5f8c8e2
Part 1 of #5831 -- Profiler workflow implementation (#6809)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells
2022-08-19 10:52:08 +02:00
Teddy
abaf8a84e9
Fixes #5661 by removing association between profiler and data quality (#6715)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs
2022-08-17 12:53:16 +02:00
Sriharsha Chintalapani
1a42428e42
Add time series extention (#6416)
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-08-04 07:22:47 -07:00
Teddy
6397b6a0b1
Fixes #6325 -- Implement multithreading for metrics computation (#6406)
* Added tests for multithreading SQA interface

* Added multithread support for metric computation

* Added thread ID to log debuger

* Cleaned up tests

* Fixed python formatting issues

* Added non blocking result processing + threadCount in config file to set numbers of threads

* Added frontend input field to set number of threads

* Fixed code smell, bug and comments from reviewer
2022-07-29 10:41:53 +02:00
Teddy
893665f33d
Fixes #6031 & #6236 -- Handle binary field & UUID when ingesting sample data (#6233)
Fixes #6031 & #6236 -- Handle binary field & UUID when ingesting sample data (#6233)
2022-07-21 20:14:44 +02:00
Teddy
e1fac99353
Fixes #5723 and implement interface processor logic (#6219)
* Added datetime for min/max

* Added profiler interface

* Update core.py to work with profiler_interface

* Implement interface logic for orm_profiler object

* Fix unique_ratio logic

* removed changes to table.json

* Added Protocol for type hint

* Changed protocol to abc + fixed sonar code smell

* Fixed py_format
2022-07-20 17:54:10 +02:00
Teddy
7431c1c226
[Issue-5188] Implement User Query for Sampler and Profiler (#5578)
* Added custom query sample for sample data ingestion

* Added logic to run table profiling against user's query

* Added tests for user query logic in profiler and sampler

* Added user profiling to tableProfile + fixed format

* staging commit

* Added logic to add profileQuery to table entity

* Added limit to sample rows
2022-06-24 14:46:34 +02:00
Pere Miquel Brull
8e9d0a73f6
Fix #3573 - Sample Data refactor & ORM converter improvements (#5265)
Fix #3573 - Sample Data refactor & ORM converter improvements (#5265)
2022-06-08 16:10:40 +02:00
Mayur Singal
db0e34c709
Fixing Test Connection for Dynamo & Glue (#4316)
* Fixing Test Connection for Dynamo

* Fixed Glue Connector

* renamed engine to connection

* Fixed the return signature

* Added dataclass
2022-04-22 11:30:59 +05:30
Pere Miquel Brull
b3087d08b9
Fix #3522 - Add timeout to profiler (#3707)
Fix #3522 - Add timeout to profiler (#3707)
2022-03-30 08:54:27 +02:00
Pere Miquel Brull
16e82d45de
Fix #3371 - Run Profiler and Tests on a % of the data (#3424)
Fix #3371 - Run Profiler and Tests on a % of the data (#3424)
2022-03-16 06:05:59 +01:00