1489 Commits

Author SHA1 Message Date
Francisco J. Jurado Moreno
512b241069
[ISSUE-5522] Fix profiler timeout (#6448)
* ChangeIsFunction by IsMethod

* Update test

* Remove whitespace
2022-07-30 10:03:10 -07:00
Teddy
a920a4c17d
Fixed SQA Warning (#6442) 2022-07-30 09:39:36 -07:00
Mayur Singal
82de3a4cc6
GCS Datalake Object Filtering Fix (#6447) 2022-07-30 08:48:06 +05:30
Sriharsha Chintalapani
27558457a7
Add ES indexing option to sort alphabetically (#6444) 2022-07-29 14:11:22 -07:00
Teddy
6397b6a0b1
Fixes #6325 -- Implement multithreading for metrics computation (#6406)
* Added tests for multithreading SQA interface

* Added multithread support for metric computation

* Added thread ID to log debuger

* Cleaned up tests

* Fixed python formatting issues

* Added non blocking result processing + threadCount in config file to set numbers of threads

* Added frontend input field to set number of threads

* Fixed code smell, bug and comments from reviewer
2022-07-29 10:41:53 +02:00
Mayur Singal
bafca3b7b6
Fix #6387: Clean Ingestion (#6405)
* Fix #6387: Clean Ingestion

* postgres fix

* Fixed Location in Sample Data
2022-07-29 08:51:58 +02:00
Pere Miquel Brull
ea918495ff
Fix #6382 - Snyk workflows (#6413)
Fix #6382 - Snyk workflows (#6413)
2022-07-28 23:06:33 +02:00
Onkar Ravgan
7fdab21a72
Glue Topology Refactor (#6231)
* rebase with main

* refactored glue for topology

* added glue unit tests

* resolved review comments

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-07-28 15:04:43 +02:00
Pere Miquel Brull
b7eb41a560
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
2022-07-28 14:46:25 +02:00
Mayur Singal
9c7d8f7d8f
Fix #5847: Handled Exception For Source Config (#6386)
* Fix #5847: Handled Exception For Source Config

* Comment Fix

* Added Test Cases
2022-07-27 20:21:16 +02:00
Mayur Singal
c6efe8c142
Fix #6158: Fivetran Connector Added (#6329)
* Fivetran Connector Added

* Fivetran unit test added

* Fixed Unit Tests

* Added Fivetran Docs & Added Fivetran in Pipeline Listing
2022-07-27 20:50:56 +05:30
Pere Miquel Brull
b5daf1737b
Add braces as literals (#6377) 2022-07-27 16:04:32 +02:00
Mayur Singal
9adaec1b3a
Fix #6248: Use PUT API instead of POST for tag & tagCategory (#6344)
* Fix #6248: Use PUT API instead of POST for tag & tagCategory

* Restructure yield_tag topology

* Renamed method to create_or_update
2022-07-27 08:41:55 +02:00
Pere Miquel Brull
53930a9253
Fix #6283 #6281 #6279 - Send compressed log, backend logs and Airflow fix (#6288)
Fix #6283 #6281 #6279 - Send compressed log, backend logs and Airflow fix (#6288)
2022-07-27 07:47:25 +02:00
Suresh Srinivas
a23a9c84c1
Fixes #6336 - Add API to list of operations per resource for policy authoring (#6337)
* Fixes #6336 - Add API to list of operations per resource to be used for writing policies

* Fixing UI build errors

* Include only JSON schema file and not JSON data files in python code generation

* Fix generate

* Update operations list

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-07-26 12:08:01 -07:00
Nahuel
fdfdaa14aa
Fix#5921: Implentation for retrieving auth provider config from Secret Manager (#6330)
* Implentation for retrieving auth provider config from Secret Manager

* Address PR comments

* Address code smells from SonarCloud

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-07-26 15:42:40 +02:00
Abhishek Pandey
bef4d412ce
uniontype-coverage-added (#6348) 2022-07-26 15:21:34 +02:00
Teddy
aae4410c93
Fies #6183 - Ability to set profile sample at the profilier workflow level (#6292)
Fies  #6183 - Ability to set profile sample at the profilier workflow level (#6292)
2022-07-25 12:08:20 +02:00
Abhishek Pandey
f125c597a9
Atlas connector fix (#4112)
* refactored-atlas-schema

* atlas-source-connector-fixed

* code-formatted

* updated-service-creation

* code-formatted

* atlas-json-updated

* atlas-removed-from-database-service

* atlas-fqn-and-execution-flow-updated

* atlas-connector-fixed

* atlas--amundsen-removed-from-db-service

* code-cleanup

* removed-unused-codes

* code-smelled

* code-smell-removed
2022-07-25 11:23:36 +05:30
Francisco J. Jurado Moreno
ae491f747f
[TASK-6295] Reduce memory footprint for S3 ingestion (#6308)
* Reduce memory footprint for parquet & CSV formats

* Add pagination, remove local var

* Add jsonl parser
2022-07-25 07:24:57 +02:00
Milan Bariya
5e5e728ea9
Create Messaging Topology (#6201)
Create Messaging Topology (#6201)
2022-07-25 07:17:15 +02:00
Pere Miquel Brull
7560635114
Fix #6280 - Bump DeltaLake version, tests and docs (#6307)
Fix #6280 - Bump DeltaLake version, tests and docs (#6307)
2022-07-24 18:49:15 +02:00
Pere Miquel Brull
e51aadae03
Fix curl vulnerability (#6311) 2022-07-24 08:24:35 -07:00
Pere Miquel Brull
810e6c7dfc
Fix #5822 - Remove looker env and rename props (#5700)
Fix #5822 - Remove looker env and rename props (#5700)
2022-07-24 16:02:26 +02:00
Pedro Sereno
14b9439b98
feature: add metabase sqllineage extraction (#6257)
* feature: add metabase sqllineage extraction

this solves issue-6190 and also fixes a bug in the existing lineage
that was trying to access key with a typo for the database name

* fix: sqllineage import
2022-07-23 13:44:31 +02:00
Francisco J. Jurado Moreno
78dd44ed32
[TASK-6140] Add OMeta header to SQLAlchemy-based queries (#6293)
[TASK-6140] Add OMeta header to SQLAlchemy-based queries (#6293)
2022-07-23 12:31:26 +02:00
Francisco J. Jurado Moreno
756dae5605
[TASK-6241] Cache ES ingestion when ingesting usage (#6276)
* Cache ingestion

* Uncomment secrets manager

* Fix experiment stuff

* Fix style

* Add LRU cache

* Add tests

* Fix code smell
2022-07-22 23:33:22 +02:00
Nahuel
4d4a2fc2cf
FIX#5916: Only admin user retrieves connection params when external Secrets Manager is configured (#6228)
* Implementation for stop sending connection credentials when user is BOT

* Change way we add the connection to the service in the Secret Manager

* Services connection is not required as we want to stop returning it when SM is configured
2022-07-22 14:02:29 -07:00
Teddy
28623c7455
Fixes #6078 - DB column named metadata conflict with SQA reserved attribute (#6282) 2022-07-22 19:05:32 +02:00
Mayur Singal
75ccc803ad
Fix #5410: Topology for Pipeline Services (#6085)
Fix #5410: Topology for Pipeline Services (#6085)
2022-07-22 13:59:14 +02:00
Teddy
d34d347a1c
overwrite default sum function to add super for BQ overflow (#6252) 2022-07-22 07:49:00 +02:00
Teddy
893665f33d
Fixes #6031 & #6236 -- Handle binary field & UUID when ingesting sample data (#6233)
Fixes #6031 & #6236 -- Handle binary field & UUID when ingesting sample data (#6233)
2022-07-21 20:14:44 +02:00
Pere Miquel Brull
385b7edac9
Prepare CI for Airflow APIs (#6240)
Prepare CI for Airflow APIs (#6240)
2022-07-21 12:29:57 +02:00
Teddy
e1fac99353
Fixes #5723 and implement interface processor logic (#6219)
* Added datetime for min/max

* Added profiler interface

* Update core.py to work with profiler_interface

* Implement interface logic for orm_profiler object

* Fix unique_ratio logic

* removed changes to table.json

* Added Protocol for type hint

* Changed protocol to abc + fixed sonar code smell

* Fixed py_format
2022-07-20 17:54:10 +02:00
Pere Miquel Brull
2936c0e4e3
Remove retry in ES query (#6191)
Remove retry in ES query (#6191)
2022-07-20 12:44:50 +02:00
Onkar Ravgan
1673b1aa30
DynamoDB Topology refactor (#5967)
DynamoDB Topology refactor (#5967)
2022-07-20 10:42:40 +02:00
Onkar Ravgan
1e1f5d4e8e
Refactored Salesforce for topology (#6197)
* Refactored salesforce for topology

* formatting fixes

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-07-20 08:28:21 +02:00
Francisco J. Jurado Moreno
0cfe6e6cb7
[ISSUE-6162] Escape wildcards on revised RS query (#6200)
[ISSUE-6162] Escape wildcards on revised RS query (#6200)
2022-07-20 06:34:26 +02:00
Suresh Srinivas
4c62237bcb
Fixes #6133 - Add support for Team Hiearchy (#6134)
* Adding different team types for supporting organization team hierarchy

* Add support for team hierarchy

* Create organization during initalization

* Team hierachy test for POST methods

* Add PUT and PATCH support for changing team hiearchy

* Added tests for the team hiearchy feature

* Restored v003 sql files and added v004

* Merge conflicts with main and adopt new JSON based configuration

* Remove configuration Organization name to simplify user MUST do configurations

* Snowflake Make Query tag Optional (#6114)

* Snowflake make query tag optional

* Modified Description

* [Backend][UsageResource] Add PUT to enable updating Usage #6117 :::::: and allow adding space to tagCAtegory and Tags (#6119)

* [Backend][UsageResource] Add PUT to enable updating Usage #6117

* [Backend][UsageResource] Allow space in tags category and primary tags #6121

* [Backend][UsageResource] Update in more genric way to make use of space in resourcePath

* ui changes for allowing space in tag and tag category name

* [Backend][Improvement]add test for put and post

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>

* Adding different team types for supporting organization team hierarchy

* Add support for team hierarchy

* Create organization during initalization

* Team hierachy test for POST methods

* Add PUT and PATCH support for changing team hiearchy

* Added tests for the team hiearchy feature

* Restored v003 sql files and added v004

* Merge conflicts with main and adopt new JSON based configuration

* Remove configuration Organization name to simplify user MUST do configurations

* Python jsonSchema code generation not respecting  defaults

* fixed failing cypress

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: mohitdeuex <105265192+mohitdeuex@users.noreply.github.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2022-07-19 11:10:52 -07:00
Mayur Singal
9308fe6df3
Fixed Marked Deleted Tables (#6199) 2022-07-19 18:14:36 +02:00
Francisco J. Jurado Moreno
710591f106
[ISSUE-6162] Upgrade RS SQL statement query (#6188)
* Upgrade RS SQL statement query

* Fix query, remove dupes
2022-07-19 17:46:12 +02:00
Nahuel
0602e8c894
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
2022-07-19 14:51:44 +02:00
Shailesh Parmar
04db051197
Revert "Pipeline lineage edge UI (#5891)" (#6159)
This reverts commit 4d898a120fd4410a8ed660db252bc270710e9323.
2022-07-19 18:07:46 +05:30
Pere Miquel Brull
9fa5d5cab2
Fix #6154 - Python Client to PATCH descriptions (#6165)
* Prepare patch description utilities for python client

* Improvements
2022-07-19 12:58:58 +02:00
Pere Miquel Brull
627883f24b
Wrap sample data ingestion (#6176) 2022-07-19 08:11:48 +02:00
Pere Miquel Brull
9ad34f51a7
Fix #5452 - Send usage with PUT (#6174)
* Send usage with PUT

* Wrap get details

* Wrap get details
2022-07-19 07:42:58 +02:00
Onkar Ravgan
8c9dc91ccf
Refactored Datalake and Deltalake for Topology (#6034)
* rebasing with main

* refactored deltalake for topology

* using requests instead of urllib

* formatting fixes

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-07-19 06:37:27 +02:00
Mayur Singal
da559150c1
Snowflake Make Query tag Optional (#6114)
* Snowflake make query tag optional

* Modified Description
2022-07-18 20:29:27 +02:00
Pere Miquel Brull
7b6332edef
Fix #6045 - Parse gracefully the connection on Test Connection (#6149)
* Parse gracefully test connection

* Test file renames
2022-07-18 18:50:27 +02:00
Onkar Ravgan
78e41a66a6
Snowflake column name empty issue (#6120)
* added whitespace if column name empty

* Added comments

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-07-18 10:21:56 +02:00