Pere Miquel Brull
e2916b2521
Organise ingestion module ( #6515 )
...
Organise ingestion module (#6515 )
2022-08-03 09:55:50 +02:00
Nahuel
a878aa911c
Fix#6212: Retrieve connection params from secret manager in CLI commands ( #6441 )
...
* Retrieve connection params from secret manager for database connectors
* Retrieve connection params from secret manager for all services except database connectors
* Stop retrieving connection from SM in Airflow rest plugin
* Retrieve connection params from secret manager for dashboard services
* Retrieve connection params when initializing Workflow/ProfilerWorkflow objects
* Align services topologies + comment changes in topology runner
* Address SonarCloud bug detected
* Update database service topology
* Address PR comments
* Address PR comments
* Address PR comments
2022-08-02 09:13:46 +02:00
Teddy
7c70adc33e
Changed ometa_service_name to database_service_name ( #6454 )
2022-08-01 07:41:46 +02:00
Francisco J. Jurado Moreno
512b241069
[ISSUE-5522] Fix profiler timeout ( #6448 )
...
* ChangeIsFunction by IsMethod
* Update test
* Remove whitespace
2022-07-30 10:03:10 -07:00
Teddy
a920a4c17d
Fixed SQA Warning ( #6442 )
2022-07-30 09:39:36 -07:00
Teddy
6397b6a0b1
Fixes #6325 -- Implement multithreading for metrics computation ( #6406 )
...
* Added tests for multithreading SQA interface
* Added multithread support for metric computation
* Added thread ID to log debuger
* Cleaned up tests
* Fixed python formatting issues
* Added non blocking result processing + threadCount in config file to set numbers of threads
* Added frontend input field to set number of threads
* Fixed code smell, bug and comments from reviewer
2022-07-29 10:41:53 +02:00
Onkar Ravgan
7fdab21a72
Glue Topology Refactor ( #6231 )
...
* rebase with main
* refactored glue for topology
* added glue unit tests
* resolved review comments
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-07-28 15:04:43 +02:00
Mayur Singal
9c7d8f7d8f
Fix #5847 : Handled Exception For Source Config ( #6386 )
...
* Fix #5847 : Handled Exception For Source Config
* Comment Fix
* Added Test Cases
2022-07-27 20:21:16 +02:00
Mayur Singal
c6efe8c142
Fix #6158 : Fivetran Connector Added ( #6329 )
...
* Fivetran Connector Added
* Fivetran unit test added
* Fixed Unit Tests
* Added Fivetran Docs & Added Fivetran in Pipeline Listing
2022-07-27 20:50:56 +05:30
Nahuel
fdfdaa14aa
Fix#5921: Implentation for retrieving auth provider config from Secret Manager ( #6330 )
...
* Implentation for retrieving auth provider config from Secret Manager
* Address PR comments
* Address code smells from SonarCloud
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-07-26 15:42:40 +02:00
Abhishek Pandey
bef4d412ce
uniontype-coverage-added ( #6348 )
2022-07-26 15:21:34 +02:00
Pere Miquel Brull
7560635114
Fix #6280 - Bump DeltaLake version, tests and docs ( #6307 )
...
Fix #6280 - Bump DeltaLake version, tests and docs (#6307 )
2022-07-24 18:49:15 +02:00
Pere Miquel Brull
810e6c7dfc
Fix #5822 - Remove looker env and rename props ( #5700 )
...
Fix #5822 - Remove looker env and rename props (#5700 )
2022-07-24 16:02:26 +02:00
Francisco J. Jurado Moreno
78dd44ed32
[TASK-6140] Add OMeta header to SQLAlchemy-based queries ( #6293 )
...
[TASK-6140] Add OMeta header to SQLAlchemy-based queries (#6293 )
2022-07-23 12:31:26 +02:00
Francisco J. Jurado Moreno
756dae5605
[TASK-6241] Cache ES ingestion when ingesting usage ( #6276 )
...
* Cache ingestion
* Uncomment secrets manager
* Fix experiment stuff
* Fix style
* Add LRU cache
* Add tests
* Fix code smell
2022-07-22 23:33:22 +02:00
Nahuel
4d4a2fc2cf
FIX#5916: Only admin user retrieves connection params when external Secrets Manager is configured ( #6228 )
...
* Implementation for stop sending connection credentials when user is BOT
* Change way we add the connection to the service in the Secret Manager
* Services connection is not required as we want to stop returning it when SM is configured
2022-07-22 14:02:29 -07:00
Teddy
28623c7455
Fixes #6078 - DB column named metadata
conflict with SQA reserved attribute ( #6282 )
2022-07-22 19:05:32 +02:00
Mayur Singal
75ccc803ad
Fix #5410 : Topology for Pipeline Services ( #6085 )
...
Fix #5410 : Topology for Pipeline Services (#6085 )
2022-07-22 13:59:14 +02:00
Teddy
893665f33d
Fixes #6031 & #6236 -- Handle binary field & UUID when ingesting sample data ( #6233 )
...
Fixes #6031 & #6236 -- Handle binary field & UUID when ingesting sample data (#6233 )
2022-07-21 20:14:44 +02:00
Teddy
e1fac99353
Fixes #5723 and implement interface processor logic ( #6219 )
...
* Added datetime for min/max
* Added profiler interface
* Update core.py to work with profiler_interface
* Implement interface logic for orm_profiler object
* Fix unique_ratio logic
* removed changes to table.json
* Added Protocol for type hint
* Changed protocol to abc + fixed sonar code smell
* Fixed py_format
2022-07-20 17:54:10 +02:00
Nahuel
0602e8c894
Fix#5924: Implement secret manager for Airflow REST Plugin ( #6128 )
...
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128 )
2022-07-19 14:51:44 +02:00
Pere Miquel Brull
7b6332edef
Fix #6045 - Parse gracefully the connection on Test Connection ( #6149 )
...
* Parse gracefully test connection
* Test file renames
2022-07-18 18:50:27 +02:00
Pere Miquel Brull
479a8de486
Fix #5885 - Provider loading fails for airflow <2.3 ( #5927 )
...
Fix #5885 - Provider loading fails for airflow <2.3 (#5927 )
2022-07-14 15:07:39 +02:00
Milan Bariya
954635b6dc
FIX 5907: Ingest Trino complex type ( #5931 )
...
* FIX 5907: Ingest Trino complex type
* Included Row & Array Datatype
* Resolved review comments
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2022-07-14 11:28:50 +05:30
nna077
baa5295cc2
Add Date/Time Metrics In Profiler Tab ( #5821 )
...
Add Date/Time Metrics In Profiler Tab (#5821 )
2022-07-13 21:23:03 +02:00
Teddy
d097199d2f
Added validation in profiler workflow to ensure service name exists and raise more explicit error ( #6036 )
...
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-07-13 14:43:48 +02:00
Teddy
f0926d9b75
Fixes #3463 -- Implement Custom SQL Data Quality Tests ( #5973 )
...
Fixes #3463 -- Implement Custom SQL Data Quality Tests (#5973 )
2022-07-12 06:21:11 +02:00
Mayur Singal
ebc9a24ee0
Connection options fix ( #5971 )
...
* Connection Options Fix
* Fixed Condition
2022-07-08 17:44:18 +02:00
Teddy
11487571b0
Fixes #5964 -- Trino Median function and columnValuesToMatchRegex wording ( #5965 )
...
* added logic for trino median function
* Fixed columnValuesToMatchRegex result wording
* Replaced table with column for toMatchRegex test
2022-07-08 15:55:50 +02:00
Teddy
48f6553fb3
Fixes #5866 -- Implement Support for Median Metric ( #5887 )
...
* Added additional table + test coverage
* Added logic for front end input fields
* Added comment for median metric
* skipping `Update owner and check description` cypress test
* Added support to run window metrics for the profiler
* Fix except code smell
* moved median metric to windown folder
* Fix pyformat
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-07-07 06:58:42 +02:00
Teddy
3a7c11424b
Fixes #3133 -- Adding Additional Column Tests ( #5867 )
...
* Added additional table + test coverage
* Added logic for front end input fields
* Added comment for median metric
* skipping `Update owner and check description` cypress test
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-07-06 10:12:29 +02:00
Teddy
e026d625d6
Fixes #5713 where profileSample default to 100 ( #5714 )
...
* Fix random sample modulo and rand for postgres
* Remove sampleProfile default and return declarative meta when no sample profile is passed
* fixed py_format
* isolated random sample partition logic into its own methid
2022-06-28 19:27:55 +02:00
Mayur Singal
71bf32e438
Fix #5562 : Handle Analysis Date for Usage ( #5582 )
...
Fix #5562 : Handle Analysis Date for Usage (#5582 )
2022-06-27 15:16:12 +02:00
Milan Bariya
338d1efe40
FIX 5419: Glue pipeline connector not working ( #5467 )
...
FIX 5419: Glue pipeline connector not working (#5467 )
2022-06-27 11:38:41 +02:00
Teddy
7431c1c226
[Issue-5188] Implement User Query for Sampler and Profiler ( #5578 )
...
* Added custom query sample for sample data ingestion
* Added logic to run table profiling against user's query
* Added tests for user query logic in profiler and sampler
* Added user profiling to tableProfile + fixed format
* staging commit
* Added logic to add profileQuery to table entity
* Added limit to sample rows
2022-06-24 14:46:34 +02:00
Pere Miquel Brull
0144a1d46a
Fix #5423 & #3412 - Looker Lineage and Usage ( #5581 )
...
Fix #5423 & #3412 - Looker Lineage and Usage (#5581 )
2022-06-23 13:01:40 +02:00
Pere Miquel Brull
0ecc9f0da6
Fix #5459 - Remove sql-metadata in favor of sqllineage ( #5494 )
...
Fix #5459 - Remove sql-metadata in favor of sqllineage (#5494 )
2022-06-21 18:02:50 +02:00
Teddy
28336b0a43
Fix #3575 Data Quality - Partitioned Tables ( #5441 )
...
Co-authored-by: Teddy Crepineau <teddycrepineau@Teddys-MacBook-Pro.local>
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
2022-06-14 12:37:44 -07:00
Pere Miquel Brull
02fece727b
Fix #5307 - Define a better topology for Ingestion Sources ( #5366 )
...
Fix #5307 - Define a better topology for Ingestion Sources (#5366 )
2022-06-13 12:17:54 +02:00
Pere Miquel Brull
8e9d0a73f6
Fix #3573 - Sample Data refactor & ORM converter improvements ( #5265 )
...
Fix #3573 - Sample Data refactor & ORM converter improvements (#5265 )
2022-06-08 16:10:40 +02:00
Milan Bariya
ec1aca926a
Fix #5210 : Review database field for connectors ( #5225 )
...
* Fix #5210 : Review database field for connectors
2022-06-06 10:48:23 +05:30
Mayur Singal
b3428771a3
Fix #4767 : Refactor Usage & Lineage ( #5190 )
...
Fix #4767 : Refactor Usage & Lineage (#5190 )
2022-06-03 10:12:28 +02:00
Onkar Ravgan
4671ca00b0
Refactored PowerBi Connector ( #5107 )
...
Refactored PowerBi Connector (#5107 )
2022-06-01 10:36:12 +02:00
Abhishek Pandey
50409eac24
removed-required-from-aws-credential ( #5219 )
...
removed-required-from-aws-credential (#5219 )
2022-05-31 08:45:02 +02:00
Abhishek Pandey
12e8a1fcf6
required-fields-updated-in-snowflake-and-athena ( #5143 )
...
required-fields-updated-in-snowflake-and-athena (#5143 )
2022-05-30 14:57:26 +02:00
Pere Miquel Brull
0005bc1292
Fix #5108 - Add Map and Struct support for Athena & Block them for profiler ( #5181 )
...
Fix #5108 - Add Map and Struct support for Athena & Block them for profiler (#5181 )
2022-05-30 06:53:16 +02:00
Pere Miquel Brull
02c2dd9494
Fix #4689 - Rename FQDN to FQN and clean unused code ( #5165 )
2022-05-26 21:00:18 +02:00
Teddy
bbe5cfa9e7
[ISSUE-3450] openmetadata-ingestion great-expectations submodule ( #4752 )
...
[ISSUE-3450] openmetadata-ingestion great-expectations submodule (#4752 )
2022-05-26 15:16:03 +02:00
Mayur Singal
f9bb54ed91
Fix #5162 : Sample usage fixed ( #5163 )
...
* Fix #5162 : Sample usage fixed
* Test Fix
2022-05-26 15:15:41 +02:00
Mayur Singal
41ee3a5aaf
Fix #3940 : Refractor Sql Source ( #5046 )
...
Fix #3940 : Refractor Sql Source (#5046 )
2022-05-25 15:41:38 +02:00