Teddy
c7ac28f2c2
Fixes #11357 - Implement profiler custom metric processing ( #14021 )
...
* feat: add backend support for custom metrics
* feat: fix python test
* feat: support custom metrics computation
* feat: updated tests for custom metrics
* feat: added dl support for min max of datetime
* feat: added is safe query check for query sampler
* feat: added support for custom metric computation in dl
* feat: added explicit addProper for pydantic model import fo Extra
* feat: added custom metric to returned obj
* feat: wrapped trino import in __init__
* feat: fix python linting
* feat: fix typing in 3.8
2023-11-17 17:51:39 +01:00
Pere Miquel Brull
6c0e9f5061
Part of #7272 - Centralize Workflows, Status, and Exception Management ( #13029 )
...
* Prep changes
* Prep changes
* prep changes
* Update imports
* Format
* Prep delete
* Prep delete
* Fix sink
* Prep test
* Commit
* passing either
* passing either
* Prep Either
* Metadata source with Either
* Update status
* Merge remote-tracking branch 'upstream/main' into issue-7272
* Format
* Linting
* Linting
* Linting
* Linting
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Comments
2023-08-30 15:49:42 +02:00
Onkar Ravgan
0f8ebf4afe
Fixed Tableau-Bigquery lineage issue ( #12570 )
2023-07-26 16:07:48 +05:30
Mayur Singal
0c6435ae9b
Fix #2957 : Add support for qlik sense enterprise connector ( #12450 )
2023-07-19 21:08:05 +05:30
vanshika18
9174cea1ab
Add support for Superset datamodels ( #12183 )
2023-07-13 14:10:37 +05:30
Onkar Ravgan
f07c421264
Removed Empty Description Assignment to entities and added database name logic to tableau ( #12031 )
2023-06-19 19:19:42 +05:30
Teddy
d0cffdcd66
Fixes Issue #11438 - Implement threshold and startegy for custom SQL ( #11847 )
...
* feat: Add threshold and strategy logic on the custom SQL object test
* feat: ran python linting
* feat: added safety checks for custom sql query
* feat: ran python linting
2023-06-02 09:41:31 +02:00
Teddy
60de33d7cf
Fixes #11384 - Implement mem. optimization for sys. metrics ( #11460 )
...
* fix: optimize system metrics retrieval for memory
* fix: ran python linting
* fix: logic to retrieve unique system metrics operations
* fix: added logic to clean up query before parsing it
* fix: added E2E tests for rds, bq, snflk system metrics
* fix: ran python linting
* fix: fix postgres query + add default byte size to env var
* fix: ran python linting
2023-05-09 12:05:35 +02:00
Onkar Ravgan
7e9c02fe6f
Fixed clean_query method for \n ( #11389 )
...
* Fixed clean query method
* fixed regex and tests
* updated regex
2023-05-03 18:08:54 +05:30
Mayur Singal
9f2b10f6e2
Fix #10858 : Fix dashbaord & pipeline url ( #10957 )
2023-04-10 09:48:49 +00:00
Teddy
fc9b64b52a
Fix histogram labels ( #10797 )
...
* fix: added logic to format bin labels for histogram metric
* fix: histogram labels
* fix: added types for arguments
2023-03-28 12:59:45 +02:00
Nahuel
dd225d1463
Fix: Clean Source and SourceStatus inheritance ( #10687 )
...
* Fix: clean source and sourcestatus inheritance
* Remove auth parameters from mysql.yaml
* Fix failing test
* Fix pylint checks
* Add missing docstrings
2023-03-21 12:57:48 +01:00
Teddy
b44972ef60
Fixes #8470 - Implements refinement functions for web analytics events ( #8528 )
...
* Moved webanalytics type in its own folder
* Added data insight chart api endpoint
* Jave formatting
* Added resource descriptor
* Added metadata entity endpoint
* Added aggregation endpoint for dataInsight
* Fix tag name
* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight
* added domo to test subpackage
* cleaned up branch by removing commit from issue-8353 that were not merged in main
* Added web analytics data refinement
* Added get_status function
* Added from __futur__ for typing
* Fixed typos brought up during reviews
2022-11-07 17:08:20 +01:00
Ayush Shah
2d7d89754c
Remove Click and Add ArgParse ( #8182 )
2022-10-31 18:12:26 +05:30
Teddy
83b048c07c
Fixes #8206 -- Implement data insight refinement for entity report data ( #8351 )
...
* added data insight module to refine and ingest data
* Added json schema for data insight workflow
* Added mixin for data insight
* Added tests for data insight workflow and methods
* Fixed python style
* Fixed python linting
* Fixed unit test
* Added data insight worflow + workflow logging
* Fix python style
* Renamed analytic -> data_insight + added test to make + fixed style
* Fixed java style
* Fixed code smells + python style
* Ignore ES linting for now
* Fixed comments from review
* fix python formatting
* Fixe PipelineType attribute for data insight
2022-10-26 11:18:08 +02:00
Nahuel
36b12bd6f1
Fix lineage issues with merge_into and copy grants queries ( #8335 )
2022-10-24 17:22:22 +02:00
Pere Miquel Brull
d48fd468d4
Tag PATCH + Cleanup of helpers methods ( #8150 )
...
* cleanup
* lint
* Add tag patch
* Fix rename
* Dont kill tests
2022-10-15 14:56:30 +02:00
Teddy
f883863b8a
Fixes #7490 - Split Profiler and TestSuite Interface ( #8032 )
...
* Clean up test suite workflow and interface
* Fixed tests
* Split profiler and testSuite interfaces
* Cleaned up workflows and runners
* Fixed code formatting
* - remove old code
- remove `table` attribute used for testing and used mock instead
* Fixed execution bugs from refactor
* Fixed static type checking for profiler/api/workflow.py
* Fixed linting
* Added __init__ files
2022-10-11 15:57:25 +02:00
Onkar Ravgan
107eeef8c7
Added fixes according to pylint ( #8009 )
...
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-10-10 12:53:47 +02:00
Pere Miquel Brull
783557e6da
Fix Pylint for ingestion/sink ( #7961 )
...
* pylint on sink
* Add missing registration
2022-10-07 07:02:27 +02:00
Teddy
e244a241c3
added logic to clean up quotes from table constraints ( #7959 )
2022-10-05 16:09:33 +02:00
Mayur Singal
66a934c837
Fix #4694 : Refactor Source Status ( #6480 )
...
* Fix #4694 : Refactor Source Status
* Using Calculate Time Decorator
* Fixed Decorator
* Fixed Decorator
* perf_counter
* Move Calculate Sucesss to Soruce
* make py_format
* Fixed Sample Data
* Fixed Pipeline, Messaging, Dashboard & MlModel Service
* Fix ZeroDivisionError
2022-08-17 14:39:50 +05:30
Pere Miquel Brull
6701ae6416
Fix #6489 - Update backup CLI ( #6488 )
...
Fix #6489 - Update backup CLI (#6488 )
2022-08-03 12:01:57 +02:00
Nahuel
a878aa911c
Fix#6212: Retrieve connection params from secret manager in CLI commands ( #6441 )
...
* Retrieve connection params from secret manager for database connectors
* Retrieve connection params from secret manager for all services except database connectors
* Stop retrieving connection from SM in Airflow rest plugin
* Retrieve connection params from secret manager for dashboard services
* Retrieve connection params when initializing Workflow/ProfilerWorkflow objects
* Align services topologies + comment changes in topology runner
* Address SonarCloud bug detected
* Update database service topology
* Address PR comments
* Address PR comments
* Address PR comments
2022-08-02 09:13:46 +02:00
Pere Miquel Brull
53930a9253
Fix #6283 #6281 #6279 - Send compressed log, backend logs and Airflow fix ( #6288 )
...
Fix #6283 #6281 #6279 - Send compressed log, backend logs and Airflow fix (#6288 )
2022-07-27 07:47:25 +02:00
Pere Miquel Brull
9fa5d5cab2
Fix #6154 - Python Client to PATCH descriptions ( #6165 )
...
* Prepare patch description utilities for python client
* Improvements
2022-07-19 12:58:58 +02:00
Pere Miquel Brull
0ecc9f0da6
Fix #5459 - Remove sql-metadata in favor of sqllineage ( #5494 )
...
Fix #5459 - Remove sql-metadata in favor of sqllineage (#5494 )
2022-06-21 18:02:50 +02:00
Mayur Singal
b15ce6f52b
Fix #5338 : Dashboard Connector's to return create entities ( #5341 )
...
Fix #5338 : Dashboard Connector's to return create entities (#5341 )
2022-06-08 11:03:48 +02:00
Pere Miquel Brull
9138c70dee
Fix #4696 - Only store URL suffixes when extracting metadata ( #5281 )
...
* Only store URL suffix
* Fix test service name
2022-06-03 11:43:40 +02:00
Mayur Singal
b3428771a3
Fix #4767 : Refactor Usage & Lineage ( #5190 )
...
Fix #4767 : Refactor Usage & Lineage (#5190 )
2022-06-03 10:12:28 +02:00
Pere Miquel Brull
02c2dd9494
Fix #4689 - Rename FQDN to FQN and clean unused code ( #5165 )
2022-05-26 21:00:18 +02:00
Pere Miquel Brull
8e121fb306
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector ( #5125 )
...
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector (#5125 )
2022-05-25 08:35:16 +02:00
Pere Miquel Brull
0c51ecde63
Fix #2830 - Centralise loggers and update format ( #4570 )
...
Fix #2830 - Centralise loggers and update format (#4570 )
2022-04-29 06:54:30 +02:00
Onkar Ravgan
148d7f47ec
Refactor DBT for schema changes ( #4497 )
...
Refactor DBT for schema changes (#4497 )
2022-04-26 14:10:47 +02:00
Onkar Ravgan
c4aa07858d
ISSUE-2925: Added support to get dbt files via s3, gcs and http sources ( #3736 )
2022-04-19 19:36:24 +05:30
Pere Miquel Brull
2aecebd595
Fix #4189 - Fix airflow lineage callback ( #4191 )
...
Fix #4189 - Fix airflow lineage callback (#4191 )
2022-04-18 15:13:26 +02:00
Pere Miquel Brull
43be01b92e
Fix #4091 & #3948 - Create services correctly ( #4101 )
2022-04-13 18:04:52 +02:00
Sriharsha Chintalapani
4a572713a7
Fix #3865 : Service Connection & Schema Name - Fix ES indexes
2022-04-10 09:47:36 -07:00
Mayur Singal
42dcb383c0
Lineage Revamp for Schema changes ( #3995 )
...
* Fix #3592 : Refractored Lineage Ingestion
2022-04-10 08:44:30 -07:00
Ayush Shah
3840eb77c0
Refactoring Connectors - Metabase, MariaDB, Looker ( #3977 )
...
* Metabase Fix
* Looker, Metabase and Mariadb
* modified metabase
2022-04-09 16:56:12 +05:30
Pere Miquel Brull
bd4071bd64
Fix #3826 & #3886 - Profiler workflow & filter pattern ( #3893 )
...
Fix #3826 & #3886 - Profiler workflow & filter pattern (#3893 )
2022-04-06 17:05:00 +02:00
Mayur Singal
7292695bd3
Sample Data Fix ( #3888 )
...
* Sample Data Fix
2022-04-06 18:26:54 +05:30
Pere Miquel Brull
63533eb388
Fix for connectors based on refactoring of schemas V2 ( #3870 )
...
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-04-05 18:33:25 -07:00
Pere Miquel Brull
b3480693e4
Fix #3824 - OMeta Schema and JSON Connections ( #3861 )
...
Fix #3824 - OMeta Schema and JSON Connections (#3861 )
2022-04-05 21:20:39 +02:00
Pere Miquel Brull
e2539c5e83
Fix #3844 - First iteration for deprecating MetadataServerConfig ( #3853 )
...
* Style
* deprecate MetadataServerConfig
* Remove audience from Okta
2022-04-05 18:02:45 +05:30
Sriharsha Chintalapani
7b3e459eb3
Fix #3659 Refactor Service Connection String to be specific to per service ( #3804 )
...
* Fix #3659 Refactor Service Connection String to be specific to per service
* Simplify and centralize Airflow Pipeline info for REST (#3740 )
* Remove code
* Modified Configs based on refactoring schema (#3816 )
* Clean WorkflowContext
Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-04-04 12:46:09 -07:00
Mayur Singal
a38b08882a
Fix #3566 : ingesting sample data with space in name ( #3586 )
2022-03-22 18:29:03 +05:30
codingwithabhi
55c65b5c53
issue-3420: debug-logger-added ( #3434 )
2022-03-15 19:13:31 +05:30
codingwithabhi
2ef2edbf17
Bigquery lineage mapping ( #2849 )
...
* lineage-boilerplate-code-added
* bigquery-lineage-completed
* lineage-code-optimised
* code-cleaned
* code-smell-fixed
* code-refined
* added-database-name-in-fqdn
* code-smell-removed
* exception-handled
* removed-print-statement
* formatted-setup.py
* added-sqllineage-version
* included-valid-type-none
* debug-added-in-logger
* lineage-logic-updated
* updated-lineage-params
* global-config-variables-removed
* return-type-updated-in-entity
* updated-conditionin-from-entity
* imported-sqllineage-inside-method
* code-smell-removed
* code-formatted
2022-02-24 11:59:00 +05:30
Mayur Singal
4739aaa096
Fix #2905 : Added MSSQL Usage Connector ( #2948 )
...
* Fix #2509 : added mssql usage connector
* removed comment
* removed unused variables
* fixed code smell
* renamed sql variable
2022-02-24 01:01:03 +05:30
