Teddy
f3da919329
Feat: Backend Support for Custom Metrics ( #13965 )
...
* feat: add backend support for custom metrics
* feat: fix python test
2023-11-17 19:16:35 +05:30
Mohit Yadav
10d8ec84fe
Logs added for Search Indexing and Stats issue fixed ( #13956 )
...
* Logs added for Search Indexing and Stats issue fixed
* Fix uninstall error
* Add error handling
* fix lint
* Push Job Level Exception on top
* disable flaky tests
* Fix Logs not visible in Search
---------
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-11-13 23:39:56 +05:30
Pere Miquel Brull
c742835766
Auto Tagger Application - Preparing the Ingestion Framework ( #13862 )
...
* Prepare the skeleton for generic app registration
* Prepare the skeleton for generic app registration
* Handle app runner
* Prepare the skeleton for generic app registration
* Prepare the skeleton for generic app registration
* Allow deployment
* Fix PII APP
* Fix lint
* Fix PII APP
* Fix PII APP
* Prepare config-based external apps
* Prepare config-based external apps
* Fix lint
* Prepare config-based external apps
* Fix DI errors
* Amend comments
2023-11-13 08:58:38 +01:00
Mayur Singal
367bac9064
Fix #13787 : Add support for ES data types ( #13916 )
...
* Fix #13787 : Add support for ES data types
* fixed tests
---------
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-11-10 20:14:42 +05:30
Pere Miquel Brull
7c06116b53
Add deprecation warnings ( #13927 )
2023-11-10 15:17:07 +05:30
Pere Miquel Brull
b250cd8808
Fix #13699 - Add separator for Storage Container manifest ( #13924 )
...
* Fix #13699 - Add separator for Storage Container manifest
* Fix #13906 - Fix add_mlmodel_lineage description field
* Add separator
* Add separator
2023-11-10 10:44:47 +01:00
Mayur Singal
a8145a82fa
Fix #13603 : Configurable Sample Data Rows for Profiler ( #13807 )
...
* Fix #13603 : Configurable Sample Data Rows
* Fix #13603 : Configurable Sample Data Rows for Profiler
* fix table config
* support configurable overwriting of sample data
* add support for schema and database profiler configuration
* chore(ui): put sampleDataStorageConfig under advanced config
* fix tests
* py format
* chore(ui): add sampleDataCount in table profiler config
* fix tests
* pylint & tests
* feat(ui): add profiler settings tab in database and database schema page
* chore(ui): show different inputs for profile sample type
* schema changes to make default storange config null
* add unit test
* schema changes to fix api
* update profiler setting schema
* move profiler settings to manage button
* sync locals
* fix(ui): unit tests
* fix tests
* py format
* fix lint
* minor improvements
* chore(ui): update profiler settings schema
* resolve review comments
* pytest
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-11-09 18:49:42 +05:30
Onkar Ravgan
c7834e74cc
fixed avro recursive record ( #13856 )
2023-11-06 16:27:06 +05:30
Teddy
d025e217d6
fix: catch not Either type in workflow and return explicit error message ( #13796 )
2023-11-02 13:02:26 +01:00
Ayush Shah
0a04ce85bb
Add Multilingual Support in EntityLink ( #13826 )
2023-11-02 14:35:22 +05:30
Teddy
10904049e4
fix: handle lower and upper case name ( #13778 )
2023-10-31 09:51:13 +01:00
Ayush Shah
bfb361dc85
Fix Bigquery lineage Pytests ( #13695 )
2023-10-25 11:15:41 +05:30
Ayush Shah
57cb72c26f
Fix Checkstyle ( #13683 )
2023-10-23 15:51:40 +05:30
Iaroslav Frolikov
420da29841
Fixes #13607 : BigQuery lineage ingestion fails when using GcpCredentialsPath authentication config ( #13608 )
2023-10-23 15:42:06 +05:30
Onkar Ravgan
0f0bccdd45
Converted and fixed pipelinestatus timestamps to milliseconds ( #13670 )
...
* fixed pipelinestatus timestamps in mills
* Added migrations
2023-10-20 09:39:24 -07:00
Pere Miquel Brull
8cf8720a9d
Clean Airflow Lineage Backend and migrate status to millis ( #13666 )
...
* Clean Airflow Lineage Backend and migrate status to millis
* Format
* chore(ui): update executions startTs and endTs to millis
* Remove lineage providers
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-10-20 15:42:38 +02:00
Pere Miquel Brull
660bf01a5b
Fix Stored Procedures Lineage for multi-db processes ( #13655 )
2023-10-20 09:14:08 +02:00
Teddy
1cbdfb3ae7
Fixes #12601 - column filter for profiler workflow ( #13535 )
...
* fix: sample data ingestion to match entity profiler column setting
* fix: python linting
* fix: updated fn call
* fix: added logic to handle json filed in datalake connector
* fix: handle NA values in parsing
* fix: reverted sampler changes from #13338
* fix: reverted metric changes from #13338
* fix: added datalake profiler ingestion test
* fix: python linting
* fix: removed normalization of json blob in NoSQL db
2023-10-12 14:51:38 +02:00
Onkar Ravgan
6e013246a7
dbt fixed null sql updates and source descriptions ( #13467 )
2023-10-12 11:07:58 +05:30
Teddy
e57849b732
Fixes #12298 - Update report data type to camel case ( #13505 )
...
* fix: updated DI to camelCase
* fix: ran linting
* fix: added migration
* fix: remove extra parenthesis in migration file
* fix: psql migration query
* fix: OS compose host
* fix: removed commented code block
2023-10-11 08:14:21 +02:00
Pere Miquel Brull
d3da2d1b9f
Register Ingestion pipelines just from YAML ( #13501 )
...
* Register Ingestion pipelines just from YAML
* Format
2023-10-10 07:04:04 +02:00
Ayush Shah
08d7ee6d55
Fixes #13052 : Datalake Nested Columns Sample Data ingestion ( #13338 )
2023-10-08 20:08:51 +05:30
Mayur Singal
0090286924
Fix Bigquery Test connection for multiproject ( #13380 )
...
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-05 14:50:42 +05:30
Pere Miquel Brull
0282574bdd
Create ometa client once and pass it around & improve pycln config ( #13310 )
...
* Create ometa client once and pass it around & improve pycln config
* Fix
* Fix
* Fix tests
* Fix maven ci
* Fix tests
* Fix tests
* Fix tests
* Format
* Fix DI
2023-10-04 09:14:03 +02:00
Pere Miquel Brull
d915254fac
Prepare Storage Connector for ADLS & Docs ( #13376 )
...
* Prepare Storage Connector for ADLS & Docs
* Format
* Fix test
2023-10-02 12:15:09 +02:00
Teddy
6ca71ae323
Issue 12679 - Handle Entity Object Instantiation Error + Refactor Workflow ( #13384 )
...
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes
* feat: __init__.py files creation
* feat: updated workflow import classes in code and doc
* feat: moved kpi runner from runner to processor folder
* fix: skip failure on list entities
* feat: deleted unused files
* feat: updated status reporter
* feat: ran linting
* feat: fix test error with typing and fqn
* feat: updated test dependencies
* feat: ran linting
* feat: move execution order up
* feat: updated cost analysis report to align with new workflow
* feat: fix entity already exists for pipeline entity status
* feat: ran python linting
* feat: move skip_on_failure to method
* feat: ran linting
---------
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-02 12:05:30 +02:00
Cristian Calugaru
5d8457b597
Fixes ISSUE-10587: global manifest option for storage services ( #12017 )
...
* global manifest option for storage services
* added a no metadata config source option for global manifest s3 services option
* merge fixes
* more merge fixes.
* black stuff
* test fixes
* formatting
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-28 07:55:40 +02:00
Ayush Shah
04760177f6
Fixes 13321: Fix Test Connection timeout ( #13323 )
2023-09-25 15:17:38 +05:30
Mayur Singal
65f65137e6
Fix: Bigquery query log not picked up for multiproject ( #13313 )
2023-09-25 08:07:48 +05:30
Teddy
e9ef7b5e81
Issue-12857: Remove ES Dependency from DI Workflow ( #13303 )
...
* feat: move elasticsearch indexing to backend + introduced EntityTimeSeries interface for timeseries type object
* feat: make reportData.json inherit from EntityTimeSeriesInterface
* feat: updated type to Object
* feat: deleted elasticsearch dependencies
* feat: removed elasticsearch indexing from workflow
* feat: added data insight sample data
* feat: cleaned up tests
2023-09-21 16:17:47 -07:00
Pere Miquel Brull
18a4513ccc
Fix #13237 - Rename to instanceDomain and test DomoDashboard charts ( #13247 )
...
* Rename sandboxDomain to instanceDomain
* Test Get Charts in DomoDashboard
* Fix schemas
* Fix test
* Fix test
* Rename to Auto Tag PII
* Fix query test
* Fix query test
* Fix query test
2023-09-19 14:14:04 +02:00
Teddy
b07dac8c1d
Issue-12305 - Add logic to sort test cases result by latest execution ( #13143 )
...
* fix: added logic to store test case result state with test case entity
* fix: added relative path logic in python unit test
* fix: added logic to sort testCases based on last execution date of test case result
* fix: ran python linting
* fix: added test case listing order param in frontend call
* fix: added tests around test suite result summary state
* fix: remove isDeleted repeated check
* fix: isolated test suite for maven tests + fix some UI failing tests
* fix: run java linting
2023-09-19 08:21:38 +02:00
Sriharsha Chintalapani
02094179e6
Fix #12899 : UI to use Tier TAG displayName if provided ( #13232 )
...
* Fix #12899 : UI to use Tier TAG displayName if provided
* fix python test
2023-09-17 14:28:10 -07:00
Pere Miquel Brull
442528267c
Simplify topology & update context management ( #13196 )
2023-09-15 09:44:42 +02:00
Ayush Shah
5fea08cd33
Datalake: Add manifest file support, fix profiler metrics, add array and json column type support ( #13017 )
2023-09-13 15:15:49 +05:30
Pere Miquel Brull
f0995cbddc
Part of #12998 - Prep Stored Procedures Skeleton for Snowflake ( #13121 )
...
* Prep Stored Procedures Skeleton for Snowflake
* Update pylint and add migrations
* Fix test
* Reuse source url computation
2023-09-12 14:25:42 +02:00
Mayur Singal
4e633877b3
Fix ElasticSearch Test Connection & Deploy ( #13061 )
2023-09-08 12:40:48 +05:30
Pere Miquel Brull
de7e06d024
Update structure for PII processing ( #13079 )
...
* Update structure for PII processing
* Fix tests
* Fix tests
* Lint
* Remove typo
2023-09-06 11:30:46 +02:00
Pere Miquel Brull
a9b13de893
Move workflows back into src ( #13100 )
...
* Move workflows back into src
* Fix test
2023-09-06 11:30:33 +02:00
Onkar Ravgan
a3ca8b6e66
Added project filter pattern to dashboard entity ( #12925 )
2023-09-05 11:01:28 +05:30
vanshika18
db592a157e
fix #12518 support get column lineage in spline ( #13030 )
2023-09-04 17:16:04 +05:30
Pere Miquel Brull
a3bfd4e696
Part of #11968 - Restructure Profiler Workflow and PII Processor ( #13059 )
...
* Structure PII
* Restructure Profiler Workflow
* Update signature for abc
* remove profiler sink
* Fix tests
* Fix lint
* Fix test
* Fix test
2023-09-04 11:02:57 +02:00
Mayur Singal
029786d773
Fix #9207 : Add source in lineage details ( #13046 )
2023-09-04 11:05:56 +05:30
Pere Miquel Brull
6c0e9f5061
Part of #7272 - Centralize Workflows, Status, and Exception Management ( #13029 )
...
* Prep changes
* Prep changes
* prep changes
* Update imports
* Format
* Prep delete
* Prep delete
* Fix sink
* Prep test
* Commit
* passing either
* passing either
* Prep Either
* Metadata source with Either
* Update status
* Merge remote-tracking branch 'upstream/main' into issue-7272
* Format
* Linting
* Linting
* Linting
* Linting
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Comments
2023-08-30 15:49:42 +02:00
vanshika18
3cc15e6d0b
Fixed #8418 added new connector couchbase ( #12836 )
2023-08-29 11:16:32 +05:30
Pere Miquel Brull
2f3a50e95f
Fix #8885 - Misleading 404 logs & suffix cleanup ( #12916 )
...
* Fix #8885 - Misleading 404 debug logs
* Clean get_suffix complexity
* Prepare airflow config
* Add bootstrap comment
* Revert "Add bootstrap comment"
This reverts commit cc053513df6e25bcc5e742786adb91339461fdd2.
* Revert "Prepare airflow config"
This reverts commit f57645fe953e17d6153652daa1f42c14b97c942d.
* add removed method
* Add icons for docs
* Add missing routes
2023-08-21 11:35:36 +02:00
Mayur Singal
2f7a2193e0
Fix #12302 : Add ElasticSearch Ingestion Source ( #12892 )
2023-08-18 15:10:31 +05:30
Onkar Ravgan
5b47fd4acf
Added source url to entities ( #12901 )
...
* Added source url to entites
* added support to create and update sourceUrl
* fixed pytests
---------
Co-authored-by: 07Himank <himank07mehta@gmail.com>
2023-08-18 10:17:38 +02:00
Onkar Ravgan
3114ef9b5c
Add support for JWT auth in Trino ( #12823 )
...
* Added jwt field in trino
* fixed postgres migration
* fixed pytests
2023-08-17 07:22:58 +02:00
Pere Miquel Brull
a183fc67e2
Fix ADLS parquet reads ( #12840 )
...
* Fix ADLS parquet reads
* Generalize service methods
* Fix tests
2023-08-14 19:57:06 -07:00