1157 Commits

Author SHA1 Message Date
Teddy
3dc642989c
Fixes #7729 - Add logic to compute passed/failed rows (#14472)
* feat: add test case resolution task workflow

* chore: add migration for test case resolution feature

* fix: removed required field for object compatibiity in older migrations

* fix: minor testCaseResolution status logic

* chore: revert migration for test case incident

* chore: update migration file

* style: renamed variables

* feat: added logic to compute failed/passed rows

* feat: add support for row level computation in schema

* chore: add test definition migration

* feat: add logic to explicitly compute row level failure

* chore: clean up code

* style: fix java

* style: fix pyton format

* fix: unhidde API for incident manager

* style: fix java styling
2023-12-27 13:38:51 +01:00
Pere Miquel Brull
1339d6518f
MINOR - Fix py CI (#14488) 2023-12-22 16:27:21 +01:00
Pere Miquel Brull
b786064bc2
#11857 - Store workflow status in the Ingestion Pipeline Status (#14462)
* Register StackTraceError in spec

* Register StackTraceError in spec

* Register StackTraceError in spec

* Add todos

* Update status

* docs

* format

* Fix tests

* Fix tests

* Fix tests

* Ignore generated

* Fix tests

* Fix tests

* Tests

* Try constants

* Try constants

* Print

* Print

* Print

* order

* Fix service name

* fix ui error

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-22 15:43:50 +01:00
Pere Miquel Brull
7e8e4a7e68
#14320 - FIx Airflow Callback datetime conversion (#14487)
* #14320 - don't cast twice

* #14320 - Fix Airflow Callback datetime conversion

* import
2023-12-22 15:43:41 +01:00
Onkar Ravgan
79444f4a24
Fix 13823: Validate and Parse gracefully IngestionPipeline (#14461)
* fixed ingestion pipeline parsing

* Added validation for automation workflow

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-12-22 09:56:39 +05:30
Ayush Shah
ebc0a551e5
Fixes 12947: Add Support For DQ and Profiler in Databricks Unity Catalog (#14424) 2023-12-20 21:18:05 +05:30
Pere Miquel Brull
e16ad3c5dc
MINOR - Clean ingestion ES code and auth providers (#14358)
* MINOR - Clean ingestion ES code and auth providers

* clean parser

* Clean security config for the client

* Clean security config for the client

* Improve class conversion exceptions

* Fix tests

* Clean up java client

* Clean up java client

* clean parser

* Fix test

* Fix test

* fix NO_AUTH error

* Fix test

* Format

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-19 12:08:48 +01:00
Pere Miquel Brull
a83a5ba3a3
MINOR - Skip delta tests for 3.11 (#14398)
* MINOR - Bump delta for 3.11

* Update flags

* MINOR - Bump delta for 3.11

* Update tests regex

* Update version

* Deprecations

* Format

* Version

* Try delta spark

* Skip delta tests for 3.11

* Update ingestion/tests/unit/topology/pipeline/test_airflow.py
2023-12-18 17:01:57 +01:00
Onkar Ravgan
ef48b7eae7
MINOR: Added custom properties methods in python sdk (#14402)
* Added custom properties methods in python sdk

* fixed tests
2023-12-18 12:57:14 +05:30
Pere Miquel Brull
d8984d267e
#14340 & #13849 - Clean secret ID and improve encrypt/decrypt exception management (#14356)
* Fix supported characters in SM

* Update SM

* Fixes

* Fixes

* Improve class conversion exceptions

* Comments

* Rename noop to db secrets manager providee

* Update sm

* Fix

* db SM

* db SM

* Fix test

* UI

* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts

* update default
2023-12-18 06:43:20 +01:00
Mayur Singal
f53fcf9168
Minor: Fix e2e count mismatch (#14352)
* Minor: Fix e2e count mismatch

* fix dbt test

* fix pyformat

* use > instead of ==
2023-12-12 22:32:13 +05:30
Onkar Ravgan
e4bfd12811
fixed e2e test after patch and collate demo (#14321) 2023-12-08 19:55:49 +05:30
Lucas Garcia
fe06b5cbb2
#14235: adding dialect based on connection type to LineageParser (#14249)
* Fix #14235: adding dialect based on connection type to LineageParser

* Fix: formating changes

* Update ingestion/src/metadata/ingestion/source/dashboard/metabase/metadata.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* style: fix indentation errors

* Fix pytest

---------

Co-authored-by: LucasGarcia07 <lucas.junqueira@hurb.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-12-08 19:49:59 +05:30
NiharDoshi99
8d925c46a5
#13696: add support for dot in schema name to fetch tables (#14246) 2023-12-08 12:04:28 +05:30
Teddy
3bbf55fcda
FIXES #14049 - Split test case resolution status from test case result (#14204)
* refactor: entityFQN as ListFilter condition

* feat: implement resolution entity timeseries

* fix: rename to testCaseResolutionStatus

* ref: extracted ES query builder into private method

* ref: extract OS query builder in its own method

* ref: remove ingestion logic for test case resolution

* fix: reorganize json schemas to fix circular import in Python

* ref: object names in typescript code

* feat: added indexing of test case resolution

* feat: added test case resolution sample data

* fix: test case resolution api logic

* fix: audit logger for entityTimeSeriesInterface

* fix: DDL generation

* style: python linting

* fix: skip UI test case resolution tests

* fix: remove extension field

* fix: renamed testCaseFailureStatus to testCaseResolutionStatus

* fix: remove reviewer

* fix: rename sequenceId to stateId

* fix: re adjust search weights

* fix: removed InReview status

* style: ran python linting
2023-12-04 23:18:01 -08:00
Teddy
f571f67eda
fix: DI skip on failure when listing entities (#14226) 2023-12-04 16:59:09 +00:00
Mayur Singal
389ae79d3c
#14115: Separate Unity Catalog From Databricks (#14138) 2023-12-04 11:22:46 +05:30
Pere Miquel Brull
7fcdf08ca4
#11626 & #14131 - Lineage with other Entities & attr-based xlets (#14191)
* Add OMEntity model

* Test OMEntity

* Update repr

* Fix __str__

* Add entity ref map

* Test serializer for backend

* Fix tests

* Fix serializer

* Test runner

* Add runner tests

* Update docs

* Format
2023-12-01 06:29:44 +01:00
Onkar Ravgan
ebb2317cf0
Fix 14040: Part 1 Remove get_by_name calls from topology (#14098)
* Changed for database

* Added changes for dashboard_service

* Changed for messaging service

* Changed for mlmodel service

* Changed for pipeline service

* Changed for search service

* Changed for objectstore service

* fixed wrong import

* fixed lint

* fixes

* fixed pytests

* fixed domo db pytest

* Fixed review comments
2023-11-27 16:15:47 +05:30
Onkar Ravgan
335aa86fcc
fixed athena e2e (#14041) 2023-11-20 21:20:32 +05:30
Onkar Ravgan
b781c66206
Fixed tableau e2e counts due to new sample data in the instance (#14039)
* fixed tableau e2e new

* rmv hive test
2023-11-20 19:54:06 +05:30
Mayur Singal
9f14ef7fab
Fix #13954: Fix ParseException for older version of databricks (#14015) 2023-11-20 13:14:40 +05:30
Teddy
c7ac28f2c2
Fixes #11357 - Implement profiler custom metric processing (#14021)
* feat: add backend support for custom metrics

* feat: fix python test

* feat: support custom metrics computation

* feat: updated tests for custom metrics

* feat: added dl support for min max of datetime

* feat: added is safe query check for query sampler

* feat: added support for custom metric computation in dl

* feat: added explicit addProper for pydantic model import fo Extra

* feat: added custom metric to returned obj

* feat: wrapped trino import in __init__

* feat: fix python linting

* feat: fix typing in 3.8
2023-11-17 17:51:39 +01:00
Teddy
f3da919329
Feat: Backend Support for Custom Metrics (#13965)
* feat: add backend support for custom metrics

* feat: fix python test
2023-11-17 19:16:35 +05:30
Suresh Srinivas
5f34bd02d3
Fixes #13595 Consolidate changes by a user in a single session to a single change (#13617)
* getChangeDescription to use entity and update type to decide previous version

* Fixes #13595 - Consolidate changes by a user in a single session to a single change
2023-11-16 18:14:27 -08:00
Mohit Yadav
10d8ec84fe
Logs added for Search Indexing and Stats issue fixed (#13956)
* Logs added for Search Indexing and Stats issue fixed

* Fix uninstall error

* Add error handling

* fix lint

* Push Job Level Exception on top

* disable flaky tests

* Fix Logs not visible in Search

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-11-13 23:39:56 +05:30
Pere Miquel Brull
c742835766
Auto Tagger Application - Preparing the Ingestion Framework (#13862)
* Prepare the skeleton for generic app registration

* Prepare the skeleton for generic app registration

* Handle app runner

* Prepare the skeleton for generic app registration

* Prepare the skeleton for generic app registration

* Allow deployment

* Fix PII APP

* Fix lint

* Fix PII APP

* Fix PII APP

* Prepare config-based external apps

* Prepare config-based external apps

* Fix lint

* Prepare config-based external apps

* Fix DI errors

* Amend comments
2023-11-13 08:58:38 +01:00
Mayur Singal
367bac9064
Fix #13787: Add support for ES data types (#13916)
* Fix #13787: Add support for ES data types

* fixed tests

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-11-10 20:14:42 +05:30
Pere Miquel Brull
0eacc829a4
Fix #13794 - Add domain support to the Python SDK (#13931) 2023-11-10 11:00:06 +01:00
Pere Miquel Brull
7c06116b53
Add deprecation warnings (#13927) 2023-11-10 15:17:07 +05:30
Pere Miquel Brull
8891a9a410
Fix #13906 - Fix add_mlmodel_lineage description field (#13920) 2023-11-10 15:16:09 +05:30
Pere Miquel Brull
b250cd8808
Fix #13699 - Add separator for Storage Container manifest (#13924)
* Fix #13699 - Add separator for Storage Container manifest

* Fix #13906 - Fix add_mlmodel_lineage description field

* Add separator

* Add separator
2023-11-10 10:44:47 +01:00
Mayur Singal
4b625f7ba5
Move pandas top level import (#13926) 2023-11-10 14:15:14 +05:30
Mayur Singal
a8145a82fa
Fix #13603: Configurable Sample Data Rows for Profiler (#13807)
* Fix #13603: Configurable Sample Data Rows

* Fix #13603: Configurable Sample Data Rows for Profiler

* fix table config

* support configurable overwriting of sample data

* add support for schema and database profiler configuration

* chore(ui): put sampleDataStorageConfig under advanced config

* fix tests

* py format

* chore(ui): add sampleDataCount in table profiler config

* fix tests

* pylint & tests

* feat(ui): add profiler settings tab in database and database schema page

* chore(ui): show different inputs for profile sample type

* schema changes to make default storange config null

* add unit test

* schema changes to fix api

* update profiler setting schema

* move profiler settings to manage button

* sync locals

* fix(ui): unit tests

* fix tests

* py format

* fix lint

* minor improvements

* chore(ui): update profiler settings schema

* resolve review comments

* pytest

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-11-09 18:49:42 +05:30
Suresh Srinivas
a89e317a2b
Fixes #13863 - Show inherited relationships of an entity (#13864)
* Fixes #13863 - Show inherited relationships of an entity

* Test failure fixes

* Commenting out invalid python test
2023-11-07 09:11:06 -08:00
Onkar Ravgan
c7834e74cc
fixed avro recursive record (#13856) 2023-11-06 16:27:06 +05:30
Teddy
d025e217d6
fix: catch not Either type in workflow and return explicit error message (#13796) 2023-11-02 13:02:26 +01:00
Ayush Shah
0a04ce85bb
Add Multilingual Support in EntityLink (#13826) 2023-11-02 14:35:22 +05:30
Teddy
10904049e4
fix: handle lower and upper case name (#13778) 2023-10-31 09:51:13 +01:00
Teddy
452a33b1a0
Fixes Druid Profiler failures (#13700)
* fix: updated playwrigth test structure

* fix: druid profiler queries

* fix: python linting

* fix: python linting

* fix: do not compute random sample if profile sample is 100

* fix: updated workflow to test on push

* fix: move connector config to category folder

* fix: updated imports

* fix: added pytest-dependency package

* fix: updated readme.md

* fix: python linting

* fix: updated profile doc for Druid sampling

* fix: empty commit for CI

* fix: added workflow constrain back

* fix: sonar code smell

* fix: added secrets to container

* Update openmetadata-docs/content/v1.2.x-SNAPSHOT/connectors/ingestion/workflows/profiler/index.md

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

* Update openmetadata-docs/content/v1.2.x-SNAPSHOT/connectors/ingestion/workflows/profiler/index.md

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

* Update ingestion/tests/e2e/entity/database/test_redshift.py

* fix: ran pylint

* fix: updated redshift env var.

* fix: import linting

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-10-25 20:47:51 +02:00
Ayush Shah
bfb361dc85
Fix Bigquery lineage Pytests (#13695) 2023-10-25 11:15:41 +05:30
Ayush Shah
57cb72c26f
Fix Checkstyle (#13683) 2023-10-23 15:51:40 +05:30
Iaroslav Frolikov
420da29841
Fixes #13607: BigQuery lineage ingestion fails when using GcpCredentialsPath authentication config (#13608) 2023-10-23 15:42:06 +05:30
Onkar Ravgan
0f0bccdd45
Converted and fixed pipelinestatus timestamps to milliseconds (#13670)
* fixed pipelinestatus timestamps in mills

* Added migrations
2023-10-20 09:39:24 -07:00
Pere Miquel Brull
8cf8720a9d
Clean Airflow Lineage Backend and migrate status to millis (#13666)
* Clean Airflow Lineage Backend and migrate status to millis

* Format

* chore(ui): update executions startTs and endTs to millis

* Remove lineage providers

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-10-20 15:42:38 +02:00
Pere Miquel Brull
660bf01a5b
Fix Stored Procedures Lineage for multi-db processes (#13655) 2023-10-20 09:14:08 +02:00
Ayush Shah
ad86d8969f
Fix E2E failures (#13648) 2023-10-19 17:49:02 +05:30
Pere Miquel Brull
255bfb95b1
Remove duplicates from entity_extension_time_series and add the const… (#13626)
* Remove duplicates from entity_extension_time_series and add the constraing if missing

* Add sort buffer and work mem

* Revert "Add sort buffer and work mem"

This reverts commit fcfff5feb60c9212bb7c1cad34b524dc8c03bfc5.
2023-10-19 12:15:02 +02:00
Ayush Shah
f94e2dbb47
Fix Hive Bytes issue, add athena yaml, fix bigquerymultiple project id token issue (#13640) 2023-10-18 23:48:21 +05:30
Ayush Shah
ac9e8c9e89
Add E2E - Oracle, Athena. Remove Duplicated code (#13563) 2023-10-18 16:57:06 +05:30