717 Commits

Author SHA1 Message Date
Sriharsha Chintalapani
49c1756583
MINOR - Test Case Ownership, keep severity, add test case result incidentId (#14691)
* MINOR - Test Case Ownership, keep severity, add test case result incidentId

* fix immutability

* Fix tests

* MINOR - Test Case Ownership, keep severity, add test case result incidentId

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-01-11 23:36:29 -08:00
Shiyang Xiao
241f3c68d7
Fixes #14413: Add SAS connector (#14415)
* feat: SAS Viya connector

* refactor SASCatalog to SAS

* add SAS logo to UI and connection documentation

* doc changes

* modify ingestion logic

* revert original changes

* added support for dataflow & perfect logic for reports/datatables

* add filter doc

* more updates to perfect ingestion for each asset type

* fix a bug with table lineage not created properly

* Delete ingestion/pipelines/sasCatalog.yaml

* precomit fix

* Conversion to database connector

* minor fixes

* make custom properties type generic

* Add SAS javaEnum

* add dummy variable for sas.yaml

---------

Co-authored-by: lizmc <liz.mcintosh@sas.com>
Co-authored-by: Shiyang Xiao <Shiyang.Xiao@sas.com>
2024-01-11 06:46:57 -08:00
Ayush Shah
9c6d202555
Add Sample data, modify regex pattern (#14467) 2024-01-11 14:23:33 +05:30
Mayur Singal
190212c8ac
Fix #11556: Add support for Db2 for IBM i (#14680) 2024-01-11 12:35:52 +05:30
Mayur Singal
9288620c29
Minor: Fix Feature Tables of Connectors (#14658) 2024-01-10 12:48:42 +05:30
Mayur Singal
2832bbcc8c
Fix #14362: Spark Pipeline Connector & Docs (#14626) 2024-01-09 12:40:43 +05:30
Sriharsha Chintalapani
8f37d7da05
Fix #14610: includeEmptyTestSuite behavior causing not include newly created test suites (#14611)
* Fix #14610: includeEmptyTestSuite behavior causing not include newly created test suites

* Fix tests

* Fix tests

* Fix tests
2024-01-08 18:58:47 -08:00
Mohit Yadav
dad886eaba
Remove LMAX Disruptor , use queue and Quartz (#14422)
* Remove LMAX Disruptor , use queue and Quartz

* Spotless Fix

* Spotless formatting

* Review Comments

* Improvements

* Improvements to add Retry and DLQ

* Spotless fix

* Update Stats

* Fix stats

* Fix delete Failures

* Fix Failing Tests

* Fix Failing Tests

* Fix Failing Tests

* Postgres Fix

* Fix Schema Change for Change Event Table

* Failing test

* Fix Error for non ChangeDescriptions

* Adds Observability On Top of Event Subscription

* Make List Separate

* Add more filters for Test Suite

* Populate Default Schema Updates to Look for

* typo

* Add matchFieldChange and other in paramAdditionalContext

* Review Comments

* Spotless fix

* Merge issue

* Update Filter as per new Syntax

* Fix Failing Test
2024-01-09 00:13:45 +05:30
Sriharsha Chintalapani
b9781746ce
Fix #8496: Segregate ES indexes for use in shared infrastructure (#14606)
* Fix #8496: Segregate ES indexes for use in shared infrastructure

* Fix tests

* use entityType aggregation key for counts

* localisation

* Revert openmetadata.yaml changes

* Update sample_data.yaml

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
2024-01-08 08:29:11 -08:00
Pere Miquel Brull
e8eb2bda3c
MINOR - Hide AutoTagger, add MetaPilot search, and fix apps status calls (#14201)
* MINOR - Hide AutoTagger & MetaPilot search & Fix apps status calls

* Fix method call

* Fix method call

* Fix method call

* "additionalProperties": true

* Use EntityRef

* Use ES sink

* Update UI schema type
2024-01-08 14:57:07 +01:00
Pere Miquel Brull
b92fd5cde4
MINOR - Move test case incident to the top of the data_quality_data_time_series (#14600)
* Add field and index

* MINOR - Move test case incident to the top of the data_quality_data_time_series table

* Fix test

* Fix compile

* Format

* Update incidentId

* Rename field

* Fix patch
2024-01-08 07:42:15 +01:00
Ayush Shah
d513dc4d4d
#11890: Add GCS Json schema changes (#14559)
* Issue-11890: Add GCS Json schema changes

* Modify Gcs Security config

* Add Test Def for gcs

* Add gcs for storage_metadata_config
2024-01-06 19:27:36 -08:00
Mayur Singal
a789fc86d6
Fix #13053: Remove Connection URI config MongoDB (#14584)
* Fix #13053: Remove Connection URI cofig MongoDB

* pyformat & test fixes
2024-01-05 10:51:12 -08:00
Pere Miquel Brull
0255171218
MINOR - Create Test Case Resolution ts entry & delete resolution when… (#14541)
* MINOR - Create Test Case Resolution ts entry & delete resolution when Test Case is deleted
2024-01-05 09:15:49 +01:00
Sriharsha Chintalapani
34ff582cf2
Allow user login using username and reassign roles based on users' ldap groups when login (#14550)
* Allow users to login with their user name in LDAP mode, and assign roles to user according to their LDAP group while login

* Fix #12503: Ldap feature: Allow user login using username and reassign roles based on users' ldap groups when login

---------

Co-authored-by: wentian <zb-wentian@kingmed.com.cn>
Co-authored-by: EricWent <135603556+EricWent@users.noreply.github.com>
2024-01-05 10:46:07 +05:30
Cristian Calugaru
1631172528
Issue 10587 - Storage Service doesn't handle deleted containers (#14497)
* global manifest option for storage services

* added a no metadata config source option for global manifest s3 services option

* merge fixes

* more merge fixes.

* black stuff

* test fixes

* formatting

* storage service container automatic deletion support

* reverted auto-modified files.

* styling

* linting
2023-12-30 13:27:16 -08:00
Pere Miquel Brull
1c2219517e
#13649 - Add Pipeline Status Skipped (#14524)
* #13649 - Add Pipeline Status Skipped

* Format
2023-12-29 09:41:59 +01:00
Teddy
2f679a9a48
FIXES #14294: return null if no aggregation value is present (#14514)
* fix: return null is no aggregation value is present

* fix: added Optional typing

* style: java linting

* fix: use Optional type hint

* style: java linting
2023-12-29 08:05:36 +01:00
Pere Miquel Brull
b84ce33b80
#11799 - Fix Airfow ownership & add pipeline tasks (#14510)
* Fix airflow owner and add tasks

* Add pipeline tasks ownership

* MINOR - Fix py CI

* Add pipeline tasks ownership

* Add pipeline tasks ownership

* MINOR - Fix py CI

* MINOR - Fix py CI

* Add pipeline tasks ownership

* patch team

* patch team

* Format
2023-12-28 10:25:00 -08:00
Teddy
3dc642989c
Fixes #7729 - Add logic to compute passed/failed rows (#14472)
* feat: add test case resolution task workflow

* chore: add migration for test case resolution feature

* fix: removed required field for object compatibiity in older migrations

* fix: minor testCaseResolution status logic

* chore: revert migration for test case incident

* chore: update migration file

* style: renamed variables

* feat: added logic to compute failed/passed rows

* feat: add support for row level computation in schema

* chore: add test definition migration

* feat: add logic to explicitly compute row level failure

* chore: clean up code

* style: fix java

* style: fix pyton format

* fix: unhidde API for incident manager

* style: fix java styling
2023-12-27 13:38:51 +01:00
Saurabh Yadav
0691c96eaf
FIxes 13543: Added Duckdb Ingestion Connector (#14468)
* changes

* Added duck db ingestion connector
2023-12-25 18:07:09 -08:00
Pere Miquel Brull
b786064bc2
#11857 - Store workflow status in the Ingestion Pipeline Status (#14462)
* Register StackTraceError in spec

* Register StackTraceError in spec

* Register StackTraceError in spec

* Add todos

* Update status

* docs

* format

* Fix tests

* Fix tests

* Fix tests

* Ignore generated

* Fix tests

* Fix tests

* Tests

* Try constants

* Try constants

* Print

* Print

* Print

* order

* Fix service name

* fix ui error

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-22 15:43:50 +01:00
Teddy
fe4c353bf3
Fixes #14331 - Add Test Case Incident Manager Migration (#14386)
* feat: add test case resolution task workflow

* chore: add migration for test case resolution feature

* fix: removed required field for object compatibiity in older migrations

* fix: minor testCaseResolution status logic

* chore: revert migration for test case incident

* chore: update migration file

* chore: clean up code

* style: fix java stylecheck
2023-12-22 11:01:51 +01:00
Ayush Shah
ebc0a551e5
Fixes 12947: Add Support For DQ and Profiler in Databricks Unity Catalog (#14424) 2023-12-20 21:18:05 +05:30
Ayush Shah
c3fac5e97e
Fix Adls connection typo (#14449) 2023-12-20 14:25:54 +01:00
NiharDoshi99
492092720d
#12908 support for table owner in postgres (#14289)
* support pick up postgres table owner

* added owner wrapper

* optimize the code

* remove pylint error

* minor fix

* changes as per comments

* fix python checksytyle and comments

* moved process owner to database service

* update docs for version 1.3

---------

Co-authored-by: Vanshika Kabra <vanshikakabra@Vanshikas-MacBook-Pro.local>
2023-12-20 18:42:59 +05:30
Onkar Ravgan
7d032977d7
Fix 14084: Removed Airflow Mssql backend connection (#14420)
* Removed Airflow Mssql backend conn

* added breaking changes docs
2023-12-20 12:28:28 +05:30
Ayush Shah
03f884b376
MINOR: Fix Typos (#14448) 2023-12-20 10:30:23 +05:30
Pere Miquel Brull
e16ad3c5dc
MINOR - Clean ingestion ES code and auth providers (#14358)
* MINOR - Clean ingestion ES code and auth providers

* clean parser

* Clean security config for the client

* Clean security config for the client

* Improve class conversion exceptions

* Fix tests

* Clean up java client

* Clean up java client

* clean parser

* Fix test

* Fix test

* fix NO_AUTH error

* Fix test

* Format

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-19 12:08:48 +01:00
Sriharsha Chintalapani
1294f93e15
Minor: Migrate to latest google code style library to support Java 17 and beyond (#14429)
* Minor: add spotless and use simplecontext

* Remove context from rule evaluaiton

* Fix EventSubscription tests

* Minor: Migrate to latest google code style library to support Java 17 and beyond

* Minor: Ignore code style migration from git blame
2023-12-18 12:56:17 -08:00
Pere Miquel Brull
a83a5ba3a3
MINOR - Skip delta tests for 3.11 (#14398)
* MINOR - Bump delta for 3.11

* Update flags

* MINOR - Bump delta for 3.11

* Update tests regex

* Update version

* Deprecations

* Format

* Version

* Try delta spark

* Skip delta tests for 3.11

* Update ingestion/tests/unit/topology/pipeline/test_airflow.py
2023-12-18 17:01:57 +01:00
Onkar Ravgan
1af1caf065
Prepare alation connector schemas (#14407) 2023-12-18 13:07:42 +05:30
Onkar Ravgan
ef48b7eae7
MINOR: Added custom properties methods in python sdk (#14402)
* Added custom properties methods in python sdk

* fixed tests
2023-12-18 12:57:14 +05:30
Pere Miquel Brull
d8984d267e
#14340 & #13849 - Clean secret ID and improve encrypt/decrypt exception management (#14356)
* Fix supported characters in SM

* Update SM

* Fixes

* Fixes

* Improve class conversion exceptions

* Comments

* Rename noop to db secrets manager providee

* Update sm

* Fix

* db SM

* db SM

* Fix test

* UI

* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts

* update default
2023-12-18 06:43:20 +01:00
Teddy
20bc238413
FIX #14248 - Implement test case resolution task workflow (#14323)
* feat: add test case resolution task workflow

* fix: conflict with main
2023-12-12 10:13:11 +01:00
Onkar Ravgan
d5ece87766
Fixed query millis duration desc (#14332) 2023-12-11 17:40:54 +05:30
Ayush Shah
b8d0b3b6e4
#14168: Mark SP and Data model deletion (#14219) 2023-12-06 08:57:41 +05:30
Onkar Ravgan
b3578cd496
Fix 14040: Part 2 Add Patch Request when updating the entities from create request (#14224) 2023-12-05 17:56:40 +05:30
Mohit Yadav
a68198298b
Adds a validation API (#14244)
* Adds a validation API

* Adds a validation API

* Add info for reindexing
2023-12-05 15:12:30 +05:30
Teddy
3bbf55fcda
FIXES #14049 - Split test case resolution status from test case result (#14204)
* refactor: entityFQN as ListFilter condition

* feat: implement resolution entity timeseries

* fix: rename to testCaseResolutionStatus

* ref: extracted ES query builder into private method

* ref: extract OS query builder in its own method

* ref: remove ingestion logic for test case resolution

* fix: reorganize json schemas to fix circular import in Python

* ref: object names in typescript code

* feat: added indexing of test case resolution

* feat: added test case resolution sample data

* fix: test case resolution api logic

* fix: audit logger for entityTimeSeriesInterface

* fix: DDL generation

* style: python linting

* fix: skip UI test case resolution tests

* fix: remove extension field

* fix: renamed testCaseFailureStatus to testCaseResolutionStatus

* fix: remove reviewer

* fix: rename sequenceId to stateId

* fix: re adjust search weights

* fix: removed InReview status

* style: ran python linting
2023-12-04 23:18:01 -08:00
Ashish Gupta
42463ff40b
#14134: supported retention period in table entity (#14163)
* supported retention period in table entity

* Add retention period updates

* supported unit test

* added curd operation for retention center

* minor changes

* fix modal issues and added validation

* added unit test for retention period

* fix code smell

* fix sonarcloud

* minor chnages

* Fix java code styling

* added hours in retention period

* changes as per comments

* fix sonar

* remove localization keys

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2023-12-05 10:42:37 +05:30
Mohit Yadav
0ae2baa9f6
Add Bulk Assets for Domain , Data product and Team (#14222)
* Add Bulk Assets for Domain , Data product and Team

* repetitive code
2023-12-04 19:06:16 +05:30
Mayur Singal
389ae79d3c
#14115: Separate Unity Catalog From Databricks (#14138) 2023-12-04 11:22:46 +05:30
Mohit Yadav
64a56b0183
FEAT Fix Issues in Glosary (#14197)
* Fix User Startup Email

* Add Apis for validation and addtion glossary tags and assets

* Update Status to ApiStatus

* Unique Tags to be listed

* Fix in tag additon

* fix: asset selection modal improvements (#14152)

* fix: assets fqn issue (#14140)

* fix: assets fqn issue

* fix: unit tests

* fix: escape fqn in remaining areas

* fix: domain cypress for adding assets

* fix: add assets to data products cypress

* update imports

* add cypress for glossary term

* fix: loading in asset modal and add cypress

* fix: unit tests

* fix: reintroduce filters in asset selection modal

* fix: update assets

* localization

* Fixes and Refactor redundant code

* Revert

* Add Delete API for removing tags from assets

* ADd ES calls

* Fix Assests added in case of errors

* ui updates

* ui changes

* Add Conditional Validate for SubFields

---------

Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2023-12-03 00:57:03 +05:30
Pere Miquel Brull
f7a4a67834
MINOR - Add Collate URL to MetaPilot App config (#14186) 2023-11-30 11:11:56 +01:00
chyueyi
b6b337e09a
feat: add support for doris datasource (#14087)
* feat: add support for doris datasource

* fix: fix python style check

* fix: add pydoris dependency

* fix: add pydoris dependency

* fix: py_format_check

* fix: parse error when doris view column is VARCHAR(*), check data length if not digit then return 1

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-11-28 13:57:52 +05:30
karanh37
853a83f8b9
fix: allow config update for external app (#14034)
* fix: allow config update for external app

* Prep config

* Add configs

* fix: call configure app on saving of configurations

* fix: add autocomplete widget for json schema forms

* ORganize configuration handling

* Allow config

* Format

* Fix test

* Prep migrations for allowConfiguration

* TODO

* Add pg migrations

* fix: add allowConfiguration support

* Autotagger

* Update install flow

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-11-27 16:55:33 -08:00
Sachin Chaurasiya
6ff27d90de
fix(#14121): hive connection metastoreConnection field is not updating the value correctly (#14123) 2023-11-27 16:34:06 +05:30
Mohit Yadav
19dbc217b0
Add code flow Configuration (#14026) 2023-11-20 08:44:13 +05:30
Suresh Srinivas
c4d6f681c8
Fixes #14006 Validate tag labels before creating/updating an entity (#14027) 2023-11-19 14:15:17 -08:00