2826 Commits

Author SHA1 Message Date
Onkar Ravgan
ef48b7eae7
MINOR: Added custom properties methods in python sdk (#14402)
* Added custom properties methods in python sdk

* fixed tests
2023-12-18 12:57:14 +05:30
Pere Miquel Brull
d8984d267e
#14340 & #13849 - Clean secret ID and improve encrypt/decrypt exception management (#14356)
* Fix supported characters in SM

* Update SM

* Fixes

* Fixes

* Improve class conversion exceptions

* Comments

* Rename noop to db secrets manager providee

* Update sm

* Fix

* db SM

* db SM

* Fix test

* UI

* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts

* update default
2023-12-18 06:43:20 +01:00
Mayur Singal
3e1bef48c7
Minor: Fix casing in MSSQL test connection query (#14412)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-12-17 16:02:32 +01:00
Pere Miquel Brull
6d046f7e3b
Revert "Update Dockerfile.ci file to use new MySQL Keys (#14400)" (#14416)
This reverts commit 33f2fdd49cae50c93cbf1f8544402bda41070a01.
2023-12-16 16:46:33 +01:00
Tom Bushell
bb3f9a4441
Fixes #9919: Cleaned up postgres lineage and fixed function in postgres query_parser (#13678)
* Replaced s.query with query_text in filters for clarity

* Removed redundant _iter function

* Updated process_table_query to return None when there are no queries to process

* Modified process_table_query to yield None if there are no queries

* Resolved formating issues and modified process_table_query to return None instead of yielding None

* pylint, fix bugs and improve code

* fix import

* pyformat

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-12-15 20:14:19 +05:30
Preet Shah
33f2fdd49c
Update Dockerfile.ci file to use new MySQL Keys (#14400)
* Update Dockerfile.ci file to use new MySQL Keys

* Update ingestion Dockerfiles files to use new MySQL Keys

* Update ingestion Dockerfiles files

* Update Dockerfiles to not hardcode the new keys

* Update comments
2023-12-15 15:46:30 +05:30
Pere Miquel Brull
eaacc693bd
#12027 - Add support for Python 3.11 (#14385)
* Fix datamodel codegen and bump versions

* Add 3.11 tests

* Update hive

* pandas

* pandas
2023-12-14 15:46:58 +01:00
Pere Miquel Brull
d4d8f3655f
#14043 - Fix metadata backup memory profile (#14378)
* Update looker docs

* Fix db dump
2023-12-14 15:46:35 +01:00
C. Kris
74df616679
Fixes #14215: Add missing decode stage to gz/zip files in json ingestion reader. (#14375)
* add decoding stage to gz/zip files.

Files that where zip/gz where not being decoded.
This was leading to a error when we wanted them to be.

* remove unnecessary comment

---------

Co-authored-by: Carl Kristensen <carl.johan.coelho.kristensen@schibsted.com>
2023-12-14 12:47:58 +01:00
Mayur Singal
4f65287dae
Minor: Airflow skip task status if task deleted (#14373) 2023-12-13 18:34:14 +05:30
Mayur Singal
35d4c64e69
Minor: Improve exception handling in Superset (#14371) 2023-12-13 18:33:59 +05:30
Mayur Singal
f53fcf9168
Minor: Fix e2e count mismatch (#14352)
* Minor: Fix e2e count mismatch

* fix dbt test

* fix pyformat

* use > instead of ==
2023-12-12 22:32:13 +05:30
Teddy
20bc238413
FIX #14248 - Implement test case resolution task workflow (#14323)
* feat: add test case resolution task workflow

* fix: conflict with main
2023-12-12 10:13:11 +01:00
Onkar Ravgan
7f6311ec4c
Fixed entity req left check (#14341) 2023-12-12 10:57:09 +05:30
Teddy
a4f9b0ec4e
fix: renamed API endpoints from resolution to status (#14330) 2023-12-11 14:09:40 +01:00
Onkar Ravgan
e4bfd12811
fixed e2e test after patch and collate demo (#14321) 2023-12-08 19:55:49 +05:30
Lucas Garcia
fe06b5cbb2
#14235: adding dialect based on connection type to LineageParser (#14249)
* Fix #14235: adding dialect based on connection type to LineageParser

* Fix: formating changes

* Update ingestion/src/metadata/ingestion/source/dashboard/metabase/metadata.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* style: fix indentation errors

* Fix pytest

---------

Co-authored-by: LucasGarcia07 <lucas.junqueira@hurb.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-12-08 19:49:59 +05:30
NiharDoshi99
8d925c46a5
#13696: add support for dot in schema name to fetch tables (#14246) 2023-12-08 12:04:28 +05:30
Onkar Ravgan
cb94243370
fix desc and tags update from patch (#14305) 2023-12-07 18:39:38 +05:30
Onkar Ravgan
4ade6fd39f
Remove col displayname from patch req (#14279) 2023-12-06 10:12:04 -08:00
Mayur Singal
d6a01ca5b1
Minor: Remove retension period from patch request (#14276)
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-12-06 20:29:44 +05:30
Akash Jain
6eefd51180
fix: AIRFLOW_DB_PROPERTIES default value in docker compose files (#14274) 2023-12-06 19:56:19 +05:30
Onkar Ravgan
c7e32c8abf
Fixes: 14240 and Fixes 13365: Bulk patch descriptions from dbt (#13999) 2023-12-06 19:00:31 +05:30
chyueyi
070ff1f605
feat: add profile for doris connector (#14214) 2023-12-06 08:59:56 +05:30
Ayush Shah
b8d0b3b6e4
#14168: Mark SP and Data model deletion (#14219) 2023-12-06 08:57:41 +05:30
Onkar Ravgan
b3578cd496
Fix 14040: Part 2 Add Patch Request when updating the entities from create request (#14224) 2023-12-05 17:56:40 +05:30
Teddy
3bbf55fcda
FIXES #14049 - Split test case resolution status from test case result (#14204)
* refactor: entityFQN as ListFilter condition

* feat: implement resolution entity timeseries

* fix: rename to testCaseResolutionStatus

* ref: extracted ES query builder into private method

* ref: extract OS query builder in its own method

* ref: remove ingestion logic for test case resolution

* fix: reorganize json schemas to fix circular import in Python

* ref: object names in typescript code

* feat: added indexing of test case resolution

* feat: added test case resolution sample data

* fix: test case resolution api logic

* fix: audit logger for entityTimeSeriesInterface

* fix: DDL generation

* style: python linting

* fix: skip UI test case resolution tests

* fix: remove extension field

* fix: renamed testCaseFailureStatus to testCaseResolutionStatus

* fix: remove reviewer

* fix: rename sequenceId to stateId

* fix: re adjust search weights

* fix: removed InReview status

* style: ran python linting
2023-12-04 23:18:01 -08:00
Ashish Gupta
42463ff40b
#14134: supported retention period in table entity (#14163)
* supported retention period in table entity

* Add retention period updates

* supported unit test

* added curd operation for retention center

* minor changes

* fix modal issues and added validation

* added unit test for retention period

* fix code smell

* fix sonarcloud

* minor chnages

* Fix java code styling

* added hours in retention period

* changes as per comments

* fix sonar

* remove localization keys

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2023-12-05 10:42:37 +05:30
Teddy
f571f67eda
fix: DI skip on failure when listing entities (#14226) 2023-12-04 16:59:09 +00:00
Mayur Singal
5965d4c63b
Minor: Fix airflow empty status (#14216) 2023-12-04 07:54:13 -08:00
Mayur Singal
4843d85bf9
Minor: Fix kafka Topic Entity Import (#14227) 2023-12-04 21:08:33 +05:30
Mayur Singal
b57e6710b6
Minor: Fix 'str' object has no attribute 'serviceType' for SP Lineage (#14221) 2023-12-04 19:52:02 +05:30
Preet Shah
20cf8f63ab
Update ingestion_dependency.sh (#14139)
* Update ingestion_dependency.sh

* Add parameter for DB connection properties

---------

Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
2023-12-04 18:09:42 +05:30
Mayur Singal
389ae79d3c
#14115: Separate Unity Catalog From Databricks (#14138) 2023-12-04 11:22:46 +05:30
Harpreet
b50f6f8939
#14028: fix the sed command issue in ingestion dep script (#14029)
* fix the sed command issue in ingestion dep script

* remove sql_alchemy_conn from airflow.cfg and use env variable

* try execute ingestion dep script without source
2023-12-03 11:27:36 -08:00
Pere Miquel Brull
7fcdf08ca4
#11626 & #14131 - Lineage with other Entities & attr-based xlets (#14191)
* Add OMEntity model

* Test OMEntity

* Update repr

* Fix __str__

* Add entity ref map

* Test serializer for backend

* Fix tests

* Fix serializer

* Test runner

* Add runner tests

* Update docs

* Format
2023-12-01 06:29:44 +01:00
Onkar Ravgan
b403c2b250
Added SP lineage as edge (#14180) 2023-11-30 16:36:52 +05:30
Mayur Singal
2092a52048
Minor: Fix E2E Tests failure (#14171) 2023-11-29 12:30:37 -08:00
Teddy
8ff70e31fe
MINOR: fixes ingestion of sample data for custom metrics (#14170)
* fix: updated sample data ingestoon for custom metrics

* style: fix python linting

* style: fix java linting
2023-11-29 17:28:19 +01:00
Mayur Singal
49a5557e44
#14099: fix bigquery test connection & SP query (#14106)
* Fix #14099: fix bigquery test connection & sp query

* fix unquote condition

* pyformat
2023-11-28 14:02:46 +01:00
chyueyi
b6b337e09a
feat: add support for doris datasource (#14087)
* feat: add support for doris datasource

* fix: fix python style check

* fix: add pydoris dependency

* fix: add pydoris dependency

* fix: py_format_check

* fix: parse error when doris view column is VARCHAR(*), check data length if not digit then return 1

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-11-28 13:57:52 +05:30
Onkar Ravgan
b56b67bf67
Fixed overwriting data for service entites (#14135) 2023-11-28 08:49:29 +01:00
karanh37
853a83f8b9
fix: allow config update for external app (#14034)
* fix: allow config update for external app

* Prep config

* Add configs

* fix: call configure app on saving of configurations

* fix: add autocomplete widget for json schema forms

* ORganize configuration handling

* Allow config

* Format

* Fix test

* Prep migrations for allowConfiguration

* TODO

* Add pg migrations

* fix: add allowConfiguration support

* Autotagger

* Update install flow

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-11-27 16:55:33 -08:00
Onkar Ravgan
ebb2317cf0
Fix 14040: Part 1 Remove get_by_name calls from topology (#14098)
* Changed for database

* Added changes for dashboard_service

* Changed for messaging service

* Changed for mlmodel service

* Changed for pipeline service

* Changed for search service

* Changed for objectstore service

* fixed wrong import

* fixed lint

* fixes

* fixed pytests

* fixed domo db pytest

* Fixed review comments
2023-11-27 16:15:47 +05:30
Mayur Singal
aa00802f6d
Fix #14100: mssql fix usage query, make table name lowercase (#14101) 2023-11-27 09:37:37 +01:00
VolkovGeoPhy
c95de19a09
great-expectations~=0.18.0 (#14056) 2023-11-23 12:33:07 +01:00
Mayur Singal
4bce6a9aaf
Athena: Add support for work groups in usage & lineage (#14072) 2023-11-23 11:13:35 +05:30
Teddy
131eea32f8
Add Custom metrics sample data (#14069)
* feat: add sample data for custom metrics

* feat: update project.scripts to `metadata`
2023-11-22 11:57:19 +01:00
Pere Miquel Brull
c7e758eccc
Fix pyproject - TypeError: 'list' object is not a mapping (#14064)
* Fix pyproject - TypeError: 'list' object is not a mapping

* Add dynamic optional-dependencies

* Add dynamic optional-dependencies

* Bump datamodel

* Bump datamodel
2023-11-22 08:38:47 +01:00
Pere Miquel Brull
caaf0e7a1d
Fix #12436 - Migrate to pyproject.toml (#14025)
* test

* Use pyproject.toml

* Fix pylint

* makefile

* makefile

* Fix pylint

* isort

* pyproject

* Airflow apis pyproject

* Remove ingestion core

* isort

* Fix makefile help
2023-11-22 07:10:37 +01:00