3385 Commits

Author SHA1 Message Date
Onkar Ravgan
79444f4a24
Fix 13823: Validate and Parse gracefully IngestionPipeline (#14461)
* fixed ingestion pipeline parsing

* Added validation for automation workflow

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-12-22 09:56:39 +05:30
Aldwyn Cabarrubias
0e4f80899e
Update profiler_interface.py (#14287)
Transpose row=None before try statement in compute_metrics_in_thread function

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-12-21 10:37:59 +01:00
Ayush Shah
ebc0a551e5
Fixes 12947: Add Support For DQ and Profiler in Databricks Unity Catalog (#14424) 2023-12-20 21:18:05 +05:30
NiharDoshi99
492092720d
#12908 support for table owner in postgres (#14289)
* support pick up postgres table owner

* added owner wrapper

* optimize the code

* remove pylint error

* minor fix

* changes as per comments

* fix python checksytyle and comments

* moved process owner to database service

* update docs for version 1.3

---------

Co-authored-by: Vanshika Kabra <vanshikakabra@Vanshikas-MacBook-Pro.local>
2023-12-20 18:42:59 +05:30
Onkar Ravgan
7d032977d7
Fix 14084: Removed Airflow Mssql backend connection (#14420)
* Removed Airflow Mssql backend conn

* added breaking changes docs
2023-12-20 12:28:28 +05:30
Onkar Ravgan
841fd9cfe7
Fixed superset excp (#14447) 2023-12-20 09:30:37 +05:30
Pere Miquel Brull
e16ad3c5dc
MINOR - Clean ingestion ES code and auth providers (#14358)
* MINOR - Clean ingestion ES code and auth providers

* clean parser

* Clean security config for the client

* Clean security config for the client

* Improve class conversion exceptions

* Fix tests

* Clean up java client

* Clean up java client

* clean parser

* Fix test

* Fix test

* fix NO_AUTH error

* Fix test

* Format

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-19 12:08:48 +01:00
Pere Miquel Brull
0e92a975e3
#14425 - Create ingestion-base-slim image (#14426)
* #14425 - Create ingestion-base-slim image

* Format

* Bump airflow

* Bump constraints
2023-12-19 11:09:38 +01:00
Pere Miquel Brull
a83a5ba3a3
MINOR - Skip delta tests for 3.11 (#14398)
* MINOR - Bump delta for 3.11

* Update flags

* MINOR - Bump delta for 3.11

* Update tests regex

* Update version

* Deprecations

* Format

* Version

* Try delta spark

* Skip delta tests for 3.11

* Update ingestion/tests/unit/topology/pipeline/test_airflow.py
2023-12-18 17:01:57 +01:00
Onkar Ravgan
ef48b7eae7
MINOR: Added custom properties methods in python sdk (#14402)
* Added custom properties methods in python sdk

* fixed tests
2023-12-18 12:57:14 +05:30
Pere Miquel Brull
d8984d267e
#14340 & #13849 - Clean secret ID and improve encrypt/decrypt exception management (#14356)
* Fix supported characters in SM

* Update SM

* Fixes

* Fixes

* Improve class conversion exceptions

* Comments

* Rename noop to db secrets manager providee

* Update sm

* Fix

* db SM

* db SM

* Fix test

* UI

* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts

* update default
2023-12-18 06:43:20 +01:00
Mayur Singal
3e1bef48c7
Minor: Fix casing in MSSQL test connection query (#14412)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-12-17 16:02:32 +01:00
Pere Miquel Brull
6d046f7e3b
Revert "Update Dockerfile.ci file to use new MySQL Keys (#14400)" (#14416)
This reverts commit 33f2fdd49cae50c93cbf1f8544402bda41070a01.
2023-12-16 16:46:33 +01:00
Tom Bushell
bb3f9a4441
Fixes #9919: Cleaned up postgres lineage and fixed function in postgres query_parser (#13678)
* Replaced s.query with query_text in filters for clarity

* Removed redundant _iter function

* Updated process_table_query to return None when there are no queries to process

* Modified process_table_query to yield None if there are no queries

* Resolved formating issues and modified process_table_query to return None instead of yielding None

* pylint, fix bugs and improve code

* fix import

* pyformat

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-12-15 20:14:19 +05:30
Preet Shah
33f2fdd49c
Update Dockerfile.ci file to use new MySQL Keys (#14400)
* Update Dockerfile.ci file to use new MySQL Keys

* Update ingestion Dockerfiles files to use new MySQL Keys

* Update ingestion Dockerfiles files

* Update Dockerfiles to not hardcode the new keys

* Update comments
2023-12-15 15:46:30 +05:30
Pere Miquel Brull
eaacc693bd
#12027 - Add support for Python 3.11 (#14385)
* Fix datamodel codegen and bump versions

* Add 3.11 tests

* Update hive

* pandas

* pandas
2023-12-14 15:46:58 +01:00
Pere Miquel Brull
d4d8f3655f
#14043 - Fix metadata backup memory profile (#14378)
* Update looker docs

* Fix db dump
2023-12-14 15:46:35 +01:00
C. Kris
74df616679
Fixes #14215: Add missing decode stage to gz/zip files in json ingestion reader. (#14375)
* add decoding stage to gz/zip files.

Files that where zip/gz where not being decoded.
This was leading to a error when we wanted them to be.

* remove unnecessary comment

---------

Co-authored-by: Carl Kristensen <carl.johan.coelho.kristensen@schibsted.com>
2023-12-14 12:47:58 +01:00
Mayur Singal
4f65287dae
Minor: Airflow skip task status if task deleted (#14373) 2023-12-13 18:34:14 +05:30
Mayur Singal
35d4c64e69
Minor: Improve exception handling in Superset (#14371) 2023-12-13 18:33:59 +05:30
Mayur Singal
f53fcf9168
Minor: Fix e2e count mismatch (#14352)
* Minor: Fix e2e count mismatch

* fix dbt test

* fix pyformat

* use > instead of ==
2023-12-12 22:32:13 +05:30
Teddy
20bc238413
FIX #14248 - Implement test case resolution task workflow (#14323)
* feat: add test case resolution task workflow

* fix: conflict with main
2023-12-12 10:13:11 +01:00
Onkar Ravgan
7f6311ec4c
Fixed entity req left check (#14341) 2023-12-12 10:57:09 +05:30
Teddy
a4f9b0ec4e
fix: renamed API endpoints from resolution to status (#14330) 2023-12-11 14:09:40 +01:00
Onkar Ravgan
e4bfd12811
fixed e2e test after patch and collate demo (#14321) 2023-12-08 19:55:49 +05:30
Lucas Garcia
fe06b5cbb2
#14235: adding dialect based on connection type to LineageParser (#14249)
* Fix #14235: adding dialect based on connection type to LineageParser

* Fix: formating changes

* Update ingestion/src/metadata/ingestion/source/dashboard/metabase/metadata.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* style: fix indentation errors

* Fix pytest

---------

Co-authored-by: LucasGarcia07 <lucas.junqueira@hurb.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-12-08 19:49:59 +05:30
NiharDoshi99
8d925c46a5
#13696: add support for dot in schema name to fetch tables (#14246) 2023-12-08 12:04:28 +05:30
Onkar Ravgan
cb94243370
fix desc and tags update from patch (#14305) 2023-12-07 18:39:38 +05:30
Onkar Ravgan
4ade6fd39f
Remove col displayname from patch req (#14279) 2023-12-06 10:12:04 -08:00
Mayur Singal
d6a01ca5b1
Minor: Remove retension period from patch request (#14276)
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-12-06 20:29:44 +05:30
Akash Jain
6eefd51180
fix: AIRFLOW_DB_PROPERTIES default value in docker compose files (#14274) 2023-12-06 19:56:19 +05:30
Onkar Ravgan
c7e32c8abf
Fixes: 14240 and Fixes 13365: Bulk patch descriptions from dbt (#13999) 2023-12-06 19:00:31 +05:30
chyueyi
070ff1f605
feat: add profile for doris connector (#14214) 2023-12-06 08:59:56 +05:30
Ayush Shah
b8d0b3b6e4
#14168: Mark SP and Data model deletion (#14219) 2023-12-06 08:57:41 +05:30
Onkar Ravgan
b3578cd496
Fix 14040: Part 2 Add Patch Request when updating the entities from create request (#14224) 2023-12-05 17:56:40 +05:30
Teddy
3bbf55fcda
FIXES #14049 - Split test case resolution status from test case result (#14204)
* refactor: entityFQN as ListFilter condition

* feat: implement resolution entity timeseries

* fix: rename to testCaseResolutionStatus

* ref: extracted ES query builder into private method

* ref: extract OS query builder in its own method

* ref: remove ingestion logic for test case resolution

* fix: reorganize json schemas to fix circular import in Python

* ref: object names in typescript code

* feat: added indexing of test case resolution

* feat: added test case resolution sample data

* fix: test case resolution api logic

* fix: audit logger for entityTimeSeriesInterface

* fix: DDL generation

* style: python linting

* fix: skip UI test case resolution tests

* fix: remove extension field

* fix: renamed testCaseFailureStatus to testCaseResolutionStatus

* fix: remove reviewer

* fix: rename sequenceId to stateId

* fix: re adjust search weights

* fix: removed InReview status

* style: ran python linting
2023-12-04 23:18:01 -08:00
Ashish Gupta
42463ff40b
#14134: supported retention period in table entity (#14163)
* supported retention period in table entity

* Add retention period updates

* supported unit test

* added curd operation for retention center

* minor changes

* fix modal issues and added validation

* added unit test for retention period

* fix code smell

* fix sonarcloud

* minor chnages

* Fix java code styling

* added hours in retention period

* changes as per comments

* fix sonar

* remove localization keys

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2023-12-05 10:42:37 +05:30
Teddy
f571f67eda
fix: DI skip on failure when listing entities (#14226) 2023-12-04 16:59:09 +00:00
Mayur Singal
5965d4c63b
Minor: Fix airflow empty status (#14216) 2023-12-04 07:54:13 -08:00
Mayur Singal
4843d85bf9
Minor: Fix kafka Topic Entity Import (#14227) 2023-12-04 21:08:33 +05:30
Mayur Singal
b57e6710b6
Minor: Fix 'str' object has no attribute 'serviceType' for SP Lineage (#14221) 2023-12-04 19:52:02 +05:30
Preet Shah
20cf8f63ab
Update ingestion_dependency.sh (#14139)
* Update ingestion_dependency.sh

* Add parameter for DB connection properties

---------

Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
2023-12-04 18:09:42 +05:30
Mayur Singal
389ae79d3c
#14115: Separate Unity Catalog From Databricks (#14138) 2023-12-04 11:22:46 +05:30
Harpreet
b50f6f8939
#14028: fix the sed command issue in ingestion dep script (#14029)
* fix the sed command issue in ingestion dep script

* remove sql_alchemy_conn from airflow.cfg and use env variable

* try execute ingestion dep script without source
2023-12-03 11:27:36 -08:00
Pere Miquel Brull
7fcdf08ca4
#11626 & #14131 - Lineage with other Entities & attr-based xlets (#14191)
* Add OMEntity model

* Test OMEntity

* Update repr

* Fix __str__

* Add entity ref map

* Test serializer for backend

* Fix tests

* Fix serializer

* Test runner

* Add runner tests

* Update docs

* Format
2023-12-01 06:29:44 +01:00
Onkar Ravgan
b403c2b250
Added SP lineage as edge (#14180) 2023-11-30 16:36:52 +05:30
Mayur Singal
2092a52048
Minor: Fix E2E Tests failure (#14171) 2023-11-29 12:30:37 -08:00
Teddy
8ff70e31fe
MINOR: fixes ingestion of sample data for custom metrics (#14170)
* fix: updated sample data ingestoon for custom metrics

* style: fix python linting

* style: fix java linting
2023-11-29 17:28:19 +01:00
Mayur Singal
49a5557e44
#14099: fix bigquery test connection & SP query (#14106)
* Fix #14099: fix bigquery test connection & sp query

* fix unquote condition

* pyformat
2023-11-28 14:02:46 +01:00
chyueyi
b6b337e09a
feat: add support for doris datasource (#14087)
* feat: add support for doris datasource

* fix: fix python style check

* fix: add pydoris dependency

* fix: add pydoris dependency

* fix: py_format_check

* fix: parse error when doris view column is VARCHAR(*), check data length if not digit then return 1

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-11-28 13:57:52 +05:30