2916 Commits

Author SHA1 Message Date
Mayur Singal
3b61938c6d
Minor: Fix Mysql E2E Count (#14631) 2024-01-09 11:28:43 +05:30
Pere Miquel Brull
24643a397a
#14492 - Fix Snowflake SP parsing with empty signature (#14623) 2024-01-08 11:16:35 -08:00
Pere Miquel Brull
e8eb2bda3c
MINOR - Hide AutoTagger, add MetaPilot search, and fix apps status calls (#14201)
* MINOR - Hide AutoTagger & MetaPilot search & Fix apps status calls

* Fix method call

* Fix method call

* Fix method call

* "additionalProperties": true

* Use EntityRef

* Use ES sink

* Update UI schema type
2024-01-08 14:57:07 +01:00
Pere Miquel Brull
b92fd5cde4
MINOR - Move test case incident to the top of the data_quality_data_time_series (#14600)
* Add field and index

* MINOR - Move test case incident to the top of the data_quality_data_time_series table

* Fix test

* Fix compile

* Format

* Update incidentId

* Rename field

* Fix patch
2024-01-08 07:42:15 +01:00
Onkar Ravgan
ecdb7b9f41
Fixes 14109 and 14325: Optimised Tableau Connector (#14548)
* Optimised tableau conn

* Added comment
2024-01-08 06:33:05 +01:00
Mayur Singal
c3903b3f45
Minor: Fix IBM DB2 Depenency Installation Issue (#14613) 2024-01-08 10:18:04 +05:30
Ayush Shah
d513dc4d4d
#11890: Add GCS Json schema changes (#14559)
* Issue-11890: Add GCS Json schema changes

* Modify Gcs Security config

* Add Test Def for gcs

* Add gcs for storage_metadata_config
2024-01-06 19:27:36 -08:00
Mayur Singal
a789fc86d6
Fix #13053: Remove Connection URI config MongoDB (#14584)
* Fix #13053: Remove Connection URI cofig MongoDB

* pyformat & test fixes
2024-01-05 10:51:12 -08:00
Mayur Singal
d82770a91e
Fix #14421: Airflow - Handle pipeline status when dag killed (#14562) 2024-01-05 18:14:26 +05:30
Pere Miquel Brull
0255171218
MINOR - Create Test Case Resolution ts entry & delete resolution when… (#14541)
* MINOR - Create Test Case Resolution ts entry & delete resolution when Test Case is deleted
2024-01-05 09:15:49 +01:00
Mayur Singal
bf74ef9e22
Minor: Filter Create SP & Declare Queries for MSSQL Lineage (#14586) 2024-01-05 12:13:54 +05:30
Onkar Ravgan
7fdff15648
fixed dbt runresults ts (#14572) 2024-01-04 12:36:23 +05:30
Onkar Ravgan
a892267fbb
Fixed salesforce table req (#14570) 2024-01-04 10:45:58 +05:30
Mayur Singal
d197164125
Fix #14176: Add AzureSQL dialect in lineage (#14563) 2024-01-03 08:11:05 -08:00
NiharDoshi99
78bd8f98b5
#10203 delete temp creds files (#14536) 2024-01-02 13:09:15 +05:30
Cristian Calugaru
1631172528
Issue 10587 - Storage Service doesn't handle deleted containers (#14497)
* global manifest option for storage services

* added a no metadata config source option for global manifest s3 services option

* merge fixes

* more merge fixes.

* black stuff

* test fixes

* formatting

* storage service container automatic deletion support

* reverted auto-modified files.

* styling

* linting
2023-12-30 13:27:16 -08:00
mgorsk1
5ae83b284a
Fixes 14528: skip statuses for tasks that are no longer present in Airflow DAG (#14529)
* 🐛 Fix attribute name

* import AirflowTask model

* make py_format

* make py_format_check
2023-12-30 13:26:34 -08:00
Pere Miquel Brull
f4bbca3f72
MINOR - Clean topology & add tests (#14527)
* Clean topo

* Format

* Add tests

* Fix tests

* Merge main
2023-12-29 17:00:59 +01:00
Pere Miquel Brull
3613d0b18a
MINOR - Airflow Lineage Backend to verify the fingerprint (#14530)
* hash

* MINOR - Airflow Lineage Backend to verify the fingerprint
2023-12-29 16:36:28 +01:00
Takahiro Nakayama
7d05d9f50a
Fixes: GetTags-403 Permission denied on resource project None. (#14521)
* Fix "GetTags-403 Permission denied on resource project None."

Signed-off-by: Takahiro Nakayama <civitaspo@gmail.com>
2023-12-29 15:09:59 +05:30
Pere Miquel Brull
1c2219517e
#13649 - Add Pipeline Status Skipped (#14524)
* #13649 - Add Pipeline Status Skipped

* Format
2023-12-29 09:41:59 +01:00
Pere Miquel Brull
4b6034c527
#13638 - Filter task instance by removed state (#14515)
* Clean name logging and filter out removed tasks

* Update client logging

* Update client logging
2023-12-29 06:56:44 +01:00
Pere Miquel Brull
b84ce33b80
#11799 - Fix Airfow ownership & add pipeline tasks (#14510)
* Fix airflow owner and add tasks

* Add pipeline tasks ownership

* MINOR - Fix py CI

* Add pipeline tasks ownership

* Add pipeline tasks ownership

* MINOR - Fix py CI

* MINOR - Fix py CI

* Add pipeline tasks ownership

* patch team

* patch team

* Format
2023-12-28 10:25:00 -08:00
Mayur Singal
0c14a73fd1
Minor: Handle Column Lineage Exceptions (#14505)
* Minor: Handle Column Lineage Exceptions

* py format

* pylint
2023-12-27 13:04:10 -08:00
Teddy
61ef55290e
MINOR - generic profiler optimization for sampling and BQ (#14507)
* fix: limit sampling to specific column

* fix: handle bigquery struct columns

* fix: default partition to 1 DAY for BQ

* fix: default to __TABLES__ for BQ table metrics

* style: ran python linting

* style: fix linting

* fix: python style

* fix: set partition to DAY if not HOUR
2023-12-27 19:13:44 +01:00
Teddy
3dc642989c
Fixes #7729 - Add logic to compute passed/failed rows (#14472)
* feat: add test case resolution task workflow

* chore: add migration for test case resolution feature

* fix: removed required field for object compatibiity in older migrations

* fix: minor testCaseResolution status logic

* chore: revert migration for test case incident

* chore: update migration file

* style: renamed variables

* feat: added logic to compute failed/passed rows

* feat: add support for row level computation in schema

* chore: add test definition migration

* feat: add logic to explicitly compute row level failure

* chore: clean up code

* style: fix java

* style: fix pyton format

* fix: unhidde API for incident manager

* style: fix java styling
2023-12-27 13:38:51 +01:00
Saurabh Yadav
0691c96eaf
FIxes 13543: Added Duckdb Ingestion Connector (#14468)
* changes

* Added duck db ingestion connector
2023-12-25 18:07:09 -08:00
Shilpa Vernekar
0dfa1e2de1
Broken links corrected (#14489) 2023-12-22 13:32:47 -08:00
Pere Miquel Brull
1339d6518f
MINOR - Fix py CI (#14488) 2023-12-22 16:27:21 +01:00
Pere Miquel Brull
b786064bc2
#11857 - Store workflow status in the Ingestion Pipeline Status (#14462)
* Register StackTraceError in spec

* Register StackTraceError in spec

* Register StackTraceError in spec

* Add todos

* Update status

* docs

* format

* Fix tests

* Fix tests

* Fix tests

* Ignore generated

* Fix tests

* Fix tests

* Tests

* Try constants

* Try constants

* Print

* Print

* Print

* order

* Fix service name

* fix ui error

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-22 15:43:50 +01:00
Pere Miquel Brull
7e8e4a7e68
#14320 - FIx Airflow Callback datetime conversion (#14487)
* #14320 - don't cast twice

* #14320 - Fix Airflow Callback datetime conversion

* import
2023-12-22 15:43:41 +01:00
Onkar Ravgan
79444f4a24
Fix 13823: Validate and Parse gracefully IngestionPipeline (#14461)
* fixed ingestion pipeline parsing

* Added validation for automation workflow

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-12-22 09:56:39 +05:30
Aldwyn Cabarrubias
0e4f80899e
Update profiler_interface.py (#14287)
Transpose row=None before try statement in compute_metrics_in_thread function

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-12-21 10:37:59 +01:00
Ayush Shah
ebc0a551e5
Fixes 12947: Add Support For DQ and Profiler in Databricks Unity Catalog (#14424) 2023-12-20 21:18:05 +05:30
NiharDoshi99
492092720d
#12908 support for table owner in postgres (#14289)
* support pick up postgres table owner

* added owner wrapper

* optimize the code

* remove pylint error

* minor fix

* changes as per comments

* fix python checksytyle and comments

* moved process owner to database service

* update docs for version 1.3

---------

Co-authored-by: Vanshika Kabra <vanshikakabra@Vanshikas-MacBook-Pro.local>
2023-12-20 18:42:59 +05:30
Onkar Ravgan
7d032977d7
Fix 14084: Removed Airflow Mssql backend connection (#14420)
* Removed Airflow Mssql backend conn

* added breaking changes docs
2023-12-20 12:28:28 +05:30
Onkar Ravgan
841fd9cfe7
Fixed superset excp (#14447) 2023-12-20 09:30:37 +05:30
Pere Miquel Brull
e16ad3c5dc
MINOR - Clean ingestion ES code and auth providers (#14358)
* MINOR - Clean ingestion ES code and auth providers

* clean parser

* Clean security config for the client

* Clean security config for the client

* Improve class conversion exceptions

* Fix tests

* Clean up java client

* Clean up java client

* clean parser

* Fix test

* Fix test

* fix NO_AUTH error

* Fix test

* Format

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-19 12:08:48 +01:00
Pere Miquel Brull
0e92a975e3
#14425 - Create ingestion-base-slim image (#14426)
* #14425 - Create ingestion-base-slim image

* Format

* Bump airflow

* Bump constraints
2023-12-19 11:09:38 +01:00
Pere Miquel Brull
a83a5ba3a3
MINOR - Skip delta tests for 3.11 (#14398)
* MINOR - Bump delta for 3.11

* Update flags

* MINOR - Bump delta for 3.11

* Update tests regex

* Update version

* Deprecations

* Format

* Version

* Try delta spark

* Skip delta tests for 3.11

* Update ingestion/tests/unit/topology/pipeline/test_airflow.py
2023-12-18 17:01:57 +01:00
Onkar Ravgan
ef48b7eae7
MINOR: Added custom properties methods in python sdk (#14402)
* Added custom properties methods in python sdk

* fixed tests
2023-12-18 12:57:14 +05:30
Pere Miquel Brull
d8984d267e
#14340 & #13849 - Clean secret ID and improve encrypt/decrypt exception management (#14356)
* Fix supported characters in SM

* Update SM

* Fixes

* Fixes

* Improve class conversion exceptions

* Comments

* Rename noop to db secrets manager providee

* Update sm

* Fix

* db SM

* db SM

* Fix test

* UI

* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts

* update default
2023-12-18 06:43:20 +01:00
Mayur Singal
3e1bef48c7
Minor: Fix casing in MSSQL test connection query (#14412)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-12-17 16:02:32 +01:00
Pere Miquel Brull
6d046f7e3b
Revert "Update Dockerfile.ci file to use new MySQL Keys (#14400)" (#14416)
This reverts commit 33f2fdd49cae50c93cbf1f8544402bda41070a01.
2023-12-16 16:46:33 +01:00
Tom Bushell
bb3f9a4441
Fixes #9919: Cleaned up postgres lineage and fixed function in postgres query_parser (#13678)
* Replaced s.query with query_text in filters for clarity

* Removed redundant _iter function

* Updated process_table_query to return None when there are no queries to process

* Modified process_table_query to yield None if there are no queries

* Resolved formating issues and modified process_table_query to return None instead of yielding None

* pylint, fix bugs and improve code

* fix import

* pyformat

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-12-15 20:14:19 +05:30
Preet Shah
33f2fdd49c
Update Dockerfile.ci file to use new MySQL Keys (#14400)
* Update Dockerfile.ci file to use new MySQL Keys

* Update ingestion Dockerfiles files to use new MySQL Keys

* Update ingestion Dockerfiles files

* Update Dockerfiles to not hardcode the new keys

* Update comments
2023-12-15 15:46:30 +05:30
Pere Miquel Brull
eaacc693bd
#12027 - Add support for Python 3.11 (#14385)
* Fix datamodel codegen and bump versions

* Add 3.11 tests

* Update hive

* pandas

* pandas
2023-12-14 15:46:58 +01:00
Pere Miquel Brull
d4d8f3655f
#14043 - Fix metadata backup memory profile (#14378)
* Update looker docs

* Fix db dump
2023-12-14 15:46:35 +01:00
C. Kris
74df616679
Fixes #14215: Add missing decode stage to gz/zip files in json ingestion reader. (#14375)
* add decoding stage to gz/zip files.

Files that where zip/gz where not being decoded.
This was leading to a error when we wanted them to be.

* remove unnecessary comment

---------

Co-authored-by: Carl Kristensen <carl.johan.coelho.kristensen@schibsted.com>
2023-12-14 12:47:58 +01:00
Mayur Singal
4f65287dae
Minor: Airflow skip task status if task deleted (#14373) 2023-12-13 18:34:14 +05:30