2916 Commits

Author SHA1 Message Date
Pere Miquel Brull
203e2b5fff
Simplify Pipeline Service Client Response and prep test connection for ingestion base (#10334)
* Simplify Response and prep test connection

* Format
2023-02-25 15:52:14 -08:00
Cristian Calugaru
bbf54afaf5
sample data for object store service and containers (#10336)
* sample data for object store service and containers

* formatting

* linting
2023-02-25 14:43:38 -08:00
Austin Witt
2d65cf02b0
[fixed] profiler would not fetch serivceConnection from OpenMetadata server when serviceConnection was omitted from workflow YAML (#10335) 2023-02-25 10:11:49 -08:00
Onkar Ravgan
21a30af25c
Added uri in jsonchema for aws credentials (#10315)
* Added uri in jsonchema

* fixed sample data
2023-02-24 22:17:38 +05:30
Milan Bariya
c1a8553e07
Airflow API change for test connection (#10182)
* Airflow API change for test connection

* Improve logic

* Pydantic change

* Improve logic
2023-02-24 11:43:11 +05:30
Onkar Ravgan
8191f92438
Added dbt oracle fix (#10282)
* Added dbt oracle fix

* Addressed review comments

* Added unit tests
2023-02-23 19:17:54 +05:30
Mayur Singal
0fafbf70ba
Support postgres multiple DB ingestion for user with no access (#10207) 2023-02-23 15:42:40 +05:30
Mayur Singal
9dbfea2fca
Add Database & Schema Description: Snowflake (#10276) 2023-02-23 15:40:48 +05:30
Mayur Singal
50af4990e2
Fix Clickhouse Types (#10295) 2023-02-23 09:06:15 +00:00
Onkar Ravgan
e9f1cb1074
Added dbt unit test (#10047)
* Added dbt unit test

* fixed typo

* Added dbt manifest schema tests

* Addressed review comments
2023-02-22 16:09:37 +01:00
Mayur Singal
cd4461397d
Add impyla as scheme for hive connector (#10270) 2023-02-22 16:54:56 +05:30
Teddy
83be5d933b
Fixes #9301 - Refactor TestSuite and Remove Pandas from Base Requirements (#10244)
* feat(testSuite): extracted out column test for SQA type

* refactor(testSuite): extracted SQA column and table tests into their own classes

* refactor(testSuite): Added pkutil namespace package style for test suite classes

* refactor(testSuite): added dynamic importer function for test cases

* refactor(testSuite): black formatting

* refactor(testSuite): fixed linting issues

* refactor(testSuite): refactor metrics for dataframe

* refactor(testSuite): Added Mixins and base methods

* refactor(testSuite): extrcated out get bound for floats

* refactor(testSuite): Added pandas column test cases

* refactor(testSuite): Deleted old column tests

* refactor(testSuite): Added table tests for datalake

* refactor(testSuite): Removed old tests definition

* refactor(testSuite): changed registry to dynamic class inport

* refactor(testSuite): renamed dl_fn to df_fn

* refactor(testSuite): updated registry unit test

* refactor(testSuite): updated import path to sqa like column

* refactor(testSuite): cleaned up imports in old files

* refactor(testSuite): harmonzied SQALikeColumn object to replicate SQA Column object

* refactor(testSuite): linting

* refactor(testSuite): linting

* refactor(testSuite): raise expection on DQ exception

* refactor(testSuite): linting

* refactor(testSuite): removed pandas from base requirements

* refactor(testSuite): Added __futur__ for py3.7 type hint

* refactor(testSuite): added `df` to good-names

* refactor(testSuite): renamed Handler to Validator

* refactor(testSuite): Added test inheritance for column tests

* refactor(testSuite): cleaned up column type check

* refactor(testSuite): cleaned up typo

* refactor(testSuite): extracted main table test logic into parent class

* refactor(testSuite): linting

* refactor(testSuite): linting fixes

* refactor(testSuite): address doc string and linting issues
2023-02-22 09:42:34 +01:00
Abhishek Pandey
c678b54b05
issue-9734: job-id-param-added-for-dbt-cloud-ingestion (#10248)
UI changes will be handled with https://github.com/open-metadata/OpenMetadata/issues/10269
2023-02-22 12:18:03 +05:30
Mayur Singal
42a8622981
Fix #10215: Fix Query Log Usage (#10232) 2023-02-21 17:41:22 +05:30
Teddy
219adbe9d9
fix(profiler): snowflake system metric (#10246) 2023-02-21 07:12:40 +00:00
Ayush Shah
785142d86a
Add policy tags from Bigquery (#10189) 2023-02-20 19:13:45 +00:00
Abhishek Pandey
f53d5349ef
null-type-column-handling-added (#10234) 2023-02-20 20:07:47 +05:30
VolkovGeoPhy
7a59bc7676
>= grpc-tools 1.47.2 (Done) (#10218) 2023-02-20 18:07:27 +05:30
Stéphane Sol
7ae091318e
limit dbt runs to successful state (#10209)
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-02-19 03:19:01 +05:30
Onkar Ravgan
a5d51d0f55
Fixed entity reference for dbt and glue (#10201)
* fixed entity ref for dbt

* fixed entity ref for glue
2023-02-14 17:28:28 +05:30
Suresh Srinivas
afad0a4769
Fixes #10123 - Change entityReference in createRequests to fullyQualifiedName (#10124)
* Change entityReference to entity name or fullyQualifiedName

* Change backend code and tests to use FQN

* UI change for using fqns instead of EntityReference

* Ingestion framework changes for using fqns instead of EntityReference

* Fix test failures

* Fixed python tests and sample data new

* fix: minor ui changes for fqn

* Fixed python integration tests

* Fixed superset tests

* fix UI tests

* fix type issue

* fix cypress

* fix name for testcase

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-02-13 13:38:55 +05:30
Mayur Singal
610b6535fa
Fix unused dbName for non SQA sources (#10170) 2023-02-10 07:17:09 +01:00
Mayur Singal
0bea02c202
Superset Unit Tests (#10078) 2023-02-09 19:45:24 +05:30
NiharDoshi99
9bb99407e5
fix: snowflake frequently join table (#10157) 2023-02-09 18:05:52 +05:30
Milan Bariya
6424b57f51
Update test connection function for common source (#10116)
* Update test connection function for common source

* Update test connection function for common source

* Update test connection function

* Update test connection function

* Update test connection function

* Change based on comments
2023-02-09 07:03:48 +01:00
Mayur Singal
392107bc4a
Datalake Avro & Json Lines Support (#10129) 2023-02-08 17:31:25 +00:00
NiharDoshi99
34a0cc147e
Fix: Added changes for Pii sensitive (#10119)
* Fix: added changes for pii sensitive

* Fix: removed comments

* Fix: python checkstyle

* differtiate between sensitive and non sensitive tag

* fix: python test

* fix: added tests

* fix: maven CI
2023-02-08 16:00:47 +00:00
Teddy
b62f97f27b
fix(profiler): imporve logic for snowflake to pick up DML ran on different WH (#10150) 2023-02-08 14:53:43 +00:00
Onkar Ravgan
6d94a03fc7
Added dbt fixes (#10132) 2023-02-08 12:42:19 +01:00
NiharDoshi99
012f96a7a2
Domo issue (#10080)
* Domo issue

* added pydantic models and changes as per comment

* Fix: added ownership details

* fixed pytest
2023-02-07 17:16:26 +05:30
Pere Miquel Brull
fb15c896b3
Handle XLets in groups for AirflowLineageRunner (#10114)
* Handle XLets in groups

* Linting

* Linting
2023-02-07 06:49:46 +01:00
Pere Miquel Brull
1f11fdefb4
Databricks comments & view definitions in bulk (#10112)
* Add databricks comments

* Add view definition

* Add view definition
2023-02-06 08:55:15 +01:00
Suresh Srinivas
1990ac56d9
10041 Part1 - Refactor and clean up System APIs (#10042)
* 10041 Part1 - Refactor and clean up System APIs

* Update the config resource endpoint url
2023-02-03 16:25:40 -08:00
Onkar Ravgan
67479940cf
Use es for fqn in dbt datamodel (#10102)
* Use es for fqn in datamodel

* Corrected error message
2023-02-03 19:43:03 +05:30
Mayur Singal
01cad0f811
Add json zip support for datalake (#10100) 2023-02-03 13:03:55 +00:00
Mayur Singal
4d181ba8dc
Tableau Owner Fix (#10101) 2023-02-03 17:20:11 +05:30
Teddy
0a77f8a3f2
Fixes issue #10082 -- Unsupported field types for bigquery (#10091)
* fix(profile): Raise nicer error for BQ unsupported interval type for timestamp

* fix(profiler): run checkstyle
2023-02-02 19:00:56 +01:00
Teddy
5e1e975e0e
Fixes #10018 - Limit user fields to team (#10050)
* fix(dataInsight): Limit field for user request

* fix(dataInsight): revert changes added for testing
2023-02-02 15:00:55 +01:00
Nahuel
b9a3c06104
Bump main branch to version 1.0.0 (#10040)
* Bump to version 0.13.2

* Bump mvn projects to 1.0.0-SNAPSHOT

* Bump python projects to 1.0.0.dev0

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-02-02 12:56:14 +01:00
Teddy
4f707f8531
fix(pylint): adjust code to conform with pylint 2.16 release (#10055) 2023-02-01 18:03:56 +01:00
Pere Miquel Brull
f0f3f0be6a
Add looker unit tests (#9691)
* Add looker tests

* Empty-Commit

* Install GE for tests

* Fix usage details python name

* Add missing test requirement
2023-02-01 09:20:26 +00:00
Ayush Shah
747fcf569b
Add docs - quicksight, lineage... (#10023) 2023-01-31 15:17:40 +00:00
Teddy
ba08302ea1
Issue #7291 - Implements Table Rows Inserted to be Between test (#9813)
* staging commit

* staging commit

* refactor: partitioning logic

* refactor (tests): move to parametrized tests for test validations

* refactor: local variables into global

* (feat): Added logic for table row inserted test

* (feat): fix python checkstyle

* feature: extracted get_query_filter logic into its own function
2023-01-31 15:57:51 +01:00
Onkar Ravgan
b754121425
Added dbt Manifest V8 Compatibility (#10024)
* Added v8 compatibility

* Added condition
2023-01-31 14:06:06 +05:30
Teddy
0b3996d2fa
Remove profiler support for dynamodb (#10014)
* fix(DynamoDB): remove profiler support and cleaned up documentation

* fix(DynamoDB): remove supportProfiler filed in tests
2023-01-31 08:29:35 +01:00
Nahuel
10b15e782e
Fix: error getting 'certificatePath' from SSLConfig object (#10017) 2023-01-30 18:38:13 +00:00
Mayur Singal
8c7c21bd54
Fix Superset Owners & Descriptions (#10007) 2023-01-30 14:40:52 +00:00
Pere Miquel Brull
bd0e307912
Fix mlmodel client (#9997) 2023-01-30 11:41:39 +01:00
Ayush Shah
725542b607
Fix Glue PIpeline (#9991) 2023-01-30 06:13:01 +00:00
Onkar Ravgan
949989fb1c
Added dbt parser (#9982)
* Added dbt parser

* Added library dependency

* format and final fixes

* Addressed review comments

* Fixed typo
2023-01-29 20:47:39 +01:00