2362 Commits

Author SHA1 Message Date
Nahuel
b94927b3ea
Bug: fix flaky timeout test (#10509) 2023-03-10 10:55:56 +01:00
NiharDoshi99
34bf175ca1
adding quotes around alias (#10506) 2023-03-10 15:18:58 +05:30
NiharDoshi99
9b0b06fe25
Added flag for overriding owners (#10425)
* added flag for overriding owners

* docs update

* adding owners in patch call

* adding owners in patch call

* fix: python test case

* changes as per comment

* changes as per comment

* ui changes
2023-03-10 15:01:56 +05:30
Teddy
cea10e8301
fix: handle None owner (#10497) 2023-03-10 08:12:14 +00:00
Milan Bariya
bc4abc44ed
Add steps in test_connection function (#10408)
* Add steps in test_connection funstion

* Add steps in test_connection funstion

* Add steps in test_connection funstion

* Fix: Pylint issue

* Status response changefor statuscode 200

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-10 10:02:26 +05:30
NiharDoshi99
ddbc0311e8
fix: mark table deleted dependencies (#10476) 2023-03-09 09:15:05 +00:00
Suresh Srinivas
4c6d184ef5
Fixes #10480 Glossary rename results in rename of Classification with… (#10486)
* Fixes #10480 Glossary rename results in rename of Classification with the same name

* Rename TagSource Tag to Classification
2023-03-09 00:30:36 -08:00
Nahuel
ed884cf79a
Bug: Update sqllineage-openmetadata + add timeout for parsing queries (#10474)
* Update sqllineage-openmetadata version + add timeout

* Pyimpala fix colnames, comments and dialect sql compilation (#10470)

* Fix col names and comments for impala hive

* Fix cols, comments and impala sql compilation

* Handle hive types

* Format

* Added doc in avro array and tests (#10473)

* Fixed: Add job definition id field for dbt cloud in UI #10269 (#10472)

* fixed Add job definition id field for dbt cloud in UI #10269

* sync-localization file

* fixed failing unit test and add unit test for the changes

* Address PR comments

* Update tests

* Pylint clean

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-03-08 20:49:02 +01:00
Teddy
56f7ac2c2c
Fix few profiler bugs (#10465)
* fix: handle overflow in profiler + fix minor bug from new metrics

* fix: flaky test from profiler

* fix: fix code smell
2023-03-08 18:01:25 +01:00
Onkar Ravgan
4d11db4220
Added doc in avro array and tests (#10473) 2023-03-08 20:16:50 +05:30
Pere Miquel Brull
a05e56feba
Pyimpala fix colnames, comments and dialect sql compilation (#10470)
* Fix col names and comments for impala hive

* Fix cols, comments and impala sql compilation

* Handle hive types

* Format
2023-03-08 14:13:06 +01:00
Mayur Singal
c199f13ed0
Fix #10402: Add support for AssumeRole for AWS (#10417) 2023-03-08 15:43:33 +05:30
NiharDoshi99
d41878ec90
fix: spacy model (#10467) 2023-03-08 15:19:35 +05:30
Onkar Ravgan
ca623d0693
dbt performance improvement (#10366)
* dbt performance improvement

* reduced es calls

* Added unit tests
2023-03-08 06:37:34 +01:00
Sriharsha Chintalapani
fe73948b55
Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf (#10430)
* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Added top level parsing and unit tests

* fix(ui): show schemaText and fields both

* fix no data placeholder for fields & schema text

* addressing comments

* fixed py checkstyle

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-07 15:40:04 +01:00
Sriharsha Chintalapani
de10cd254b
New docker structure (#10438)
* Adding the different docker-compose file openmetadata and ingestion

* Added two different env files for mysql and postgres

* Updated the docker file path

* Updated the path of docker folder structure

* Fix docker

* Updating the PR with necessary changes required

---------

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-03-06 08:58:53 -08:00
Nahuel
f2e1a87b5a
Fix#10377: service connection not overwritten as expected (#10445) 2023-03-06 16:32:10 +01:00
Schlameel
fb7b12842b
#9544: Added patch owner to Python SDK. Includes tests. (#10403)
Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-06 14:32:58 +00:00
Teddy
8bc0472fae
Handle null window metric (#10440)
* fix: handle null window metrics

* fix: remove else from try/except pattern
2023-03-06 14:09:02 +00:00
Nahuel
ef1812a09d
Fix: Stop displaying authorization values in debug logs (#10443) 2023-03-06 14:56:29 +01:00
Pere Miquel Brull
477a5223eb
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
2023-03-06 14:44:16 +01:00
Mayur Singal
f7a7a16cdb
Fix hive get table issue with impala (#10424)
* Fix hive get table issue with impala

* Fix Lint
2023-03-06 11:29:54 +01:00
NiharDoshi99
895e7d2706
fix: scoped import for presidio (#10439) 2023-03-06 14:26:52 +05:30
Teddy
5208b6f684
Fixes #4368 - Add Histogram Metric (#10422) 2023-03-03 21:56:32 +01:00
Nahuel
247016307d
Fix#8648: Mask sensitive info from API responses (#10307)
* Mask sensitive info from API responses

* Rename converter classes

* Add missing Java classes from JSON schemas and class converters

* Update test service connection schema

* Update datalakeConnection JSON schema and fix some tests

* Fix AlertsRuleEvaluatorResourceTest and minor error in run_local_docker.sh

* Fix Pipeline and Database service tests

* Minor refactor

* Fix CsvUtilTest

* Fix EventMonitorFactoryTest

* Fix CloudWatchEventMonitorTest

* Update datalake metadata

* Update bigquery metadata

* Fix test connection functionality

* Fix OMeta service api test

* Update gcsValues title and revert changes in GH actions

* Mask sensitive enabled by default for local docker

* Add missing tests

* Address PR comments

* Address PR comments

* fix ui breaks on gcsValues.json

* Address PR comments

* Minor refactor

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-03 18:10:01 +00:00
NiharDoshi99
1ff76f5e65
pii tagging using spacy (#10256)
* WIP: pii tagging using spacy

* added test cases and changes as per comment

* fix python checkstyle

* fix python checkstyle

* added score, test_cases and docs update

* solved merge conflict

* fix python checkstyle

* remove pii tagging using regex

* fix python test

* lib changes and added some test case

* changed as per comment

* fix: python test

* fix: changes to get source_config

* fix: changes as per comment
2023-03-03 18:33:18 +05:30
Teddy
775ca75e87
fix #10173 handle cases where entity would be deleted from OM (#10364)
* fix(dataInsight): handle cases where entity would be deleted from OM

* Update ingestion/src/metadata/data_insight/processor/web_analytic_report_data_processor.py

Added explanation in code comments

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

* fix(dataInsight): tests failure

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-03-03 12:30:15 +01:00
Mayur Singal
92fee5cf4f
Fix #10292: Remove DB & Schema Usage Filters (#10378)
* Fix #10292: Remove DB & Schema Usage Filters

* Fix Lint

* lint
2023-03-03 06:39:55 +01:00
Milan Bariya
7faec7880a
Update test connection (#10317)
* Update test connection

* Update test connection

* Update test connection with Tags

* Change based on comments

* Fix Pytest

* Service connectin arg in all connecter

* Fix pytest and conflict
2023-03-02 14:22:38 +05:30
Teddy
6a4df5f460
fix: removed pandas dependencies for non pandas profiler/testSuite workflows (#10380) 2023-03-01 16:38:50 +01:00
Kevin R
1c85f1b7fb
fix: Use endpointURL and region in s3_fs (#10305) 2023-03-01 14:05:24 +01:00
Mayur Singal
a11f7d7e37
Fix #8981: Move Connection Options and Connection Args to main properties (#10301) 2023-03-01 17:12:28 +05:30
Teddy
754074f1be
Fixes #7758 - Added Column value and Integer Range Partitionning (#10350)
* feat(profiler): renamed  module to

* feat(profiler): added dbt-artifacts-parser to test setup.py

* feat(profiler): refactor workflow and interface

* feat(profiler): linting

* feat(profiler): removed old profiler modules

* feat(profiler): added support for value and integer range partition

* feat(profiler): fixed linting

* feat(profiler): added partitionning support for datalake profiler

* feat(profiler): removed `ProfilerInterfaceArgs` class

* feat(profiler): address comments

* feat(profiler): Added `OTHER` as an `IntervalType` for UI type generation
2023-03-01 08:20:38 +01:00
Mayur Singal
f6ba024b5f
Fix #10263: Fetch Vertica Schema Comments (#10363) 2023-03-01 11:35:01 +05:30
Mayur Singal
55a28ffd40
Trino Optimize table comments (#10367) 2023-03-01 10:39:28 +05:30
Mayur Singal
909cedeca1
Support Snowflake Geometary Type (#10359) 2023-02-28 18:59:01 +05:30
Sriharsha Chintalapani
d21fcad203
Fix patch tag for columns (#10191)
* Fix patch tag for columns

* Fix patch tag for columns

* fix pylint issues

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-02-26 15:33:47 +01:00
Cristian Calugaru
bbf54afaf5
sample data for object store service and containers (#10336)
* sample data for object store service and containers

* formatting

* linting
2023-02-25 14:43:38 -08:00
Austin Witt
2d65cf02b0
[fixed] profiler would not fetch serivceConnection from OpenMetadata server when serviceConnection was omitted from workflow YAML (#10335) 2023-02-25 10:11:49 -08:00
Milan Bariya
c1a8553e07
Airflow API change for test connection (#10182)
* Airflow API change for test connection

* Improve logic

* Pydantic change

* Improve logic
2023-02-24 11:43:11 +05:30
Onkar Ravgan
8191f92438
Added dbt oracle fix (#10282)
* Added dbt oracle fix

* Addressed review comments

* Added unit tests
2023-02-23 19:17:54 +05:30
Mayur Singal
0fafbf70ba
Support postgres multiple DB ingestion for user with no access (#10207) 2023-02-23 15:42:40 +05:30
Mayur Singal
9dbfea2fca
Add Database & Schema Description: Snowflake (#10276) 2023-02-23 15:40:48 +05:30
Mayur Singal
50af4990e2
Fix Clickhouse Types (#10295) 2023-02-23 09:06:15 +00:00
Mayur Singal
cd4461397d
Add impyla as scheme for hive connector (#10270) 2023-02-22 16:54:56 +05:30
Teddy
83be5d933b
Fixes #9301 - Refactor TestSuite and Remove Pandas from Base Requirements (#10244)
* feat(testSuite): extracted out column test for SQA type

* refactor(testSuite): extracted SQA column and table tests into their own classes

* refactor(testSuite): Added pkutil namespace package style for test suite classes

* refactor(testSuite): added dynamic importer function for test cases

* refactor(testSuite): black formatting

* refactor(testSuite): fixed linting issues

* refactor(testSuite): refactor metrics for dataframe

* refactor(testSuite): Added Mixins and base methods

* refactor(testSuite): extrcated out get bound for floats

* refactor(testSuite): Added pandas column test cases

* refactor(testSuite): Deleted old column tests

* refactor(testSuite): Added table tests for datalake

* refactor(testSuite): Removed old tests definition

* refactor(testSuite): changed registry to dynamic class inport

* refactor(testSuite): renamed dl_fn to df_fn

* refactor(testSuite): updated registry unit test

* refactor(testSuite): updated import path to sqa like column

* refactor(testSuite): cleaned up imports in old files

* refactor(testSuite): harmonzied SQALikeColumn object to replicate SQA Column object

* refactor(testSuite): linting

* refactor(testSuite): linting

* refactor(testSuite): raise expection on DQ exception

* refactor(testSuite): linting

* refactor(testSuite): removed pandas from base requirements

* refactor(testSuite): Added __futur__ for py3.7 type hint

* refactor(testSuite): added `df` to good-names

* refactor(testSuite): renamed Handler to Validator

* refactor(testSuite): Added test inheritance for column tests

* refactor(testSuite): cleaned up column type check

* refactor(testSuite): cleaned up typo

* refactor(testSuite): extracted main table test logic into parent class

* refactor(testSuite): linting

* refactor(testSuite): linting fixes

* refactor(testSuite): address doc string and linting issues
2023-02-22 09:42:34 +01:00
Abhishek Pandey
c678b54b05
issue-9734: job-id-param-added-for-dbt-cloud-ingestion (#10248)
UI changes will be handled with https://github.com/open-metadata/OpenMetadata/issues/10269
2023-02-22 12:18:03 +05:30
Mayur Singal
42a8622981
Fix #10215: Fix Query Log Usage (#10232) 2023-02-21 17:41:22 +05:30
Teddy
219adbe9d9
fix(profiler): snowflake system metric (#10246) 2023-02-21 07:12:40 +00:00
Ayush Shah
785142d86a
Add policy tags from Bigquery (#10189) 2023-02-20 19:13:45 +00:00