1614 Commits

Author SHA1 Message Date
Teddy
9d858ede3a
fix: declaring self.data after calling self.token (#10634) 2023-03-16 17:28:02 +01:00
Milan Bariya
fadefff3e1
Fix databricks timeout issue (#10613)
* Fix databricks timeout issue

* Change based on comments

* Change timeOut word to connectionTimeout
2023-03-16 16:13:49 +01:00
Pere Miquel Brull
cd6ece1cf4
Fix oracle bulk view definitions and comments & improve metadata version (#10620)
* Add more info to client version

* lower oracle views and comments
2023-03-16 14:11:22 +01:00
Mohit Yadav
b982d3fe2b
Query as entity (#10449)
* added query as an entity

* changed name of the variables and methods

* Added Resource Descriptors

* testcase bug fix

* addressing comments

* added script for table query migration

* added script for table query migration postgresql

* bug fix

* db change for script test

* added current timestamp

* change db config from postgresql to mysql

* added extension to use fucntion gen_random_uuid()

* solving maven ci

* added queryUsage and change is migration script

* addressing comments

* addressing comments

* added queryUsage relation and testcase

* added api to insert queries in bulk

* .

* fix a test case which was failing due to latest changes

* Ingestion Changes for Query as Entity

* move query changes to latest sqls

* added tags and owner

* update PR for Query as Entity

* update type

* fixed pagination

* fix path param

* fix TestCases

* add validation criteria

* removed exisitng query apis

* checkstyle fix

* remove vote from put

* remove vote from put

* Query As Entity Ingestion Changes

* Remove unused func

* update Review Comments

* update Review Comments

* remove previous changes for Query and Update Tests

* moved Checksum to Query Util Class

* update python api

* fix python checkstyle

* Fixed Tests

* Fix pytest

* remove space changes

* remove space changes

* Fixed put_addFollowerDeleteEntity_200

* Fix usage ingestion

* Update Python SDK and tests

* pylint fix

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-15 20:55:30 -07:00
Milan Bariya
a51da74f86
Databricks Pileline code improve & Test connection update & Databricks issue solved (#10590)
* Databricks Pileline code imporove

* Change based on comments

* Change based on comments
2023-03-15 20:19:12 +05:30
NiharDoshi99
205e5c8c4f
Aligned airflow source with lineage (#10596)
* aligned airflow source with lineage

* fix python test

* changes as per comment
2023-03-15 19:29:24 +05:30
Sriharsha Chintalapani
b33587041d
Fix #5153: Add referred columns for foreign key constraint and sort_key, dist_key support (#10433)
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-03-15 18:55:51 +05:30
Onkar Ravgan
682626ffe8
Added exception handling in dashboard sources (#10573)
* added exception handling in dashboard

* updated process owner logic

* Added redash source for fixes
2023-03-15 08:30:03 +00:00
Ayush Shah
7e4ba4567b
Fix Quicksight URL (#10592) 2023-03-15 08:05:38 +00:00
Onkar Ravgan
93e554ae67
Fixed Redash Source Issues (#10570)
* Imporved redash source

* Added docs

* Addressed review comments
2023-03-14 23:00:49 +05:30
Teddy
2f4a92a17b
fix: exclude owner from page view traffic in DI (#10574)
* fix: exclude owner from page view traffic in DI

* fix: uncomment KPI creation in setup
2023-03-14 11:45:46 +00:00
NiharDoshi99
83a1858434
fix druid ingestion (#10553)
* fix druid ingestion

* changes as per comment

* changes as per comment
2023-03-14 14:47:00 +05:30
Nahuel
ffdd911785
Bug: TypeError when dashboards field is not present in a workspace response (#10568) 2023-03-14 08:58:53 +00:00
Ayush Shah
0f9c2c2164
Add Labels on DatabaseSchema Level (#10547) 2023-03-13 14:13:51 +00:00
Teddy
d03b06daf6
feat: Added logic to handle MERGE statement for bigquery (#10522) 2023-03-13 11:34:40 +01:00
NiharDoshi99
113fcc2956
added process owner (#10539)
* added process owner

* python checkstyle

* changes as per comment
2023-03-13 14:39:02 +05:30
Milan Bariya
178bc8ecc0
Fix: Databricks complex column type issue (#10469)
* Fix: Databricks complex column type issue

* Fix: Failed to fetch view definition

* Change based on comments

* Change based on comments

* Change based on comments
2023-03-13 10:09:16 +05:30
Teddy
7ec667808c
Fixes #10498 - Add support for nifi client certificate auth. (#10499)
* feat: Added nifi support for client cert auth

* feat: fix code linting

* feat: addressed comments for nifi connections

* feat: fixed linting
2023-03-10 11:09:40 +01:00
Nahuel
b94927b3ea
Bug: fix flaky timeout test (#10509) 2023-03-10 10:55:56 +01:00
NiharDoshi99
34bf175ca1
adding quotes around alias (#10506) 2023-03-10 15:18:58 +05:30
NiharDoshi99
9b0b06fe25
Added flag for overriding owners (#10425)
* added flag for overriding owners

* docs update

* adding owners in patch call

* adding owners in patch call

* fix: python test case

* changes as per comment

* changes as per comment

* ui changes
2023-03-10 15:01:56 +05:30
Teddy
cea10e8301
fix: handle None owner (#10497) 2023-03-10 08:12:14 +00:00
Milan Bariya
bc4abc44ed
Add steps in test_connection function (#10408)
* Add steps in test_connection funstion

* Add steps in test_connection funstion

* Add steps in test_connection funstion

* Fix: Pylint issue

* Status response changefor statuscode 200

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-10 10:02:26 +05:30
NiharDoshi99
ddbc0311e8
fix: mark table deleted dependencies (#10476) 2023-03-09 09:15:05 +00:00
Suresh Srinivas
4c6d184ef5
Fixes #10480 Glossary rename results in rename of Classification with… (#10486)
* Fixes #10480 Glossary rename results in rename of Classification with the same name

* Rename TagSource Tag to Classification
2023-03-09 00:30:36 -08:00
Nahuel
ed884cf79a
Bug: Update sqllineage-openmetadata + add timeout for parsing queries (#10474)
* Update sqllineage-openmetadata version + add timeout

* Pyimpala fix colnames, comments and dialect sql compilation (#10470)

* Fix col names and comments for impala hive

* Fix cols, comments and impala sql compilation

* Handle hive types

* Format

* Added doc in avro array and tests (#10473)

* Fixed: Add job definition id field for dbt cloud in UI #10269 (#10472)

* fixed Add job definition id field for dbt cloud in UI #10269

* sync-localization file

* fixed failing unit test and add unit test for the changes

* Address PR comments

* Update tests

* Pylint clean

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-03-08 20:49:02 +01:00
Teddy
56f7ac2c2c
Fix few profiler bugs (#10465)
* fix: handle overflow in profiler + fix minor bug from new metrics

* fix: flaky test from profiler

* fix: fix code smell
2023-03-08 18:01:25 +01:00
Onkar Ravgan
4d11db4220
Added doc in avro array and tests (#10473) 2023-03-08 20:16:50 +05:30
Pere Miquel Brull
a05e56feba
Pyimpala fix colnames, comments and dialect sql compilation (#10470)
* Fix col names and comments for impala hive

* Fix cols, comments and impala sql compilation

* Handle hive types

* Format
2023-03-08 14:13:06 +01:00
Mayur Singal
c199f13ed0
Fix #10402: Add support for AssumeRole for AWS (#10417) 2023-03-08 15:43:33 +05:30
NiharDoshi99
d41878ec90
fix: spacy model (#10467) 2023-03-08 15:19:35 +05:30
Onkar Ravgan
ca623d0693
dbt performance improvement (#10366)
* dbt performance improvement

* reduced es calls

* Added unit tests
2023-03-08 06:37:34 +01:00
Sriharsha Chintalapani
fe73948b55
Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf (#10430)
* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Added top level parsing and unit tests

* fix(ui): show schemaText and fields both

* fix no data placeholder for fields & schema text

* addressing comments

* fixed py checkstyle

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-07 15:40:04 +01:00
Sriharsha Chintalapani
de10cd254b
New docker structure (#10438)
* Adding the different docker-compose file openmetadata and ingestion

* Added two different env files for mysql and postgres

* Updated the docker file path

* Updated the path of docker folder structure

* Fix docker

* Updating the PR with necessary changes required

---------

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-03-06 08:58:53 -08:00
Nahuel
f2e1a87b5a
Fix#10377: service connection not overwritten as expected (#10445) 2023-03-06 16:32:10 +01:00
Schlameel
fb7b12842b
#9544: Added patch owner to Python SDK. Includes tests. (#10403)
Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-06 14:32:58 +00:00
Teddy
8bc0472fae
Handle null window metric (#10440)
* fix: handle null window metrics

* fix: remove else from try/except pattern
2023-03-06 14:09:02 +00:00
Nahuel
ef1812a09d
Fix: Stop displaying authorization values in debug logs (#10443) 2023-03-06 14:56:29 +01:00
Pere Miquel Brull
477a5223eb
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
2023-03-06 14:44:16 +01:00
Mayur Singal
f7a7a16cdb
Fix hive get table issue with impala (#10424)
* Fix hive get table issue with impala

* Fix Lint
2023-03-06 11:29:54 +01:00
NiharDoshi99
895e7d2706
fix: scoped import for presidio (#10439) 2023-03-06 14:26:52 +05:30
Teddy
5208b6f684
Fixes #4368 - Add Histogram Metric (#10422) 2023-03-03 21:56:32 +01:00
Nahuel
247016307d
Fix#8648: Mask sensitive info from API responses (#10307)
* Mask sensitive info from API responses

* Rename converter classes

* Add missing Java classes from JSON schemas and class converters

* Update test service connection schema

* Update datalakeConnection JSON schema and fix some tests

* Fix AlertsRuleEvaluatorResourceTest and minor error in run_local_docker.sh

* Fix Pipeline and Database service tests

* Minor refactor

* Fix CsvUtilTest

* Fix EventMonitorFactoryTest

* Fix CloudWatchEventMonitorTest

* Update datalake metadata

* Update bigquery metadata

* Fix test connection functionality

* Fix OMeta service api test

* Update gcsValues title and revert changes in GH actions

* Mask sensitive enabled by default for local docker

* Add missing tests

* Address PR comments

* Address PR comments

* fix ui breaks on gcsValues.json

* Address PR comments

* Minor refactor

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-03 18:10:01 +00:00
NiharDoshi99
1ff76f5e65
pii tagging using spacy (#10256)
* WIP: pii tagging using spacy

* added test cases and changes as per comment

* fix python checkstyle

* fix python checkstyle

* added score, test_cases and docs update

* solved merge conflict

* fix python checkstyle

* remove pii tagging using regex

* fix python test

* lib changes and added some test case

* changed as per comment

* fix: python test

* fix: changes to get source_config

* fix: changes as per comment
2023-03-03 18:33:18 +05:30
Teddy
775ca75e87
fix #10173 handle cases where entity would be deleted from OM (#10364)
* fix(dataInsight): handle cases where entity would be deleted from OM

* Update ingestion/src/metadata/data_insight/processor/web_analytic_report_data_processor.py

Added explanation in code comments

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

* fix(dataInsight): tests failure

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-03-03 12:30:15 +01:00
Mayur Singal
92fee5cf4f
Fix #10292: Remove DB & Schema Usage Filters (#10378)
* Fix #10292: Remove DB & Schema Usage Filters

* Fix Lint

* lint
2023-03-03 06:39:55 +01:00
Milan Bariya
7faec7880a
Update test connection (#10317)
* Update test connection

* Update test connection

* Update test connection with Tags

* Change based on comments

* Fix Pytest

* Service connectin arg in all connecter

* Fix pytest and conflict
2023-03-02 14:22:38 +05:30
Teddy
6a4df5f460
fix: removed pandas dependencies for non pandas profiler/testSuite workflows (#10380) 2023-03-01 16:38:50 +01:00
Kevin R
1c85f1b7fb
fix: Use endpointURL and region in s3_fs (#10305) 2023-03-01 14:05:24 +01:00
Mayur Singal
a11f7d7e37
Fix #8981: Move Connection Options and Connection Args to main properties (#10301) 2023-03-01 17:12:28 +05:30