2916 Commits

Author SHA1 Message Date
Teddy
d03b06daf6
feat: Added logic to handle MERGE statement for bigquery (#10522) 2023-03-13 11:34:40 +01:00
Pere Miquel Brull
db292eaa0b
Prepare exit handler & Add missing arg to test connection (#10519)
* Prepare exit handler

* clean envs
2023-03-13 10:50:10 +01:00
NiharDoshi99
113fcc2956
added process owner (#10539)
* added process owner

* python checkstyle

* changes as per comment
2023-03-13 14:39:02 +05:30
Onkar Ravgan
30e5d696a1
Fixed dbt e2e filter test (#10536) 2023-03-13 12:05:52 +05:30
Milan Bariya
178bc8ecc0
Fix: Databricks complex column type issue (#10469)
* Fix: Databricks complex column type issue

* Fix: Failed to fetch view definition

* Change based on comments

* Change based on comments

* Change based on comments
2023-03-13 10:09:16 +05:30
Teddy
7ec667808c
Fixes #10498 - Add support for nifi client certificate auth. (#10499)
* feat: Added nifi support for client cert auth

* feat: fix code linting

* feat: addressed comments for nifi connections

* feat: fixed linting
2023-03-10 11:09:40 +01:00
Nahuel
b94927b3ea
Bug: fix flaky timeout test (#10509) 2023-03-10 10:55:56 +01:00
NiharDoshi99
34bf175ca1
adding quotes around alias (#10506) 2023-03-10 15:18:58 +05:30
NiharDoshi99
9b0b06fe25
Added flag for overriding owners (#10425)
* added flag for overriding owners

* docs update

* adding owners in patch call

* adding owners in patch call

* fix: python test case

* changes as per comment

* changes as per comment

* ui changes
2023-03-10 15:01:56 +05:30
Teddy
cea10e8301
fix: handle None owner (#10497) 2023-03-10 08:12:14 +00:00
Milan Bariya
bc4abc44ed
Add steps in test_connection function (#10408)
* Add steps in test_connection funstion

* Add steps in test_connection funstion

* Add steps in test_connection funstion

* Fix: Pylint issue

* Status response changefor statuscode 200

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-10 10:02:26 +05:30
Pere Miquel Brull
81dec813a0
Don't store the OM connection in the Ingestion Pipeline or Workflow (#10448)
* Do not store OM connection

* Migration to remove the server connection

* Update tests

* Add workflow masking and secrets manager

* Fix failing test

---------

Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
2023-03-09 17:32:40 +01:00
Sriharsha Chintalapani
d9e4fbdebb
Fix #10454: Improve Search Relevancy, by adding functional scoring an… (#10455)
* Fix #10454: Improve Search Relevancy, by adding functional scoring and add ngram analyzer; Fix #10452: Enable Table and Column search by BOTH name and displayName

* fix stylecheck

* Undo changes in table example names

* remove ngram from teams & users

* Fix topic tags

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf (#10430)

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Added top level parsing and unit tests

* fix(ui): show schemaText and fields both

* fix no data placeholder for fields & schema text

* addressing comments

* fixed py checkstyle

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>

* revert common_broker_source changes

* revert common_broker_source changes

* remove changes to user & team indexes

* fix team index

* fix glossary & tag index

* Fix to TopicIndex

* fix advance search pre-requisites cypress failure

* fix group advance search cy failures

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-09 21:37:08 +05:30
Teddy
f27266628d
update histogram sample data (#10495) 2023-03-09 15:19:57 +01:00
NiharDoshi99
ddbc0311e8
fix: mark table deleted dependencies (#10476) 2023-03-09 09:15:05 +00:00
Suresh Srinivas
4c6d184ef5
Fixes #10480 Glossary rename results in rename of Classification with… (#10486)
* Fixes #10480 Glossary rename results in rename of Classification with the same name

* Rename TagSource Tag to Classification
2023-03-09 00:30:36 -08:00
Nahuel
ed884cf79a
Bug: Update sqllineage-openmetadata + add timeout for parsing queries (#10474)
* Update sqllineage-openmetadata version + add timeout

* Pyimpala fix colnames, comments and dialect sql compilation (#10470)

* Fix col names and comments for impala hive

* Fix cols, comments and impala sql compilation

* Handle hive types

* Format

* Added doc in avro array and tests (#10473)

* Fixed: Add job definition id field for dbt cloud in UI #10269 (#10472)

* fixed Add job definition id field for dbt cloud in UI #10269

* sync-localization file

* fixed failing unit test and add unit test for the changes

* Address PR comments

* Update tests

* Pylint clean

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-03-08 20:49:02 +01:00
Teddy
56f7ac2c2c
Fix few profiler bugs (#10465)
* fix: handle overflow in profiler + fix minor bug from new metrics

* fix: flaky test from profiler

* fix: fix code smell
2023-03-08 18:01:25 +01:00
Onkar Ravgan
4d11db4220
Added doc in avro array and tests (#10473) 2023-03-08 20:16:50 +05:30
Pere Miquel Brull
a05e56feba
Pyimpala fix colnames, comments and dialect sql compilation (#10470)
* Fix col names and comments for impala hive

* Fix cols, comments and impala sql compilation

* Handle hive types

* Format
2023-03-08 14:13:06 +01:00
Mayur Singal
c199f13ed0
Fix #10402: Add support for AssumeRole for AWS (#10417) 2023-03-08 15:43:33 +05:30
NiharDoshi99
d41878ec90
fix: spacy model (#10467) 2023-03-08 15:19:35 +05:30
Onkar Ravgan
d4fafa3168
Downgraded conf-kafka lib (#10466) 2023-03-08 14:22:38 +05:30
Onkar Ravgan
ca623d0693
dbt performance improvement (#10366)
* dbt performance improvement

* reduced es calls

* Added unit tests
2023-03-08 06:37:34 +01:00
Pere Miquel Brull
63e285ef8f
Fix mysql (#10456) 2023-03-07 12:57:44 -08:00
Sriharsha Chintalapani
fe73948b55
Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf (#10430)
* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Added top level parsing and unit tests

* fix(ui): show schemaText and fields both

* fix no data placeholder for fields & schema text

* addressing comments

* fixed py checkstyle

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-07 15:40:04 +01:00
Sriharsha Chintalapani
de10cd254b
New docker structure (#10438)
* Adding the different docker-compose file openmetadata and ingestion

* Added two different env files for mysql and postgres

* Updated the docker file path

* Updated the path of docker folder structure

* Fix docker

* Updating the PR with necessary changes required

---------

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-03-06 08:58:53 -08:00
Nahuel
f2e1a87b5a
Fix#10377: service connection not overwritten as expected (#10445) 2023-03-06 16:32:10 +01:00
Schlameel
fb7b12842b
#9544: Added patch owner to Python SDK. Includes tests. (#10403)
Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-06 14:32:58 +00:00
Teddy
8bc0472fae
Handle null window metric (#10440)
* fix: handle null window metrics

* fix: remove else from try/except pattern
2023-03-06 14:09:02 +00:00
Nahuel
ef1812a09d
Fix: Stop displaying authorization values in debug logs (#10443) 2023-03-06 14:56:29 +01:00
Pere Miquel Brull
050da1e2d1
Add service type to container (#10441) 2023-03-06 14:44:30 +01:00
Pere Miquel Brull
477a5223eb
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
2023-03-06 14:44:16 +01:00
Mayur Singal
f7a7a16cdb
Fix hive get table issue with impala (#10424)
* Fix hive get table issue with impala

* Fix Lint
2023-03-06 11:29:54 +01:00
NiharDoshi99
895e7d2706
fix: scoped import for presidio (#10439) 2023-03-06 14:26:52 +05:30
Teddy
5208b6f684
Fixes #4368 - Add Histogram Metric (#10422) 2023-03-03 21:56:32 +01:00
Nahuel
247016307d
Fix#8648: Mask sensitive info from API responses (#10307)
* Mask sensitive info from API responses

* Rename converter classes

* Add missing Java classes from JSON schemas and class converters

* Update test service connection schema

* Update datalakeConnection JSON schema and fix some tests

* Fix AlertsRuleEvaluatorResourceTest and minor error in run_local_docker.sh

* Fix Pipeline and Database service tests

* Minor refactor

* Fix CsvUtilTest

* Fix EventMonitorFactoryTest

* Fix CloudWatchEventMonitorTest

* Update datalake metadata

* Update bigquery metadata

* Fix test connection functionality

* Fix OMeta service api test

* Update gcsValues title and revert changes in GH actions

* Mask sensitive enabled by default for local docker

* Add missing tests

* Address PR comments

* Address PR comments

* fix ui breaks on gcsValues.json

* Address PR comments

* Minor refactor

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-03 18:10:01 +00:00
NiharDoshi99
1ff76f5e65
pii tagging using spacy (#10256)
* WIP: pii tagging using spacy

* added test cases and changes as per comment

* fix python checkstyle

* fix python checkstyle

* added score, test_cases and docs update

* solved merge conflict

* fix python checkstyle

* remove pii tagging using regex

* fix python test

* lib changes and added some test case

* changed as per comment

* fix: python test

* fix: changes to get source_config

* fix: changes as per comment
2023-03-03 18:33:18 +05:30
Teddy
775ca75e87
fix #10173 handle cases where entity would be deleted from OM (#10364)
* fix(dataInsight): handle cases where entity would be deleted from OM

* Update ingestion/src/metadata/data_insight/processor/web_analytic_report_data_processor.py

Added explanation in code comments

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

* fix(dataInsight): tests failure

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-03-03 12:30:15 +01:00
Mayur Singal
92fee5cf4f
Fix #10292: Remove DB & Schema Usage Filters (#10378)
* Fix #10292: Remove DB & Schema Usage Filters

* Fix Lint

* lint
2023-03-03 06:39:55 +01:00
Milan Bariya
7faec7880a
Update test connection (#10317)
* Update test connection

* Update test connection

* Update test connection with Tags

* Change based on comments

* Fix Pytest

* Service connectin arg in all connecter

* Fix pytest and conflict
2023-03-02 14:22:38 +05:30
Teddy
6a4df5f460
fix: removed pandas dependencies for non pandas profiler/testSuite workflows (#10380) 2023-03-01 16:38:50 +01:00
Kevin R
1c85f1b7fb
fix: Use endpointURL and region in s3_fs (#10305) 2023-03-01 14:05:24 +01:00
Mayur Singal
a11f7d7e37
Fix #8981: Move Connection Options and Connection Args to main properties (#10301) 2023-03-01 17:12:28 +05:30
Teddy
754074f1be
Fixes #7758 - Added Column value and Integer Range Partitionning (#10350)
* feat(profiler): renamed  module to

* feat(profiler): added dbt-artifacts-parser to test setup.py

* feat(profiler): refactor workflow and interface

* feat(profiler): linting

* feat(profiler): removed old profiler modules

* feat(profiler): added support for value and integer range partition

* feat(profiler): fixed linting

* feat(profiler): added partitionning support for datalake profiler

* feat(profiler): removed `ProfilerInterfaceArgs` class

* feat(profiler): address comments

* feat(profiler): Added `OTHER` as an `IntervalType` for UI type generation
2023-03-01 08:20:38 +01:00
Mayur Singal
f6ba024b5f
Fix #10263: Fetch Vertica Schema Comments (#10363) 2023-03-01 11:35:01 +05:30
Mayur Singal
55a28ffd40
Trino Optimize table comments (#10367) 2023-03-01 10:39:28 +05:30
Mayur Singal
909cedeca1
Support Snowflake Geometary Type (#10359) 2023-02-28 18:59:01 +05:30
Mayur Singal
d3e5461c58
Fix MySQL E2E CLI (#10352) 2023-02-28 07:58:03 +05:30
Sriharsha Chintalapani
d21fcad203
Fix patch tag for columns (#10191)
* Fix patch tag for columns

* Fix patch tag for columns

* fix pylint issues

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-02-26 15:33:47 +01:00