2826 Commits

Author SHA1 Message Date
NiharDoshi99
205e5c8c4f
Aligned airflow source with lineage (#10596)
* aligned airflow source with lineage

* fix python test

* changes as per comment
2023-03-15 19:29:24 +05:30
Pere Miquel Brull
0e57c56d83
Add support dbt in connection (#10591)
* Add support dbt in connection

* Add support dbt in connection

* Tests
2023-03-15 14:31:20 +01:00
Sriharsha Chintalapani
b33587041d
Fix #5153: Add referred columns for foreign key constraint and sort_key, dist_key support (#10433)
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-03-15 18:55:51 +05:30
Onkar Ravgan
682626ffe8
Added exception handling in dashboard sources (#10573)
* added exception handling in dashboard

* updated process owner logic

* Added redash source for fixes
2023-03-15 08:30:03 +00:00
Ayush Shah
7e4ba4567b
Fix Quicksight URL (#10592) 2023-03-15 08:05:38 +00:00
Onkar Ravgan
93e554ae67
Fixed Redash Source Issues (#10570)
* Imporved redash source

* Added docs

* Addressed review comments
2023-03-14 23:00:49 +05:30
Teddy
2f4a92a17b
fix: exclude owner from page view traffic in DI (#10574)
* fix: exclude owner from page view traffic in DI

* fix: uncomment KPI creation in setup
2023-03-14 11:45:46 +00:00
NiharDoshi99
83a1858434
fix druid ingestion (#10553)
* fix druid ingestion

* changes as per comment

* changes as per comment
2023-03-14 14:47:00 +05:30
Nahuel
ffdd911785
Bug: TypeError when dashboards field is not present in a workspace response (#10568) 2023-03-14 08:58:53 +00:00
Ayush Shah
0f9c2c2164
Add Labels on DatabaseSchema Level (#10547) 2023-03-13 14:13:51 +00:00
Teddy
d03b06daf6
feat: Added logic to handle MERGE statement for bigquery (#10522) 2023-03-13 11:34:40 +01:00
Pere Miquel Brull
db292eaa0b
Prepare exit handler & Add missing arg to test connection (#10519)
* Prepare exit handler

* clean envs
2023-03-13 10:50:10 +01:00
NiharDoshi99
113fcc2956
added process owner (#10539)
* added process owner

* python checkstyle

* changes as per comment
2023-03-13 14:39:02 +05:30
Onkar Ravgan
30e5d696a1
Fixed dbt e2e filter test (#10536) 2023-03-13 12:05:52 +05:30
Milan Bariya
178bc8ecc0
Fix: Databricks complex column type issue (#10469)
* Fix: Databricks complex column type issue

* Fix: Failed to fetch view definition

* Change based on comments

* Change based on comments

* Change based on comments
2023-03-13 10:09:16 +05:30
Teddy
7ec667808c
Fixes #10498 - Add support for nifi client certificate auth. (#10499)
* feat: Added nifi support for client cert auth

* feat: fix code linting

* feat: addressed comments for nifi connections

* feat: fixed linting
2023-03-10 11:09:40 +01:00
Nahuel
b94927b3ea
Bug: fix flaky timeout test (#10509) 2023-03-10 10:55:56 +01:00
NiharDoshi99
34bf175ca1
adding quotes around alias (#10506) 2023-03-10 15:18:58 +05:30
NiharDoshi99
9b0b06fe25
Added flag for overriding owners (#10425)
* added flag for overriding owners

* docs update

* adding owners in patch call

* adding owners in patch call

* fix: python test case

* changes as per comment

* changes as per comment

* ui changes
2023-03-10 15:01:56 +05:30
Teddy
cea10e8301
fix: handle None owner (#10497) 2023-03-10 08:12:14 +00:00
Milan Bariya
bc4abc44ed
Add steps in test_connection function (#10408)
* Add steps in test_connection funstion

* Add steps in test_connection funstion

* Add steps in test_connection funstion

* Fix: Pylint issue

* Status response changefor statuscode 200

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-10 10:02:26 +05:30
Pere Miquel Brull
81dec813a0
Don't store the OM connection in the Ingestion Pipeline or Workflow (#10448)
* Do not store OM connection

* Migration to remove the server connection

* Update tests

* Add workflow masking and secrets manager

* Fix failing test

---------

Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
2023-03-09 17:32:40 +01:00
Sriharsha Chintalapani
d9e4fbdebb
Fix #10454: Improve Search Relevancy, by adding functional scoring an… (#10455)
* Fix #10454: Improve Search Relevancy, by adding functional scoring and add ngram analyzer; Fix #10452: Enable Table and Column search by BOTH name and displayName

* fix stylecheck

* Undo changes in table example names

* remove ngram from teams & users

* Fix topic tags

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf (#10430)

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Added top level parsing and unit tests

* fix(ui): show schemaText and fields both

* fix no data placeholder for fields & schema text

* addressing comments

* fixed py checkstyle

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>

* revert common_broker_source changes

* revert common_broker_source changes

* remove changes to user & team indexes

* fix team index

* fix glossary & tag index

* Fix to TopicIndex

* fix advance search pre-requisites cypress failure

* fix group advance search cy failures

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-09 21:37:08 +05:30
Teddy
f27266628d
update histogram sample data (#10495) 2023-03-09 15:19:57 +01:00
NiharDoshi99
ddbc0311e8
fix: mark table deleted dependencies (#10476) 2023-03-09 09:15:05 +00:00
Suresh Srinivas
4c6d184ef5
Fixes #10480 Glossary rename results in rename of Classification with… (#10486)
* Fixes #10480 Glossary rename results in rename of Classification with the same name

* Rename TagSource Tag to Classification
2023-03-09 00:30:36 -08:00
Nahuel
ed884cf79a
Bug: Update sqllineage-openmetadata + add timeout for parsing queries (#10474)
* Update sqllineage-openmetadata version + add timeout

* Pyimpala fix colnames, comments and dialect sql compilation (#10470)

* Fix col names and comments for impala hive

* Fix cols, comments and impala sql compilation

* Handle hive types

* Format

* Added doc in avro array and tests (#10473)

* Fixed: Add job definition id field for dbt cloud in UI #10269 (#10472)

* fixed Add job definition id field for dbt cloud in UI #10269

* sync-localization file

* fixed failing unit test and add unit test for the changes

* Address PR comments

* Update tests

* Pylint clean

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-03-08 20:49:02 +01:00
Teddy
56f7ac2c2c
Fix few profiler bugs (#10465)
* fix: handle overflow in profiler + fix minor bug from new metrics

* fix: flaky test from profiler

* fix: fix code smell
2023-03-08 18:01:25 +01:00
Onkar Ravgan
4d11db4220
Added doc in avro array and tests (#10473) 2023-03-08 20:16:50 +05:30
Pere Miquel Brull
a05e56feba
Pyimpala fix colnames, comments and dialect sql compilation (#10470)
* Fix col names and comments for impala hive

* Fix cols, comments and impala sql compilation

* Handle hive types

* Format
2023-03-08 14:13:06 +01:00
Mayur Singal
c199f13ed0
Fix #10402: Add support for AssumeRole for AWS (#10417) 2023-03-08 15:43:33 +05:30
NiharDoshi99
d41878ec90
fix: spacy model (#10467) 2023-03-08 15:19:35 +05:30
Onkar Ravgan
d4fafa3168
Downgraded conf-kafka lib (#10466) 2023-03-08 14:22:38 +05:30
Onkar Ravgan
ca623d0693
dbt performance improvement (#10366)
* dbt performance improvement

* reduced es calls

* Added unit tests
2023-03-08 06:37:34 +01:00
Pere Miquel Brull
63e285ef8f
Fix mysql (#10456) 2023-03-07 12:57:44 -08:00
Sriharsha Chintalapani
fe73948b55
Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf (#10430)
* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Added top level parsing and unit tests

* fix(ui): show schemaText and fields both

* fix no data placeholder for fields & schema text

* addressing comments

* fixed py checkstyle

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-07 15:40:04 +01:00
Sriharsha Chintalapani
de10cd254b
New docker structure (#10438)
* Adding the different docker-compose file openmetadata and ingestion

* Added two different env files for mysql and postgres

* Updated the docker file path

* Updated the path of docker folder structure

* Fix docker

* Updating the PR with necessary changes required

---------

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-03-06 08:58:53 -08:00
Nahuel
f2e1a87b5a
Fix#10377: service connection not overwritten as expected (#10445) 2023-03-06 16:32:10 +01:00
Schlameel
fb7b12842b
#9544: Added patch owner to Python SDK. Includes tests. (#10403)
Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-06 14:32:58 +00:00
Teddy
8bc0472fae
Handle null window metric (#10440)
* fix: handle null window metrics

* fix: remove else from try/except pattern
2023-03-06 14:09:02 +00:00
Nahuel
ef1812a09d
Fix: Stop displaying authorization values in debug logs (#10443) 2023-03-06 14:56:29 +01:00
Pere Miquel Brull
050da1e2d1
Add service type to container (#10441) 2023-03-06 14:44:30 +01:00
Pere Miquel Brull
477a5223eb
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
2023-03-06 14:44:16 +01:00
Mayur Singal
f7a7a16cdb
Fix hive get table issue with impala (#10424)
* Fix hive get table issue with impala

* Fix Lint
2023-03-06 11:29:54 +01:00
NiharDoshi99
895e7d2706
fix: scoped import for presidio (#10439) 2023-03-06 14:26:52 +05:30
Teddy
5208b6f684
Fixes #4368 - Add Histogram Metric (#10422) 2023-03-03 21:56:32 +01:00
Nahuel
247016307d
Fix#8648: Mask sensitive info from API responses (#10307)
* Mask sensitive info from API responses

* Rename converter classes

* Add missing Java classes from JSON schemas and class converters

* Update test service connection schema

* Update datalakeConnection JSON schema and fix some tests

* Fix AlertsRuleEvaluatorResourceTest and minor error in run_local_docker.sh

* Fix Pipeline and Database service tests

* Minor refactor

* Fix CsvUtilTest

* Fix EventMonitorFactoryTest

* Fix CloudWatchEventMonitorTest

* Update datalake metadata

* Update bigquery metadata

* Fix test connection functionality

* Fix OMeta service api test

* Update gcsValues title and revert changes in GH actions

* Mask sensitive enabled by default for local docker

* Add missing tests

* Address PR comments

* Address PR comments

* fix ui breaks on gcsValues.json

* Address PR comments

* Minor refactor

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-03 18:10:01 +00:00
NiharDoshi99
1ff76f5e65
pii tagging using spacy (#10256)
* WIP: pii tagging using spacy

* added test cases and changes as per comment

* fix python checkstyle

* fix python checkstyle

* added score, test_cases and docs update

* solved merge conflict

* fix python checkstyle

* remove pii tagging using regex

* fix python test

* lib changes and added some test case

* changed as per comment

* fix: python test

* fix: changes to get source_config

* fix: changes as per comment
2023-03-03 18:33:18 +05:30
Teddy
775ca75e87
fix #10173 handle cases where entity would be deleted from OM (#10364)
* fix(dataInsight): handle cases where entity would be deleted from OM

* Update ingestion/src/metadata/data_insight/processor/web_analytic_report_data_processor.py

Added explanation in code comments

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

* fix(dataInsight): tests failure

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-03-03 12:30:15 +01:00
Mayur Singal
92fee5cf4f
Fix #10292: Remove DB & Schema Usage Filters (#10378)
* Fix #10292: Remove DB & Schema Usage Filters

* Fix Lint

* lint
2023-03-03 06:39:55 +01:00