3502 Commits

Author SHA1 Message Date
Pere Miquel Brull
78d7dd8789
[WIP] - Test Connection - Prepare the new test connection ingestion+UI logic (#10660)
* Prepare the new test connection ingestion logic

* Update test assert

* Update Test Connection for SQA Sources

* Correct return type and method doc

* Handle decryption

* Non SQA Database Sources

* Add the run_automation script in ingestion-base

* Dashboard Test Connection Changes

* Pipeline, Messagin, MlModel & Metadata Sources

* ui: test connect flow-1

* Unmask connection parameters before sending to Ariflow

* ui: test connect flow-2

* Address review comments and pylint

* pytest fix

* ui: test connect flow-3 (refactoring and style fix)

* ui: test connect flow-4 (fix test connection status logic)

* sync local file

* ui: test connect flow-5 (fix lowercase issue and styling)

* ui: test connect flow-5 (show toast notifications)

* test: add unit test

* ui: test connect flow-5 (update service page test connection button)

* Databrick fix & pytest fix

* pylint

* Update test

* Fix merge

* S3 Test connection

* add style for mandatory step

* sync locales

* chore: add service name in workflow request

* Unmask using original service connection parameters

* Fix test connection unmasking

* Wrap inspector function to eliminate error outside test conn

* Fix linting

* fix:cy test

* Fix linting

* address comment

* refactor and fix connection type casing issue

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-28 06:29:13 +02:00
Teddy
31d4662abe
Fixes #10775 -- BQ partitionning for datetime dtype (#10780)
* fix: BQ partitionning for datetime dtype

* Update ingestion/src/metadata/profiler/orm/functions/datetime.py

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

* Update ingestion/src/metadata/profiler/orm/functions/datetime.py

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

* Update ingestion/src/metadata/profiler/orm/functions/datetime.py

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-03-27 15:28:34 +02:00
Pere Miquel Brull
cc3d2b448b
Clean Airflow lineage test (#10745) 2023-03-27 11:46:34 +02:00
Cristian Calugaru
3ffde9a293
Fixes #9064 - Ingestion for S3 bucket containers (#10515)
* Prep skeleton for object store connector

* First of of ingestion piepline for bucket containers.

* addressed PR review comments.

* swaped random with secrets due to security warning.

* linter issues.

* added prefix population for containers.

* more linting

* removed temp file.

* object store unit tests, PR comments and minor refactorings

* docs update and new TestConnection approach.

* linting

* removed isStructured field from containermeta schema

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-03-26 11:35:34 +02:00
Nahuel
ef759c7e88
Fixes#8038: Change how status is handled after running workflow (#10710)
* Change how status is handled after running workflow

* Reset changes in config files

* Add auxiliary Summary class

* Improve failures handling

* Pylint error

* Pylint error

* Show result in table

* Add test

* Fix setup.py

* Add comments
2023-03-24 17:59:06 +01:00
Milan Bariya
48cc249b11
Fix glue ingestion issue (#10732) 2023-03-23 20:53:43 +05:30
Onkar Ravgan
c9feba826f
Kinesis bug fixed (#10578)
* Kinesis bug fixed

* Added sample data fix

* Fixed typo

* Added constansts

* Added try catch for sample data

* Added fixes on review comments

* Modified models

* Added sample data py models

* Fix pylint

* fixed code smell

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-03-23 13:38:56 +05:30
Teddy
3d8e7e6d41
Fixes #10290 - NUMERIC type assigned the wrong dtype in converter.py (#10708)
* fix: profiler refactor

* fix: catch division by zero error

* fix: instantiated new column object from name to delegate type handler back to dbapi

* fix: reverted columns instantiation and updated NUMBER type to NUMERIC

* fix: updated doc string for process_pii_sensitive method
2023-03-23 07:22:19 +01:00
Mayur Singal
bbce9c5aa4
Fix #10489: Handle unknown data types & store raw data type (#10563) 2023-03-23 11:41:29 +05:30
Pere Miquel Brull
ebbd611825
Update counts after #10265 (#10693) 2023-03-22 06:32:18 +01:00
Pere Miquel Brull
e2a2bcc8da
Fix search by email index keyword (#10698)
* Fix search by email index keyword

* Fix search by email index keyword
2023-03-21 20:50:47 -07:00
Pere Miquel Brull
4dbe5e4f5c
Simplify Data Insight workflow builder (#10688) 2023-03-21 14:12:20 +01:00
Nahuel
dd225d1463
Fix: Clean Source and SourceStatus inheritance (#10687)
* Fix: clean source and sourcestatus inheritance

* Remove auth parameters from mysql.yaml

* Fix failing test

* Fix pylint checks

* Add missing docstrings
2023-03-21 12:57:48 +01:00
Teddy
64205cf231
feat: Added custom event event type for web analytics (#10554)
* feat: Added custom event event type for web analytics

* feat: renamed  to

* Update openmetadata-spec/src/main/resources/json/schema/analytics/webAnalyticEventType/customEvent.json

Co-authored-by: Nahuel <nahuel@getcollate.io>

* Update openmetadata-spec/src/main/resources/json/schema/analytics/webAnalyticEventType/customEvent.json

Co-authored-by: Nahuel <nahuel@getcollate.io>

* feat: move common type to basic.json

* feat: ran linting

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-21 10:12:15 +01:00
Pere Miquel Brull
cdb14f30ff
Create stage location in the workflow, not the apis (#10676)
* Create stage location in the workflow, not the apis

* Format
2023-03-20 18:14:27 +01:00
Schlameel
df855ad8c3
Issue #3809: Add python client for Roles and Policies (#10531)
* Issue #3809: Add python client for Roles and Policies
Includes Tests

* #3809: Add python client for Roles and Policies
- Moved constants to enums in client_utils.py
- Updated all patch methods to utilized new enums
- includes tests

* #3809: Add python client for Roles and Policies
- includes tests
- merged upstream updates and updated to use new enums
2023-03-20 08:42:01 +01:00
NiharDoshi99
4fe713a875
fix: chart url (#10605)
* fix: chart url

* fix: chart url

* changes as per comment

* added test and pydantic for chart url

* changed as per comment

* changed as per comment

* fix python test

* changes as per comment

* fix python test
2023-03-20 06:38:44 +01:00
NiharDoshi99
89301729a2
fix e2e test for mysql (#10650) 2023-03-18 00:35:01 +05:30
Teddy
81bcd56aa4
fix: added fields arg. to (#10646) 2023-03-17 14:47:21 +00:00
Pere Miquel Brull
9b9cbd2d92
Update Storage Container list filter, add missing counts & rename workflow operations to automations (#10601)
* Rename Workflow package to automations

* Add list filter by root container & counts api

* Rename path

* Add glossary and glossaryTerm to count API

* Unify SqlQuery when there is no diff in mysql or psql

* chore: address feedback for object store service page

* Update glossary tests

* chore: add containers count on my data page

* fix: tag removal issue

* test: add unit test

* address comments

* address comment

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-03-16 23:34:53 +05:30
Teddy
9d858ede3a
fix: declaring self.data after calling self.token (#10634) 2023-03-16 17:28:02 +01:00
Milan Bariya
fadefff3e1
Fix databricks timeout issue (#10613)
* Fix databricks timeout issue

* Change based on comments

* Change timeOut word to connectionTimeout
2023-03-16 16:13:49 +01:00
Pere Miquel Brull
cd6ece1cf4
Fix oracle bulk view definitions and comments & improve metadata version (#10620)
* Add more info to client version

* lower oracle views and comments
2023-03-16 14:11:22 +01:00
Mohit Yadav
b982d3fe2b
Query as entity (#10449)
* added query as an entity

* changed name of the variables and methods

* Added Resource Descriptors

* testcase bug fix

* addressing comments

* added script for table query migration

* added script for table query migration postgresql

* bug fix

* db change for script test

* added current timestamp

* change db config from postgresql to mysql

* added extension to use fucntion gen_random_uuid()

* solving maven ci

* added queryUsage and change is migration script

* addressing comments

* addressing comments

* added queryUsage relation and testcase

* added api to insert queries in bulk

* .

* fix a test case which was failing due to latest changes

* Ingestion Changes for Query as Entity

* move query changes to latest sqls

* added tags and owner

* update PR for Query as Entity

* update type

* fixed pagination

* fix path param

* fix TestCases

* add validation criteria

* removed exisitng query apis

* checkstyle fix

* remove vote from put

* remove vote from put

* Query As Entity Ingestion Changes

* Remove unused func

* update Review Comments

* update Review Comments

* remove previous changes for Query and Update Tests

* moved Checksum to Query Util Class

* update python api

* fix python checkstyle

* Fixed Tests

* Fix pytest

* remove space changes

* remove space changes

* Fixed put_addFollowerDeleteEntity_200

* Fix usage ingestion

* Update Python SDK and tests

* pylint fix

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-15 20:55:30 -07:00
Milan Bariya
a51da74f86
Databricks Pileline code improve & Test connection update & Databricks issue solved (#10590)
* Databricks Pileline code imporove

* Change based on comments

* Change based on comments
2023-03-15 20:19:12 +05:30
Nahuel
07d6028149
Fix: remove avro-python3 deprecated dependency (#10602) 2023-03-15 14:15:57 +00:00
NiharDoshi99
205e5c8c4f
Aligned airflow source with lineage (#10596)
* aligned airflow source with lineage

* fix python test

* changes as per comment
2023-03-15 19:29:24 +05:30
Pere Miquel Brull
0e57c56d83
Add support dbt in connection (#10591)
* Add support dbt in connection

* Add support dbt in connection

* Tests
2023-03-15 14:31:20 +01:00
Sriharsha Chintalapani
b33587041d
Fix #5153: Add referred columns for foreign key constraint and sort_key, dist_key support (#10433)
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-03-15 18:55:51 +05:30
Onkar Ravgan
682626ffe8
Added exception handling in dashboard sources (#10573)
* added exception handling in dashboard

* updated process owner logic

* Added redash source for fixes
2023-03-15 08:30:03 +00:00
Ayush Shah
7e4ba4567b
Fix Quicksight URL (#10592) 2023-03-15 08:05:38 +00:00
Onkar Ravgan
93e554ae67
Fixed Redash Source Issues (#10570)
* Imporved redash source

* Added docs

* Addressed review comments
2023-03-14 23:00:49 +05:30
Teddy
2f4a92a17b
fix: exclude owner from page view traffic in DI (#10574)
* fix: exclude owner from page view traffic in DI

* fix: uncomment KPI creation in setup
2023-03-14 11:45:46 +00:00
NiharDoshi99
83a1858434
fix druid ingestion (#10553)
* fix druid ingestion

* changes as per comment

* changes as per comment
2023-03-14 14:47:00 +05:30
Nahuel
ffdd911785
Bug: TypeError when dashboards field is not present in a workspace response (#10568) 2023-03-14 08:58:53 +00:00
Ayush Shah
0f9c2c2164
Add Labels on DatabaseSchema Level (#10547) 2023-03-13 14:13:51 +00:00
Teddy
d03b06daf6
feat: Added logic to handle MERGE statement for bigquery (#10522) 2023-03-13 11:34:40 +01:00
Pere Miquel Brull
db292eaa0b
Prepare exit handler & Add missing arg to test connection (#10519)
* Prepare exit handler

* clean envs
2023-03-13 10:50:10 +01:00
NiharDoshi99
113fcc2956
added process owner (#10539)
* added process owner

* python checkstyle

* changes as per comment
2023-03-13 14:39:02 +05:30
Onkar Ravgan
30e5d696a1
Fixed dbt e2e filter test (#10536) 2023-03-13 12:05:52 +05:30
Milan Bariya
178bc8ecc0
Fix: Databricks complex column type issue (#10469)
* Fix: Databricks complex column type issue

* Fix: Failed to fetch view definition

* Change based on comments

* Change based on comments

* Change based on comments
2023-03-13 10:09:16 +05:30
Teddy
7ec667808c
Fixes #10498 - Add support for nifi client certificate auth. (#10499)
* feat: Added nifi support for client cert auth

* feat: fix code linting

* feat: addressed comments for nifi connections

* feat: fixed linting
2023-03-10 11:09:40 +01:00
Nahuel
b94927b3ea
Bug: fix flaky timeout test (#10509) 2023-03-10 10:55:56 +01:00
NiharDoshi99
34bf175ca1
adding quotes around alias (#10506) 2023-03-10 15:18:58 +05:30
NiharDoshi99
9b0b06fe25
Added flag for overriding owners (#10425)
* added flag for overriding owners

* docs update

* adding owners in patch call

* adding owners in patch call

* fix: python test case

* changes as per comment

* changes as per comment

* ui changes
2023-03-10 15:01:56 +05:30
Teddy
cea10e8301
fix: handle None owner (#10497) 2023-03-10 08:12:14 +00:00
Milan Bariya
bc4abc44ed
Add steps in test_connection function (#10408)
* Add steps in test_connection funstion

* Add steps in test_connection funstion

* Add steps in test_connection funstion

* Fix: Pylint issue

* Status response changefor statuscode 200

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-10 10:02:26 +05:30
Pere Miquel Brull
81dec813a0
Don't store the OM connection in the Ingestion Pipeline or Workflow (#10448)
* Do not store OM connection

* Migration to remove the server connection

* Update tests

* Add workflow masking and secrets manager

* Fix failing test

---------

Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
2023-03-09 17:32:40 +01:00
Sriharsha Chintalapani
d9e4fbdebb
Fix #10454: Improve Search Relevancy, by adding functional scoring an… (#10455)
* Fix #10454: Improve Search Relevancy, by adding functional scoring and add ngram analyzer; Fix #10452: Enable Table and Column search by BOTH name and displayName

* fix stylecheck

* Undo changes in table example names

* remove ngram from teams & users

* Fix topic tags

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf (#10430)

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Added top level parsing and unit tests

* fix(ui): show schemaText and fields both

* fix no data placeholder for fields & schema text

* addressing comments

* fixed py checkstyle

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>

* revert common_broker_source changes

* revert common_broker_source changes

* remove changes to user & team indexes

* fix team index

* fix glossary & tag index

* Fix to TopicIndex

* fix advance search pre-requisites cypress failure

* fix group advance search cy failures

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-09 21:37:08 +05:30
Teddy
f27266628d
update histogram sample data (#10495) 2023-03-09 15:19:57 +01:00