618 Commits

Author SHA1 Message Date
Sriharsha Chintalapani
9dbaabad44
Adding MD5 hash to fullyQualifiedName and names to simplify DB indexes for lookups and increasing the size of FQN (#11960)
* Fix fqn hash

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* merge commits

* merge commits

* merge commits

* Fix glossary import/export

* Fix BotResource Tests

* Fix BotResource Tests

* Fix Glossary Term tests

* Fix Glossary Term tests

* Fix Glossary Import/Export tests

* All backend tests are fixed.

* merge with main

* Fix tests and revert local changes

* Fix tests and revert local changes

* Fix NullPointerException for Glossary and Query

* Fix Tests

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-06-19 15:43:05 +05:30
Mayur Singal
8ecd88fd1b
Fix #11312: Deprecate ES Reindex on Ingestion and use reindex APIs instead (#11988) 2023-06-19 12:45:18 +05:30
Mohit Yadav
41c49b9de8
Data Insight Report Job Bug Fixes (#11975)
* Data Insight Report Job Bug Fixes

* ui: add enabled field

* test case fix

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-16 15:40:44 +05:30
Ayush Shah
f80eaf3a26
Fixes 11068: mysql & postgres iam auth (#11937) 2023-06-16 13:18:12 +05:30
Teddy
3f01ee938f
Fixes #11895 - Add Indexing and Search logic for TestCases (#11989)
* feat: added logic to delete logical test suite + added check to prevent adding existing testCases to executable test suite

* feat: added elasticsearch index logic for testCases

* feat: added deletion logic from index logic when deleting test suites

* feat: added test case index search to  endpoint

* feat: add executable/logical filter in list testSuite + filterOut tables without tests in Table resource

* feat: added summary field to testSuite

* feat: added executionSummary endpoint for test cases

* feat: removed tick marks around timestamp

* feat: addressed test failures

* feat: ran python linting

* feat: add limit to fetch all tables in TableResource testSuite test

* feat: fix conflict

* feat: ran java checkstyle

* feat: fixed mongo linting + disabled mongo failing tests

* feat: removed mongo test skip

* feat: removed unsued pytest import
2023-06-15 21:27:54 +02:00
Onkar Ravgan
d08c928801
Added project property to dashboards (#11986)
* Added projects to dashboards

* Added powerbi proj

* merge conflicts after source url

* fixed mongo pytest
2023-06-15 21:23:43 +05:30
Mayur Singal
82a0222257
SourceUrl changes for dashboard, pipeline & chart entities (#11991) 2023-06-15 14:44:48 +05:30
Mayur Singal
7fa963eec3
Fix #1076: Add mongodb support (#11943) 2023-06-15 11:14:22 +05:30
Onkar Ravgan
d409c339e4
Fix 10689: Added Powerbi Datamodels (#11923) 2023-06-14 17:00:08 +05:30
Ayush Shah
2e5560926d
Add Lineage support for mssql (#11971) 2023-06-13 17:10:26 +05:30
Pere Miquel Brull
ba5f929f77
Add fine-grained health validation for Airflow APIs (#11792)
* Add fine-grained health validation for Airflow APIs

* Add ingestion version response

* Improve messaging

* Format

* Format

* Update response

* ui:update the hook to return airflow status based on response status field

* ui: add unit test for useAirflowStatus hook

* chore: only show test connection if airflow is available

* feat: add airflow message banner

* chore: update icon and background color

* chore: update typography to text

* test: add unit test

* address comments

* chore: show banner on service detail page

* fix: update test suite api workflow to reflect new implementation

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-06-12 07:47:45 +02:00
07Himank
62af9bb633
fixed issue for lineage description (#11500)
* fixed issue for lineage description

* fixed issue while ingesting

* fixed issue while ingesting

* added test case for Lingeage with description

* addressing comments .. enhancement

* addressing comments .. enhancement

* modified py test case and removed description from addLineage as we are not using it.

* add support for topic entity and description in lineage details

* fix pylint & test

* pytest fix

* fix column lineage null issue

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-06-12 11:17:32 +05:30
Mayur Singal
175093d93e
Fix datamodel generate issue for 3.8 (#11934) 2023-06-09 12:33:21 +05:30
Mayur Singal
f43aaf4150
Add database name filed for oracle db (#11906) 2023-06-09 09:31:07 +05:30
Mayur Singal
9e6e00caf9
Fix #9282: Support Databricks Unity Catalog (#11896) 2023-06-07 11:50:31 +05:30
Suresh Srinivas
f23f67dfe9
Fixes #11617 Entity get operations check for ViewAll operation instead of specific view operations based on the field requested (#11904) 2023-06-06 15:18:57 -07:00
Mohit Yadav
57e44c22be
Make Resource Packages Dynamic (#11891) 2023-06-07 01:02:52 +05:30
Teddy
4b9f213dbf
Fixes Issue #11863 - Add Status to DQ (#11893)
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.

* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity

* feat: add status object to test case results

* feat: ran python linting

* feat: fixed  update to
2023-06-06 10:09:16 +00:00
Teddy
721869428e
Revert "Fixe Issue #11863 - Add Status logic for test case results (#11881)" (#11892)
This reverts commit 06735fe8dbaac5b267c9a2cf744ca154f88a9247.
2023-06-06 09:56:12 +02:00
Teddy
06735fe8db
Fixe Issue #11863 - Add Status logic for test case results (#11881)
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.

* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity

* feat: add status object to test case results

* feat: ran python linting
2023-06-06 09:45:49 +02:00
Ayush Shah
65f370e4aa
Rename GCS to GCP (#11812) 2023-06-06 11:57:00 +05:30
Mohit Yadav
26db825b71
Added Support for Slack Related Configuration , Added Slack Bot , Sla… (#11791)
* Added Support for Slack Related Configuration , Added Slack Bot , Slack Installer and Slack Application Configuration

* comment

* Checkstyle

* Update Encrypt and Decrypt to encrypt all settings for slack

* Fix typo

* Update version for client and bolt

* Update deps

* Remove dependency for Slack bolt servlet
2023-06-06 00:55:14 +05:30
Suresh Srinivas
d978580703
Fixes #11872 - Add authorization for Usage APIs (#11873) 2023-06-02 15:50:29 -07:00
Teddy
d0cffdcd66
Fixes Issue #11438 - Implement threshold and startegy for custom SQL (#11847)
* feat: Add threshold and strategy logic on the custom SQL object test

* feat: ran python linting

* feat: added safety checks for custom sql query

* feat: ran python linting
2023-06-02 09:41:31 +02:00
Teddy
c98a15ca19
Fixes #11705 - Update ingestion and backend to match new DQ flow (#11836)
* feat: refactor ingestion flow logic

* feat: ran python linting

* feat: update tests to match new workflow

* feat: ran python linting

* feat: update sample data test suite name

* feat: Added backend logic to support logical and executable test suites

* feat: clean up java and json code

* feat: added sample data for logical and executable test suites

* feat: remove executable from CreateTestSuite

* feat: ran python and java linting

* feat: added README info for data quality structure

* skipping cypress to keep main green

* fixed typescript type issue

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-06-01 23:19:13 -07:00
Pere Miquel Brull
fdeea71671
Fix Looker explore git link & Add BitBucket reader (#11837)
* Add looker test connection step

* Add looker test connection step

* Update Credentials

* Fix explore link and add bitbucket reader

* Format

* Fix test

* Fix spline linting

* Fix import
2023-06-02 07:19:32 +02:00
Pere Miquel Brull
11c07ee8ab
Fix #11516 - SAP Hana Connector (#11777)
* SAP Hana skeleton

* Add SAP Hana Connector

* Fix ingestion and docs

* Prep SAP Hana Profiler

* Linting

* Update index.md

* Revert: Update index.md

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-31 16:00:31 +02:00
Mohit Yadav
79d6d0bcd3
Add extension to OpenMetadata (#11786)
* Add extension to OpenMetadata

* Typo Comment

* remove log
2023-05-28 23:44:02 +05:30
Louie
956ec58930
Add ES RestHighLevelClient TCP keep alive timeout (#11731)
* Add RestHighLevelClient TCP keep alive timeout

* Merge RestHighLevelClient configs
2023-05-28 10:10:46 -07:00
Chirag Madlani
7adc291364
fix(ui): circular deps for entityReference.json (#11760)
* fix(ui): circular deps for entityReference.json

* Fix circular Dependency python

* Cap Delta Spark version

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-26 18:02:21 +05:30
Onkar Ravgan
f677408f20
Fixes 11603: Tableau Data Sources as Data Models (#11688)
* added initial changes

* Added lineage details

* fixed pylint

* Fixed name

* Fixed empty lists
2023-05-22 18:07:02 +02:00
Teddy
8c50d1af52
Fixes #4565 - Fetch Metrics from System tables (#11645)
* feat: fetch metrics from system tables

* feat: add permission doc for fetching metrics from system tables

* feat: fix E2E tests to reflect full table row count after table metric update

* feat: ran linting

* feat: fix doc string engine name + function typing

* feat: ran python linting
2023-05-22 09:04:18 +02:00
Mayur Singal
e9992a52a8
Fix #1604: Add Spline Pipeline Connector (#11562)
* Fix #1604: Add Spline Connector

* Add tests & grammer validation

* Spline UI Changes & Docs

* fix pipeline workflow doc

* chore: use common field for dbService name

* chore: use const for beta services

* chore: add service icon

* Update ingestion/src/metadata/ingestion/source/pipeline/spline/metadata.py

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-05-19 14:46:32 +05:30
Pere Miquel Brull
50ad38ea0f
Fix #11548 - Secrets Managers comms with OMeta (#11602)
* Remove secretsManagerCredentials from backend

* Remove secretsManagerCredentials from backend

* Add secrets manager loader

* Load SM in the ometa client

* Fix tests

* Fix tests

* Fix Lint

* Mock AWS region

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-19 09:43:11 +02:00
Chirag Madlani
763b96b8fe
bump(ui): rjsf to v5 (#11549)
* bump(ui): rjsf to v5
It will fix validation and extra parameter issues

* address comments
fix code smells
fix cypress failure

* revert schema files

* bigquery fix
sonar cloud fix

* fix big query

* fix superset

* fix cypress

* fix cypress for big query

* fix failures

* fix java issue

* revert connectionBasicType.json
create custom SupersetConnection.json

* fix Tableau service issues #11637
2023-05-18 08:03:54 -07:00
Mayur Singal
e4997c3749
Fix #11571: Support custom database name for glue (#11631) 2023-05-18 14:16:56 +05:30
Onkar Ravgan
e6a7ffc57c
Fixes: 11540 Added api_version and domain fields to Salesforce (#11618)
* Added sales new fields

* fixed image after merge

* Fix Failing Test

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-05-18 10:53:44 +05:30
Suresh Srinivas
e72125c3a1
11579 task - Change schema regex to allow parenthesis in the entity name (#11580) 2023-05-14 06:58:38 -07:00
Mohit Yadav
d12d32bf65
Add settings to Db for Custom Logo (#11583)
* Add settings to Db for Custom Logo

* Add test for Settings
2023-05-12 23:34:02 +05:30
Mayur Singal
ef7b02529d
Remove sqlalchemy-ibmi from db2 dependency (#11553)
* Remove sqlalchemy-ibmi from db2 dependency

* remove from json schema

* add migration

* update what's new
2023-05-11 15:03:26 +02:00
Suresh Srinivas
14b622efd0
11287 Task - Add retention period and customer properties to Database, Schema, and Tables (#11511) 2023-05-10 16:08:21 -07:00
Louie
5bce6447f6
Fixed typo SMPTS to SMTPS (#11487)
* Fixed typo SMPTS to SMTPS

* sonar lint fixes

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-05-09 11:22:20 +05:30
Nahuel
b2f2320145
Fix#10584: Update Lineage API to support lineage from table to dashboard data model (#11489)
* Fix: Update Lineage API to support lineage from table to dashboard data model

* Minor change
2023-05-08 22:02:11 -07:00
Pere Miquel Brull
80af2b20bb
Centralize SSL Schemas & Add SSL for Airflow REST (#11363) 2023-05-08 21:05:10 +02:00
Mohit Yadav
498873c011
Data insight reporting + extras (#11317)
* Refactor Data Insight Chart for Reusability

* Data Insight Reporting

* Add Schedule Info

* Add Error Handling

* Add Profile Field

* Remove Endpoint as required field

* Missing Language

* Reformat

* Remove Task and Test Result Settings

* Fix test

* fix: typescript errors

* fix: send alertType for subscriptions request

* feat: add alert data insights report page

* Fix test

* fix: alert type typescript error

* fix: loading flicker on alerts page

* chore: add placeholders for data insight report alert page

* fix: alert icon styling

* fix: confirmation is not showing up for delete operation for subscription.

* fix: delete operation is not working from alert details page

* fix Activity Feed Errors on update

* fix: test

* chore: change the description text area to the markdown editor

* Data Insight Reporting should be created on system startup

* Restrict DataInsight Report to only one

* Review Comments

* feat: show the default DataInsightReport data

* sonar fixes

* sonar fixes

* deprecate ConfirmationModal

* trigger should be updated

* feta: add edit flow for data insight report

* remove custom option for schedule info

* chore: update data insight report icon

* test: add unit test for add data insight alert form

* fix: update cypress tests

* update local message

* test: add unit tests

* move sql changes to a different file

* spacing issues

* add tag provider in createRequest

* chore: send provider in create and update alert request.

* fix: code smells

* fix: comment typo

* fix: permission issue for event subscription

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-05-08 16:54:45 +05:30
Mayur Singal
e20f75fa54
Fix #11468: Fix salesforce scheme issue (#11474) 2023-05-08 14:02:17 +05:30
Nahuel
b22192575d
Fix#11216: Use Enum for SASL Mechanism for Kafka and Redpanda connections (#11385)
* Fix#11216: Use ENUM for SASL Mechanism for Kafka and Redpanda connections

* Minor change

* Address PR comments

* Fix postgres migration
2023-05-08 09:35:49 +02:00
Mohit Yadav
1b7fb192d0
[Reindexing] Fixes and Addition for Reindexing (#11447)
* Remove Entries on Reindexing

* Fix Reindexing Count Issues

* Reindexing Cursor Print in case of error

* Add afterCursor to Request

* add comment for function
2023-05-05 14:19:51 +05:30
Nahuel
1ec6e5e285
Fix#11311: Add IBM dependency for i Series in DB2 connector (#11381) 2023-05-02 15:50:39 +02:00
Suresh Srinivas
fd243dc5cb
FeedDAO cleanup (#11369)
* Cleanup FeedDAO implmentation

* Use FeedFilter to avoid repeated code for forward/reverse pagination
2023-04-30 16:50:29 -07:00