532 Commits

Author SHA1 Message Date
Mohit Yadav
80b273a559
Fix Failing Maven CI Test (#12151) 2023-06-26 16:01:22 +05:30
Pere Miquel Brull
a9135e8db5
Mask queries, Table sample data and Topic sample data (#12139)
* Filter out PII queries

* Mask topic PII

* Mask topic PII

* Update sample data test

* Format

* Moved logic to repository or PIIMasker

* Fix test

* chore(ui): update sample data api endpoint for topic

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-26 11:47:39 +02:00
Sriharsha Chintalapani
33c752d2f5
interfaces for db migrations (#12057)
* interfaces for db migrations

* Add Data Migration for entities to 1.1

* Migration for FQN hash

* Added Tag Usage And Fixed some transaction issue in testing

* Added SERVER_CHANGE_LOG validations

* Update Path for files

* Added Postgres Migration

* Test Suite Migration [WIP]

* remove ingestion pipeline

* moved Migration to Table Initializer

* init Data source config in TableInitializer

* make checksum ignore configurable

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-06-26 13:03:44 +05:30
Pere Miquel Brull
e13a5de5ae
Fix #12083 - Backoff retry when polling for pipeline service client status (#12105)
* Fix #12083

* Fix #12083

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-06-26 06:52:21 +02:00
Teddy
efec8a1019
fix: hash name of test suite + append (#12125)
* fix: hash name of test suite + append

* fix: ran java linting

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-06-25 19:14:34 -07:00
Esmaeil Chitgar
c8b3efa205
Remove duplicated line in OpenMetadata-service pom.xml (#12134)
Remove duplicated line in OpenMetadata-service pom.xml.<sonar.tests>${project.basedir}/src/test/java</sonar.tests>
2023-06-25 10:36:15 -07:00
MihirR123
0ceb856aeb
Fix sonar issues for static variable names (#12111)
Co-authored-by: Mihir Rao <mihir@MacBook-Pro-76.local>
2023-06-22 20:32:49 -07:00
Teddy
712dcf8587
fix: added field to testCase entity to list all Test Suites (#12093)
* fix: added  field to testCase entity to list all the linked test suites (executable and logical)

* fix: resolved conflicts from main

* fix: removed unecessary comment
2023-06-22 18:00:19 +02:00
07Himank
3e3bd79f89
added search for test case index (#12100)
* added search test case index

* checkstyle
2023-06-22 19:14:32 +05:30
07Himank
9355c0677d
Open search elastic search client task (#11879)
* Added diffrent clients for elasticSearch and openSearch and did the interface implementation

* removed some comments and some changes

* Add Elastic Search Test Container

* upgarded es version

* added test cases for elastic search

* added test case for, when tag is deleted cehck its relationships also get deleted

* changed package of open,elastic to openSearch and elasticSearch

* added support for skip search test case if index is not present of that entity

* Refactor bunch of files

* unnecessary semicolon

* refactor test case indexing

* fix checkstyle

* checkstyle

* added hardcoded configs and changed package name

* fixed no response from hits

* stopped elasticsearch container and some fixes

* closed elasticsearch container

* revert

* closed client after used in test

* style

* edit es config in junit

* checking if using already es version release memory for maven ci

* revert docker file changes

* remove es container for testing the ci

* commented out all test cases for elasticsearch

* fix

* creating elastic search with reusable enabled

* trying to fix ci error

* start container only if it is not running

* revert docker changes and done changes for elastic search container

* changed var name of sort_order for es

* lower the version of es client

* fixes

* all latest version of es including docker

* Use Single sort order

* Fix Keyword issue

* Immediate refresh policy for entity created

* increase sleep time

* disable test

* added reindexing logic in run_local_docker.sh

* enabled test case

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-06-22 16:55:05 +05:30
mohitdeuex
a9a830a93a Fix failine reindexing issue 2023-06-22 14:26:46 +05:30
Pere Miquel Brull
520d5e2657
Add Pipeline Service Status Job to publish to Prometheus (#12046)
* Add Pipeline Service Status Job to publish to Prometheus

* Add Pipeline Service Status Job to publish to Prometheus

* Add config for health check interval
2023-06-22 06:53:56 +02:00
Teddy
6e92dc6938
Fixes #12069 - Test Case Summary for testSuite endpoint return 0 (#12076) 2023-06-21 20:53:18 +02:00
Ayush Shah
9fa7b78d87
Issue 11686: Add IncludeOwner + scheduleInterval in Airflow (#12005) 2023-06-21 14:18:45 +05:30
Pere Miquel Brull
bfa37df298
Add platform in Pipeline Service status msg (#12041)
* Add platform in Pipeline Service status msg

* chore(ui): show message based on platform

* update the styling

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-21 07:21:19 +02:00
Pere Miquel Brull
3c430ca722
Fix #11966 - Add delete secrets options for managed SM (#12051)
* Add delete secrets options for managed SM

* Align tests with mysql auth changes
2023-06-21 06:59:02 +02:00
Chirag Madlani
ef578ecb40
feat(ui): support disable option for tags and classification (#10925)
* feat(ui): support disable option for tags and classification

* backend changes for PATCH API to work for updating disabled flag in classifications and tags

* made UI changes to provide enable/disable option for classifications and tags

* localization changes for other languages

* Improved tags disabling functionality

* changes to disable the system tags
fixed permission checks while editing tags

* localization change for other languages

* fixed unit tests and minor improvements

* fixed failing cypress tests

* disabled the edit actions for the disabled classifications

* localization changes for other languages

* changes to use disabled badge

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2023-06-21 00:13:05 +05:30
Mohit Yadav
131967a53a
Bug fixes for Activity Feed + Restrict digits to 2' place (#12050)
* Fix Activity Feed Ownership Exception, + field specific parsing issue in case of different entities

* Restrict Percent Change to 2 Digits
2023-06-20 23:56:48 +05:30
Pere Miquel Brull
e6f9150cdf
Revert "Add delete secrets options for managed SM (#11972)" (#12049)
This reverts commit 79405080b3427db2ffe1752c6500eca6cbd3b28c.
2023-06-20 15:46:53 +02:00
Pere Miquel Brull
79405080b3
Add delete secrets options for managed SM (#11972) 2023-06-20 15:28:03 +02:00
Mayur Singal
c3cec54be9
Fix for #11807 Part 2: Add SourceUrl for table entity (#12013)
* Fix #11807 Part 2: Add SourceUrl for table entity

* address review comments: centralize sourceurl

* remove qlick

* pytest fix

* fix typo
2023-06-20 11:46:45 +02:00
Pere Miquel Brull
8751427679
Add project to Data Models JSON Schema & Implement in Looker (#12030)
* Add project to dashboards and models

* Add project to dashboards and models

* Update openmetadata-spec/src/main/resources/json/schema/entity/data/dashboardDataModel.json

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>

* Update openmetadata-spec/src/main/resources/json/schema/api/data/createDashboardDataModel.json

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>

* Fix tests

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-06-20 11:45:29 +02:00
Suresh Srinivas
e09537eeeb
Fixes #11883 - A user can resolve a task created by him (#11998)
* Fixes #11883 - A user can resolve a task created by him

* Test failure fixes
2023-06-19 16:29:05 -07:00
Mohit Yadav
7adf1ec2b5
Reindex improvement (#12036)
* Reindex added for TestCase

* Add Mapping Api endpoint
2023-06-19 23:21:29 +05:30
Teddy
76f5d3d571
Fixes #11994 - Update dbt and GE integration with new DQ flow (#12018)
* feat: updated GE integration to match new test workflow

* feat: updated unit tests to match new signature

* feat: added GE integration tests

* feat: ran python linting

* feat: updated dbt ingestion to match new TestSuite workflow

* feat: ran python linting

* feat: remove testSuite from Elasticsearch event test case update

* feat: ran java linting
2023-06-19 15:05:51 +02:00
Sriharsha Chintalapani
9dbaabad44
Adding MD5 hash to fullyQualifiedName and names to simplify DB indexes for lookups and increasing the size of FQN (#11960)
* Fix fqn hash

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* merge commits

* merge commits

* merge commits

* Fix glossary import/export

* Fix BotResource Tests

* Fix BotResource Tests

* Fix Glossary Term tests

* Fix Glossary Term tests

* Fix Glossary Import/Export tests

* All backend tests are fixed.

* merge with main

* Fix tests and revert local changes

* Fix tests and revert local changes

* Fix NullPointerException for Glossary and Query

* Fix Tests

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-06-19 15:43:05 +05:30
Mayur Singal
8ecd88fd1b
Fix #11312: Deprecate ES Reindex on Ingestion and use reindex APIs instead (#11988) 2023-06-19 12:45:18 +05:30
Sriharsha Chintalapani
146c56abab
Fix #12014: Register OpenMetadata as a service when the server starts for the first time (#12015) 2023-06-16 16:39:55 -07:00
Sriharsha Chintalapani
817c5f3d4a
Add get tableProfilerConfig test (#9324) 2023-06-16 16:02:38 +05:30
Mohit Yadav
41c49b9de8
Data Insight Report Job Bug Fixes (#11975)
* Data Insight Report Job Bug Fixes

* ui: add enabled field

* test case fix

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-16 15:40:44 +05:30
Ayush Shah
f80eaf3a26
Fixes 11068: mysql & postgres iam auth (#11937) 2023-06-16 13:18:12 +05:30
Teddy
3f01ee938f
Fixes #11895 - Add Indexing and Search logic for TestCases (#11989)
* feat: added logic to delete logical test suite + added check to prevent adding existing testCases to executable test suite

* feat: added elasticsearch index logic for testCases

* feat: added deletion logic from index logic when deleting test suites

* feat: added test case index search to  endpoint

* feat: add executable/logical filter in list testSuite + filterOut tables without tests in Table resource

* feat: added summary field to testSuite

* feat: added executionSummary endpoint for test cases

* feat: removed tick marks around timestamp

* feat: addressed test failures

* feat: ran python linting

* feat: add limit to fetch all tables in TableResource testSuite test

* feat: fix conflict

* feat: ran java checkstyle

* feat: fixed mongo linting + disabled mongo failing tests

* feat: removed mongo test skip

* feat: removed unsued pytest import
2023-06-15 21:27:54 +02:00
Onkar Ravgan
d08c928801
Added project property to dashboards (#11986)
* Added projects to dashboards

* Added powerbi proj

* merge conflicts after source url

* fixed mongo pytest
2023-06-15 21:23:43 +05:30
Mayur Singal
82a0222257
SourceUrl changes for dashboard, pipeline & chart entities (#11991) 2023-06-15 14:44:48 +05:30
Pere Miquel Brull
75aa3d5fa1
Fix #10542 & #10830 - Mask Sample Data PII & Column Metrics (#11919)
* Mask Sample Data PII

* Mask column profile

* Mask Test Cases

* Mask Test Cases

* Add tests

* Format

* Format

* Fix test case resource masking
2023-06-15 09:13:57 +02:00
Mayur Singal
7fa963eec3
Fix #1076: Add mongodb support (#11943) 2023-06-15 11:14:22 +05:30
Onkar Ravgan
d409c339e4
Fix 10689: Added Powerbi Datamodels (#11923) 2023-06-14 17:00:08 +05:30
Suresh Srinivas
7d9b27b996
Fixes #11976 Add support for suggest and request description and tags for all the entities (#11977) 2023-06-13 12:45:07 -07:00
Pere Miquel Brull
ba5f929f77
Add fine-grained health validation for Airflow APIs (#11792)
* Add fine-grained health validation for Airflow APIs

* Add ingestion version response

* Improve messaging

* Format

* Format

* Update response

* ui:update the hook to return airflow status based on response status field

* ui: add unit test for useAirflowStatus hook

* chore: only show test connection if airflow is available

* feat: add airflow message banner

* chore: update icon and background color

* chore: update typography to text

* test: add unit test

* address comments

* chore: show banner on service detail page

* fix: update test suite api workflow to reflect new implementation

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-06-12 07:47:45 +02:00
07Himank
62af9bb633
fixed issue for lineage description (#11500)
* fixed issue for lineage description

* fixed issue while ingesting

* fixed issue while ingesting

* added test case for Lingeage with description

* addressing comments .. enhancement

* addressing comments .. enhancement

* modified py test case and removed description from addLineage as we are not using it.

* add support for topic entity and description in lineage details

* fix pylint & test

* pytest fix

* fix column lineage null issue

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-06-12 11:17:32 +05:30
Pere Miquel Brull
20ebe70d9e
Update test conn definition on initialize (#11940) 2023-06-09 14:06:55 +02:00
Pere Miquel Brull
00f07190e9
Change ES mapping maximumMessageSize & retentionSize to long (#11907) 2023-06-07 12:50:24 +02:00
Sriharsha Chintalapani
4ecbbcd43c
Fix Japanese language indexing (#11903) 2023-06-07 07:22:55 +02:00
Suresh Srinivas
f23f67dfe9
Fixes #11617 Entity get operations check for ViewAll operation instead of specific view operations based on the field requested (#11904) 2023-06-06 15:18:57 -07:00
Mohit Yadav
57e44c22be
Make Resource Packages Dynamic (#11891) 2023-06-07 01:02:52 +05:30
mohitdeuex
d43ca31891 Add UUID on Metadata Service Creation in case of CREATE 2023-06-07 00:16:00 +05:30
Sriharsha Chintalapani
809fbbf536
Fix #11348: Return a proper error message when a token with wrong keyID sent (#11509)
* Fix #11348: Return a proper error message when a token with wrong keyID is sent

* Fix #11348: Return a proper error message when a token with wrong keyID is sent

* fix formatting

---------

Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2023-06-06 11:37:39 -07:00
Pere Miquel Brull
b68d063694
Update Test Connection Definition FQN (#11861)
* Update Test Connection Definition FQN

* chore: use object literal for string concatenation

* Fix test

* Update Test Connection Definition FQN

* Update Test Connection Definition FQN

* Update Test Connection Definition FQN

* Fix test

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-06 15:22:32 +02:00
Teddy
4b9f213dbf
Fixes Issue #11863 - Add Status to DQ (#11893)
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.

* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity

* feat: add status object to test case results

* feat: ran python linting

* feat: fixed  update to
2023-06-06 10:09:16 +00:00
Teddy
721869428e
Revert "Fixe Issue #11863 - Add Status logic for test case results (#11881)" (#11892)
This reverts commit 06735fe8dbaac5b267c9a2cf744ca154f88a9247.
2023-06-06 09:56:12 +02:00