2200 Commits

Author SHA1 Message Date
mohitdeuex
f51781c32d Missing try catch for es-migrate option 2023-06-30 11:58:21 +05:30
07Himank
7244c3f530
changed enum from Elastic/Open Search to elastic/open search (#12237) 2023-06-29 20:13:49 -07:00
Suresh Srinivas
b88f5519ad
Move storing common relationships to base EntityRepository (#12164) 2023-06-29 19:54:05 -07:00
Mohit Yadav
6410fcbea4
Fix Alert Intermittent Issues for some entities (#12189)
* Fix Alert Intermittent Issues for some entities

* Fix Failing test for disabled publisher

* Fix Failing test due to deleted entity

* Fix Tests

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-06-29 19:00:54 -07:00
MihirR123
41f6945cfc
Fixes #11821 Incorrect swagger documentation for Pipelines / Add status data (#12216) 2023-06-29 12:15:54 -07:00
Mohit Yadav
a8c6f8bce0
Revert "Test suite index task(addded test_suite_search_index) (#12184)" (#12233)
This reverts commit a7ef0a82b1df4ec103fcbf8f2573ed907483060f.
2023-06-29 20:18:49 +05:30
07Himank
70b76a34c1
updated the aggregate from owner.fullyQualifiedName.keyword to owner.… (#12219)
* updated the aggregate from owner.fullyQualifiedName.keyword to owner.displayName.keyword

* fix owner search for filters

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-06-29 17:06:32 +05:30
07Himank
405896e009
updated es to 7.16.3 (#12218) 2023-06-29 16:03:51 +05:30
Teddy
2963fd1137
Move table list empty test suite to dao (#12212)
* fix: moved testSuite filter in table list to DAO

* fix: fixed tests as we won't filter out test suite with no test cases

* fix: handled null queryParam

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-06-29 15:53:07 +05:30
07Himank
a7ef0a82b1
Test suite index task(addded test_suite_search_index) (#12184)
* added search test case index

* checkstyle

* working on adding test suite index

* addes test_suite_search_index

* removed comments

* improvemtn

* fixes

* added test_suite_index_mapping for jp and zh languages

* check style
2023-06-28 23:08:10 -07:00
Sriharsha Chintalapani
d346101249
Update fqn size (#12188)
* Fix #10198: Filter tags and classifications if the classification is disabled

* Fix #10198: Fix tests

* Fix #10198: Fix tests

* Fix #10198: Fix Postgres tests

* Fix #10198: Fix Postgres tests

* Fix fqnHash column size to be 768

* Fix Failing test due to primary_key size limit

---------

Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-06-29 00:31:10 +05:30
Teddy
37a7cf99b6
fix: test defintion migration for parameter definition changes (#12207) 2023-06-28 16:02:49 +00:00
07Himank
88abdc0a98
Added owner aggregations (#12205)
* added owner aggregation and ignore setting query_filter if its empty

* added owner aggregation and ignore setting query_filter if its empty

* style
2023-06-28 07:07:21 -07:00
07Himank
569539f280
updated elastic search version from 7.16.0 to 7.16.1 (#12203)
* updated elastic search version from 7.16.0 to 7.16.1

* dummy commit

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-06-28 07:05:00 -07:00
Sriharsha Chintalapani
47440849bd
Fix #10198: Filter tags and classifications if the classification is disabled (#12144)
* Fix #10198: Filter tags and classifications if the classification is disabled

* Fix #10198: Fix tests

* Fix #10198: Fix tests

* Fix #10198: Fix Postgres tests

* Fix #10198: Fix Postgres tests

* Fix #12083 - Backoff retry when polling for pipeline service client status (#12105)

* Fix #12083

* Fix #12083

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>

* Databricks Support Table Constraints (#12138)

* Databricks Support Table Constraints

* pylint fix

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

* Fix #11659: Add support for filter patterns in dbt workflow (#12063)

* Move Top Level Imports (#12145)

* interfaces for db migrations (#12057)

* interfaces for db migrations

* Add Data Migration for entities to 1.1

* Migration for FQN hash

* Added Tag Usage And Fixed some transaction issue in testing

* Added SERVER_CHANGE_LOG validations

* Update Path for files

* Added Postgres Migration

* Test Suite Migration [WIP]

* remove ingestion pipeline

* moved Migration to Table Initializer

* init Data source config in TableInitializer

* make checksum ignore configurable

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>

* Fix Migrations (#12146)

* Mask queries, Table sample data and Topic sample data (#12139)

* Filter out PII queries

* Mask topic PII

* Mask topic PII

* Update sample data test

* Format

* Moved logic to repository or PIIMasker

* Fix test

* chore(ui): update sample data api endpoint for topic

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>

* Dockerfile Update (#11066)

* Updating Dockerfile with multistage

* Updating reviewed changes

* Docker Development changes

* Docker workflow changes

* Arguments update

* Script path update

* Arguments update

* Resolving the reviewed suggestions

* Docker script update (#12115)

* Create manual.yml

* Fixing Typo

* Removing manual test github action file

* Testing IP runner

* Updating docker script for sandbox db task

* removing the ip test action

* Updating docker script for sandbox db task postgres

* Volume cleaning argument addition

* Volume cleaning argument addition

* Fix Failing Maven CI Test (#12151)

* Fix SQLLineage Test (#12152)

* Docker server workflow (#12157)

* Updating Dockerfile with multistage

* Updating reviewed changes

* Docker Development changes

* Docker workflow changes

* Arguments update

* Script path update

* Arguments update

* Resolving the reviewed suggestions

* updating workflow syntax

* remvoed ibm db arch (#12158)

* fix: resolve general feedbacks (#12153)

* fix: feedbacks

* fix: add tier as static item

* fix: feedbacks

* fix: add localization

* fix: feedbacks

* fix: add localization

* fix: breadcrumb wrapping issue

* fix: alignment issue for glossary

* fix: border issue in drawer

* ui: worked on DQ feedback part 3 (#12149)

* ui: worked on DQ feedback part 3

* updated name of test suite ingestion name
- Added description field in test case expand view
- Added functionality to update displayname of test case

* added testCase in re-indexing

* integrated permission in DQ

* fix: check on undefined bool value of clause (#12162)

* chore(ui): task feedback (#12154)

* chore(ui): task feedback

* fix copy notification info

* fix code font

* fix glue pipeline

* fix unit tests

* fix cy failures

* Update CODEOWNERS for UI code

* Address backend DQ feedback (#12150)

* fix: add jp and zh index

* fix: updated testCase tooltips

* fix: changed list testSuite permission to VEW_TEST + set owner of table for executable testSuite

* Run System Migrations on CREATE

* Update logging messages for JWT tokens (#12169)

* Update Test Case Activity Feed Messaging

* Auto tagging Windows troubleshooting (#12173)

* feat(ui): supported pagination and search in tags and term select for both table and entity level (#12155)

* supported pagination and search in tags and term select for both table and entity level

* supported pagination of tags in glossary overiew section for tags

* fix cypress and address comments

* fix cypress issue

* fixed(ui): updated tour flow and feedback (#12171)

* fixed(ui): updated tour flow and feedback

* translation-sync

* addressing comments

* feat: added view definition tab in table details page (#12172)

* Update profiler dq doc (#12182)

* doc: update documentation for profiler and dq

* doc: renamed image path from 1.0.0 to 1.1.0

* fix connection kwargs error in profiler (#12181)

* Fix Profiler Workflow infinite loop (#12185)

* Fix Profiler Workflow infinite loop

* fix: scrolling behavior feedback on mydata page (#12176)

* fix(ui): task layout update (#12175)

* fix(ui): task layout update

* fix title for edit task

* remove local for edit-task

* feat(ui): supported pagination of glossary terms in glossary overview section (#12177)

* supported pagination of glossary terms in glossay overview section

* remove unwanted try catch

* change as per comments

* Update S3 storage permissions (#12064)

* Add disabled flag to search mappings

* Revert unwanted changes

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Anuj359 <anuj.j@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2023-06-28 13:30:09 +05:30
mohitdeuex
c5b1619eb5 Update Test Case Activity Feed Messaging 2023-06-27 18:18:22 +05:30
mohitdeuex
6735db8ae7 Run System Migrations on CREATE 2023-06-27 17:18:31 +05:30
Teddy
36dd1b6b2d
Address backend DQ feedback (#12150)
* fix: add jp and zh index

* fix: updated testCase tooltips

* fix: changed list testSuite permission to VEW_TEST + set owner of table for executable testSuite
2023-06-27 11:57:05 +05:30
Mohit Yadav
80b273a559
Fix Failing Maven CI Test (#12151) 2023-06-26 16:01:22 +05:30
Pere Miquel Brull
a9135e8db5
Mask queries, Table sample data and Topic sample data (#12139)
* Filter out PII queries

* Mask topic PII

* Mask topic PII

* Update sample data test

* Format

* Moved logic to repository or PIIMasker

* Fix test

* chore(ui): update sample data api endpoint for topic

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-26 11:47:39 +02:00
Sriharsha Chintalapani
33c752d2f5
interfaces for db migrations (#12057)
* interfaces for db migrations

* Add Data Migration for entities to 1.1

* Migration for FQN hash

* Added Tag Usage And Fixed some transaction issue in testing

* Added SERVER_CHANGE_LOG validations

* Update Path for files

* Added Postgres Migration

* Test Suite Migration [WIP]

* remove ingestion pipeline

* moved Migration to Table Initializer

* init Data source config in TableInitializer

* make checksum ignore configurable

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-06-26 13:03:44 +05:30
Pere Miquel Brull
e13a5de5ae
Fix #12083 - Backoff retry when polling for pipeline service client status (#12105)
* Fix #12083

* Fix #12083

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-06-26 06:52:21 +02:00
Teddy
efec8a1019
fix: hash name of test suite + append (#12125)
* fix: hash name of test suite + append

* fix: ran java linting

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-06-25 19:14:34 -07:00
Esmaeil Chitgar
c8b3efa205
Remove duplicated line in OpenMetadata-service pom.xml (#12134)
Remove duplicated line in OpenMetadata-service pom.xml.<sonar.tests>${project.basedir}/src/test/java</sonar.tests>
2023-06-25 10:36:15 -07:00
MihirR123
0ceb856aeb
Fix sonar issues for static variable names (#12111)
Co-authored-by: Mihir Rao <mihir@MacBook-Pro-76.local>
2023-06-22 20:32:49 -07:00
Teddy
712dcf8587
fix: added field to testCase entity to list all Test Suites (#12093)
* fix: added  field to testCase entity to list all the linked test suites (executable and logical)

* fix: resolved conflicts from main

* fix: removed unecessary comment
2023-06-22 18:00:19 +02:00
07Himank
3e3bd79f89
added search for test case index (#12100)
* added search test case index

* checkstyle
2023-06-22 19:14:32 +05:30
07Himank
9355c0677d
Open search elastic search client task (#11879)
* Added diffrent clients for elasticSearch and openSearch and did the interface implementation

* removed some comments and some changes

* Add Elastic Search Test Container

* upgarded es version

* added test cases for elastic search

* added test case for, when tag is deleted cehck its relationships also get deleted

* changed package of open,elastic to openSearch and elasticSearch

* added support for skip search test case if index is not present of that entity

* Refactor bunch of files

* unnecessary semicolon

* refactor test case indexing

* fix checkstyle

* checkstyle

* added hardcoded configs and changed package name

* fixed no response from hits

* stopped elasticsearch container and some fixes

* closed elasticsearch container

* revert

* closed client after used in test

* style

* edit es config in junit

* checking if using already es version release memory for maven ci

* revert docker file changes

* remove es container for testing the ci

* commented out all test cases for elasticsearch

* fix

* creating elastic search with reusable enabled

* trying to fix ci error

* start container only if it is not running

* revert docker changes and done changes for elastic search container

* changed var name of sort_order for es

* lower the version of es client

* fixes

* all latest version of es including docker

* Use Single sort order

* Fix Keyword issue

* Immediate refresh policy for entity created

* increase sleep time

* disable test

* added reindexing logic in run_local_docker.sh

* enabled test case

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-06-22 16:55:05 +05:30
mohitdeuex
a9a830a93a Fix failine reindexing issue 2023-06-22 14:26:46 +05:30
Pere Miquel Brull
520d5e2657
Add Pipeline Service Status Job to publish to Prometheus (#12046)
* Add Pipeline Service Status Job to publish to Prometheus

* Add Pipeline Service Status Job to publish to Prometheus

* Add config for health check interval
2023-06-22 06:53:56 +02:00
Teddy
6e92dc6938
Fixes #12069 - Test Case Summary for testSuite endpoint return 0 (#12076) 2023-06-21 20:53:18 +02:00
Ayush Shah
9fa7b78d87
Issue 11686: Add IncludeOwner + scheduleInterval in Airflow (#12005) 2023-06-21 14:18:45 +05:30
Pere Miquel Brull
bfa37df298
Add platform in Pipeline Service status msg (#12041)
* Add platform in Pipeline Service status msg

* chore(ui): show message based on platform

* update the styling

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-21 07:21:19 +02:00
Pere Miquel Brull
3c430ca722
Fix #11966 - Add delete secrets options for managed SM (#12051)
* Add delete secrets options for managed SM

* Align tests with mysql auth changes
2023-06-21 06:59:02 +02:00
Chirag Madlani
ef578ecb40
feat(ui): support disable option for tags and classification (#10925)
* feat(ui): support disable option for tags and classification

* backend changes for PATCH API to work for updating disabled flag in classifications and tags

* made UI changes to provide enable/disable option for classifications and tags

* localization changes for other languages

* Improved tags disabling functionality

* changes to disable the system tags
fixed permission checks while editing tags

* localization change for other languages

* fixed unit tests and minor improvements

* fixed failing cypress tests

* disabled the edit actions for the disabled classifications

* localization changes for other languages

* changes to use disabled badge

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2023-06-21 00:13:05 +05:30
Mohit Yadav
131967a53a
Bug fixes for Activity Feed + Restrict digits to 2' place (#12050)
* Fix Activity Feed Ownership Exception, + field specific parsing issue in case of different entities

* Restrict Percent Change to 2 Digits
2023-06-20 23:56:48 +05:30
Pere Miquel Brull
e6f9150cdf
Revert "Add delete secrets options for managed SM (#11972)" (#12049)
This reverts commit 79405080b3427db2ffe1752c6500eca6cbd3b28c.
2023-06-20 15:46:53 +02:00
Pere Miquel Brull
79405080b3
Add delete secrets options for managed SM (#11972) 2023-06-20 15:28:03 +02:00
Mayur Singal
c3cec54be9
Fix for #11807 Part 2: Add SourceUrl for table entity (#12013)
* Fix #11807 Part 2: Add SourceUrl for table entity

* address review comments: centralize sourceurl

* remove qlick

* pytest fix

* fix typo
2023-06-20 11:46:45 +02:00
Pere Miquel Brull
8751427679
Add project to Data Models JSON Schema & Implement in Looker (#12030)
* Add project to dashboards and models

* Add project to dashboards and models

* Update openmetadata-spec/src/main/resources/json/schema/entity/data/dashboardDataModel.json

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>

* Update openmetadata-spec/src/main/resources/json/schema/api/data/createDashboardDataModel.json

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>

* Fix tests

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-06-20 11:45:29 +02:00
Suresh Srinivas
e09537eeeb
Fixes #11883 - A user can resolve a task created by him (#11998)
* Fixes #11883 - A user can resolve a task created by him

* Test failure fixes
2023-06-19 16:29:05 -07:00
Mohit Yadav
7adf1ec2b5
Reindex improvement (#12036)
* Reindex added for TestCase

* Add Mapping Api endpoint
2023-06-19 23:21:29 +05:30
Teddy
76f5d3d571
Fixes #11994 - Update dbt and GE integration with new DQ flow (#12018)
* feat: updated GE integration to match new test workflow

* feat: updated unit tests to match new signature

* feat: added GE integration tests

* feat: ran python linting

* feat: updated dbt ingestion to match new TestSuite workflow

* feat: ran python linting

* feat: remove testSuite from Elasticsearch event test case update

* feat: ran java linting
2023-06-19 15:05:51 +02:00
Sriharsha Chintalapani
9dbaabad44
Adding MD5 hash to fullyQualifiedName and names to simplify DB indexes for lookups and increasing the size of FQN (#11960)
* Fix fqn hash

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* merge commits

* merge commits

* merge commits

* Fix glossary import/export

* Fix BotResource Tests

* Fix BotResource Tests

* Fix Glossary Term tests

* Fix Glossary Term tests

* Fix Glossary Import/Export tests

* All backend tests are fixed.

* merge with main

* Fix tests and revert local changes

* Fix tests and revert local changes

* Fix NullPointerException for Glossary and Query

* Fix Tests

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-06-19 15:43:05 +05:30
Mayur Singal
8ecd88fd1b
Fix #11312: Deprecate ES Reindex on Ingestion and use reindex APIs instead (#11988) 2023-06-19 12:45:18 +05:30
Sriharsha Chintalapani
146c56abab
Fix #12014: Register OpenMetadata as a service when the server starts for the first time (#12015) 2023-06-16 16:39:55 -07:00
Sriharsha Chintalapani
817c5f3d4a
Add get tableProfilerConfig test (#9324) 2023-06-16 16:02:38 +05:30
Mohit Yadav
41c49b9de8
Data Insight Report Job Bug Fixes (#11975)
* Data Insight Report Job Bug Fixes

* ui: add enabled field

* test case fix

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-16 15:40:44 +05:30
Ayush Shah
f80eaf3a26
Fixes 11068: mysql & postgres iam auth (#11937) 2023-06-16 13:18:12 +05:30
Teddy
3f01ee938f
Fixes #11895 - Add Indexing and Search logic for TestCases (#11989)
* feat: added logic to delete logical test suite + added check to prevent adding existing testCases to executable test suite

* feat: added elasticsearch index logic for testCases

* feat: added deletion logic from index logic when deleting test suites

* feat: added test case index search to  endpoint

* feat: add executable/logical filter in list testSuite + filterOut tables without tests in Table resource

* feat: added summary field to testSuite

* feat: added executionSummary endpoint for test cases

* feat: removed tick marks around timestamp

* feat: addressed test failures

* feat: ran python linting

* feat: add limit to fetch all tables in TableResource testSuite test

* feat: fix conflict

* feat: ran java checkstyle

* feat: fixed mongo linting + disabled mongo failing tests

* feat: removed mongo test skip

* feat: removed unsued pytest import
2023-06-15 21:27:54 +02:00