1723 Commits

Author SHA1 Message Date
Ayush Shah
9880f06b2c
Fixes #17489: Allow non numeric numbers to be sent via Json, Replace NaN value… (#17490)
* fix: Allow non numeric numbers to be sent via Json, Replace NaN values with None in SQAProfilerInterface

Replace NaN values with None in the SQAProfilerInterface class to maintain database parity. NaN values will be cast to null in OpenMetadata. This change ensures that data handling processes account for this conversion.

* fix: histogram overflow error

* test: Add Unit Test for Null and Null Ratio Metric

* chore: Address comments

* chore: Address comments

* fix: checkstyle and message

* fix: failing tests as null count works as expected
2024-08-20 16:33:55 +05:30
Pere Miquel Brull
f7aab6b7a4
MINOR - Fix list API for automation workflows (#17494)
* MINOR - Fix list API for automation workflows

* MINOR - Fix list API for automation workflows
2024-08-20 12:40:21 +02:00
sonika-shah
d587e6f8e6
Fix : Propagation of owners and domain in search (#17400)
* Fix : Propagation of owners and domain in search

* update test

* optimize ADD_REMOVE_OWNERS_SCRIPT update query es script

* optimizes script by checking for inherited fields(inherited = true) before updating or adding them.

* fix glossary verion spec by adding slow since, test contains steps

* added slow in test as they are exceding in the runs

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2024-08-19 23:11:56 +05:30
Imri Paran
7508848376
fix(dq): data types for unique columns (#17431)
1. remove json and array from supported data types of unique column test.
2. migrations.
3. tests.
2024-08-19 14:28:42 +02:00
Mohit Yadav
2b915a53e9
Fix Ldap Username issue (#17485) 2024-08-19 16:02:54 +05:30
sonika-shah
82d3a87556
Updated the listBefore and listAfter methods to use both name and id as cursors for pagination. (#17477) 2024-08-18 10:15:57 +05:30
Sriharsha Chintalapani
29745b2abe
Minor: Fix ranking in search results (#17460)
* Minor: fix patch tags on apiendpoint

* Fix #17118: Elasticsearch HTTP Method Error in update_by_query; Fix: patch tags for request/response schema in ApiEndpoint

* Minor: Fix Search ranking based on exact match, tiering, usage, total votes
2024-08-16 16:12:51 -07:00
Sriharsha Chintalapani
39203a1156
Fix #17352: Remove TablesInitializer and bootstrap_storage.sh (#17353)
* Fix #17352: Remove TablesInitializer and bootstrap_storage.sh

* Fix #17352: Remove TablesInitializer and bootstrap_storage.sh

* Fix #17352: Remove TablesInitializer and bootstrap_storage.sh
2024-08-15 19:27:02 -07:00
IceS2
455e66bc37
Fix Missing Users errors on Data Insights Pipeline (#17446) 2024-08-14 14:26:51 +02:00
IceS2
c221f80d15
Fix DataInsights Field Getter (#17427) 2024-08-13 16:28:33 +00:00
Mayur Singal
ac2264e9c7
MINOR: DI Simplify Formulas (#17382)
* MINOR: DI Similify Formulas

* checkstyle
2024-08-13 17:42:46 +05:30
Mohit Yadav
7033870e71
Update SAML to have signed Messages (#17415)
* Update SAML to have signed Messages

* test fix
2024-08-13 11:38:37 +05:30
Siddhant
6ba4d69127
added /refresh implementation for LDAP with a refresh token expiry time of 24 hours. (#17410) 2024-08-13 11:36:43 +05:30
Mohit Yadav
f7f30799d4
Update Max Payload Size (#17414) 2024-08-12 23:21:49 +05:30
Chirag Madlani
995320a9ec
fix(ui): data insight page feedbacks (#17390)
* fix(ui): data insight page feedbacks

* Ignore null counts

* adjust advance filter

* Revert toastNotification changes

* Fix typo

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2024-08-12 22:46:43 +05:30
Siddhant
330b97a669
fix fetch loginConfiguration. (#17402) 2024-08-12 19:37:58 +05:30
IceS2
7b191b891d
MINOR: Implement Multi Data Streams. Fix Duplicating issue (#17365)
* Implement Multi Data Streams. Fix Duplicating issue

* Fix Test for multi index search

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-08-12 14:09:13 +02:00
Mohit Yadav
df9753c7bb
Add Opensearch max payload Size (#17392) 2024-08-12 11:34:52 +05:30
Mohit Yadav
aa5b9b2985
Increase Payload Size to 100MIB for ELastic Search (#17366)
* Increase Payload Size to 100MIB

* Increase batchSize Size to 500

* Add Payload Size as option in openmetadataoperations , and searchIndexJob

* //Comment

* Add Logs
2024-08-11 21:58:55 +05:30
Sriharsha Chintalapani
a6ebac9b68
Fix #17118: Elasticsearch HTTP Method Error in update_by_query; Fix: patch tags for request/response schema in ApiEndpoint (#17383)
* Minor: fix patch tags on apiendpoint

* Fix #17118: Elasticsearch HTTP Method Error in update_by_query; Fix: patch tags for request/response schema in ApiEndpoint
2024-08-10 17:56:52 -07:00
sonika-shah
8d5661a7bf
Support Elasticsearch cluster alias for E2E tests  (#17117)
* Fix backend tests and have index alias with clusterAlias appended

* Fix backend tests and have index alias with clusterAlias appended

* Fix failing tests

* alias setup

* fix suggestion not working due to alias

* fix getIndexOrAliasName method for multiple indexes

* update openmetadata.yaml

* update childAliases with clusterAlias

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-08-10 11:26:49 -07:00
IceS2
e5a7cff5a5
Updated oracle to use DBA_ tables (#17274)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2024-08-10 16:58:21 +05:30
Mayur Singal
52a7f99145
MINOR: Data insight feedbacks (#17337)
* MINOR: DI add no tier in system chart

* checkstyle

* minor ui fixes

* tier filter fix

* ignore tags and glossaries from tier table

* enhance advance search provider

* Ignore data products as well

* Empty Filter Condition & Lowecase Owner Displayname

* opensearch

* sort graph data to confirm order of graph

* address comments

* handle domain in advanced filter

---------

Co-authored-by: Chira Madlani <chirag@getcollate.io>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-08-10 14:20:14 +05:30
Sriharsha Chintalapani
98229591b5
Minor: Validate apiEndpoint response schema fields (#17370) 2024-08-09 21:49:26 -07:00
IceS2
8cfa6915d2
Fix permission for getPipelineStatus (#17356) 2024-08-09 16:13:00 +02:00
Sriharsha Chintalapani
8d67382513
Minor: Fix Postgres missing policy rule migrations for EditOwners (#17351) 2024-08-08 23:13:57 -07:00
Onkar Ravgan
945c6b2137
Fix #16039: Fixed sourceHash patch for DatabaseSchema entity (#17309) 2024-08-09 11:10:21 +05:30
IceS2
717c9709e0
Update Data Insights Pipeline to change CustomProperty key based on entityType (#17350) 2024-08-08 18:50:27 -07:00
Sriharsha Chintalapani
9acdab0d49
Minor: inherit owner/domain for apiEndpoints (#17300)
* Minor: inherit owner/domain for apiEndpoints

* test: add playwright test for owner propagation

* Minor: inherit owner/domain for apiEndpoints

* add script to propagate owners in search index of children entity

* exclude propagateOwners for testcase

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
2024-08-08 10:42:10 -07:00
IceS2
9fcc8794a5
Remove sampleData and changeDescription from Data Insights documents (#17332) 2024-08-08 07:50:32 +02:00
Mohit Yadav
4c43509203
Check Expiry on Callback (#17326) 2024-08-07 15:28:03 +05:30
IceS2
66d7da05c4
MINOR: Iterate on data insights failure handling (#17324)
* Provide better errors on DataInsights Pipeline

* Improve DI Error Log

* Improve DI Error Log for OpenSearch

* Improve DI Error Log

* Fix Checkstyle
2024-08-07 11:14:55 +02:00
Mayur Singal
d3baf4fcf2
MINOR: DI Improve system charts (#17310) 2024-08-07 14:15:09 +05:30
sonika-shah
b6d3a2fbb3
Explore tree improvements (#17277)
* Explore tree improvements backend changes

* supported resizeable left panel in explore page

* change the more button feel to text

* fix playwright test for multi-owner

* minor changes

* fix playwright test

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2024-08-07 08:40:13 +02:00
sam-mccarty-mavenclinic
11a49ef08d
Fixes 16010: Support Looker ingestion from Gitlab (#17284)
* introduce gitlab option to lookml ingestion

* fix reader and disable test

* fix copy paste in test case

* fix file read and keyset pagination for tree

* fix credentials to include gitlab credentials

* uncomment arguments for unused credentials to fix validation error

* fix credentials test

* fix credentials test

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-08-07 06:53:44 +02:00
Mayur Singal
a579431e4a
MINOR: Fix DI most view data assets aggregation (#17288) 2024-08-07 00:34:54 +05:30
IceS2
0a8ea56c99
Provide better errors on DataInsights Pipeline (#17315) 2024-08-06 17:02:03 +02:00
Mohit Yadav
9bbc6a1801
Migrate NameHash (#17317) 2024-08-06 18:41:37 +05:30
Pere Miquel Brull
744ca6227c
MINOR - Clean Automator owner migration (#17294)
* MINOR - Clean Automator owner migration

* MINOR - Clean Automator owner migration
2024-08-06 09:02:09 +02:00
IceS2
6683fd1fbb
MINOR: Data insights provide better exceptions (#17292)
* Provide better errors

* Remove break; in order to allow the pipeline to process as much as they can
2024-08-06 08:08:26 +02:00
Siddhant
40009dbeae
Slack App Integration. (#17286)
* slack bolt app integration.

* checkstyle fix.

* fix slack uninstall.

* removed callback and events from JwtFilter.
2024-08-06 10:23:05 +05:30
Mohit Yadav
76bd4f3909
[Issue-17093] Add SAML refresh token in Refresh (#17113)
* Add SAML refresh token in Refresh

* update code

* Add Cookie for refresh token

* Add Max Age

* 1 day cookie age

* Set cookie path

* Add SAml Refresh Servlet

* add saml refresh

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2024-08-05 23:58:10 +05:30
Sriharsha Chintalapani
9fcbf6a5b9
Minor: Fix Policy rule migration due to owner -> owners (#17285) 2024-08-04 19:51:55 -07:00
IceS2
84083aef4a
Improve DI error log (#17283) 2024-08-02 09:11:27 -07:00
IceS2
5f67df6606
MINOR: Update data insights migration to make the flow easier (#17280)
* Update data insights migration to make the flow easier

* Update logic to change the ReferenceTime to EndOfDay instead of StartOfNextDay

* Fix checkstyle

* Fix versionTimestamp to go to endOfDay
2024-08-02 15:20:08 +02:00
Teddy
2a854f90e1
fix: DBX profiler struct support + nested colum retrieval for profiler (#17267) 2024-08-02 11:34:10 +02:00
sonika-shah
4bb6d7ec1c
Minor : fix populateOwners method to patch all fields of owners (#17272) 2024-08-02 14:52:45 +05:30
Chirag Madlani
b5ea73ef47
test(e2e): Data insight page tests (#17236)
* test(e2e): Data insight page tests

* address comments and fix tests

* fix failing test

* fix start time

* fix tier card test failure

* fix DI running issue in test

* update patch req

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2024-08-02 11:05:45 +05:30
Mohit Yadav
f07daae196
[RbacDomain] RBAC with Domain Restrictions (#16586)
* Add domain field to entity jsons

* add subdomains ui

* Add Domain To create Request

* Typo fix

* commit progress

* allow multiple domains

* - add userDomains and teamDomains

* - Restrict Api Access

* allow multiple domain support

* - Add Listing Restricition
- Add Search Restrictions

* - Field Fix

* - Inherit Team Domains

* remove glossary filter from ui

* - Fix Tag and User Filter

* - Id -> FQN in create

* - Missing domain on indexes

* add domain in creation of team, user and glossary

* fix merge conflicts

* fix feedbacks

* fix tests

* add domain playwright

* minor changes

* fix domain fetching calls

* Add Thread Listing with EntityDomain

* Fixes for Team Join

* Mkae list mutable

* Add missing import

* Fix Team Test

* User Test case fix

* Fix User Domain

* Minor bug fixes related to sub domains

* Fix data retaining in add sub domain modal after closing

* fix NavBar unit test

* Make Domain Access Work Via Policies

* Fix errors introduced after conflict resolutions

* update render function in SubDomainsTable.component.tsx to accept an array of owners instead of a single owner

* fix java checkstyle

* fix py-test

* format

* Fix the domain e2e owner field type

* Fix Failing Tests

* fixed pytest list index

* update userDomains and teamDomains to domains

* Entity Domain to Domain for thread and changeEvent

* FIx fields

* @Ovveride

* FIx

* Remove Duplicate Field

* Remove Duplicate Field

* Fix Failing Test

* fix domain related changes

* fix subdomains loader

* Fix Owners Order

* Enforce Domain Permissions

* fix domain cleanup

* Fix Test

* rever owner change

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
2024-08-02 10:16:14 +05:30
Sriharsha Chintalapani
34b756afcd
Fix patch permissions (#17271)
* Minor: Fix processing of Patch Operations in hierarchical order

* Minor: Fix processing of Patch Operations in hierarchical order
2024-08-01 21:30:07 -07:00