100 Commits

Author SHA1 Message Date
IceS2
12a4c578a2
MINOR: Fix jsonpatch operation order (#15680)
* Mantain the OperationType Order when considering the dividing groups

* Remove reordering the jsonpatch operations from the backend

* Fix checkstyle

* Fix UnitTests to comply with no reordering

* Initial idea on how to fix our current jsonpatch builder from python

* fix(JsonUtils): Change JSONPatch library used

When creating a JSONPatch by using the 'createDiff' method, the library
we are using is not returning a correct JSONPatch when removing multiple
items from an array.

Since the library doesn't provide good ways to override this behavior
and fix it, we decided to move away from it and use the json-patch
library only for this specific operation.

* Fix linters

* Add docstrings

* Refactor patch updated on ingestion framework

* Add UnitTests

* Fix linters
2024-04-05 15:52:01 +02:00
Mohit Yadav
72bd411fe5
[Vulnerabilities] Update Dependencies to fix Vulnerabilities (#15775)
* [VBY] Fix POm version for affected dependency

* - add cp3 dependency

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-04-03 12:04:01 +05:30
Sriharsha Chintalapani
7a3a271f26
Add entityReferences and entityReferenceList as custom properties (#15448)
* Add entityReferences and entityReferenceList as custom properties

* ui: allow to add custom properties for all available property types.

* make entity type as an array

* Fix label typo and import order in AddCustomProperty component

* Remove unnecessary property types with format

* Refactor CustomPropertyTable.tsx to handle array config and enum config

* Add entity reference options and update custom property form

* Add EntityReference and EntityReferenceList as a custom property

* add validation for date format

* Add date and dateTime input support to PropertyValue component

* Fix import order and add search functionality to type dropdown

* Fix custom property cypress tests

* add input for number, email, timestamp, timeInterval, duration, time, sqlQuery,

* add input support for entityReference and entityReferenceList

* Add placeholders for email, timestamp, start time, end time, and duration inputs

* Refactor PropertyValue component to use destructuring for timeInterval object

* Add minWidth style to PropertyValue component and include SQL query editor

* Add entityReference and entityReferenceList as a generic types and not reference to the existing types

* Remove services from entity reference types list

* handle property values for different property type

* Update ExtensionTable to handle object values in CustomPropertyTable

* Add entity reference list rendering and styling

* Fix file paths in complexTypes.json

* Add regex constant for UNIX timestamp in milliseconds and update language files

* Refactor custom property configuration and add new options

* Fix import order and update property value handling

* add unit tests for different input types for different property type

* add unit test for different property type values

* fix cypress tests

* add cypress test for CRUD for different types

* add cypress test for entity ref and entity ref list property type

* add cypress tests for all the 11 entities which support custom properties

* fix cypress test for createCustomPropertyForEntity

* Add new interfaces and types for CustomPropertyTable

* add cypress test for custom property to create via APIs for entities

* add cypress tests for string,integer, markdown, number, duration, email

* add cypress test for enum property for all entities

* add cypress test for sqlQuery, timestamp and timeInterval property

* fix cypress test for sqlQuery input

* fix flaky cypress test for sqlQuery input

* address comments

* Refactor import statements and update ENTITY_PATH enum

* Update import statement in CustomProperty.ts

* change TimeInterval to Time Interval

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2024-03-26 11:02:44 +05:30
Mohit Yadav
cb411d0aa2
- Security Fix (#15664) 2024-03-22 14:14:39 +05:30
Mohit Yadav
b304cbf55f
- Fix slf4j coming from pac4j (#15636) 2024-03-21 00:27:33 +05:30
Chirag Madlani
731813c0e1
chore(ui): fix callback component not loading for okta issue (#15606)
* chore(ui): fix callback component not loading for okta issue

* Vulnerabilities fixes

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2024-03-19 19:33:46 +05:30
Mohit Yadav
91184ecf5d
Using Backend for Oidc Flow (#15557)
* - Initial Draft of Using Backend for Oidc Flow

* - Review Comments

* - Add Session Handler

* - Add Nonce Validation

* Typo

* support backend auth flow from client side

* Add Refresh Logic

* Remove oidc Config from /auth

* Auth Logout Handling

* Add Refresh as separate endpoint

* return empty on no refresh token

* support refresh token and logout

* fix route issues

* fix path issues

* Redirect to Logout on know session

* fix store issues

* exclude logout api from domain filter

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-03-19 17:15:30 +05:30
Teddy
7424b2f430
ISSUE #15381 - Fix execution summary timeout (#15378)
* feat: added dynamic method to build search aggregation from string

* fix: moved summary computation to ES

* fix: added+updated tests

* style: ran java linting
2024-03-01 16:57:34 +01:00
Pere Miquel Brull
10f9fe82a6
#15251 - Fix overwriting of airflow path segments (#15326) 2024-02-27 09:30:28 +01:00
Pere Miquel Brull
62c0cc7563
#13985 - Azure KV Secrets Manager (#15192)
* #13985 - Azure KV Secrets Manager

* Format

* #13985 - Azure KV Secrets Manager

* #13985 - Azure KV Secrets Manager

* Simplify credentials loading

* Simplify credentials loading

* Simplify credentials loading
2024-02-20 07:18:35 +01:00
Akash Jain
95bb221025
feat: Prepare main for next release cycle (#15139) 2024-02-12 16:56:50 +05:30
Mohit Yadav
d18a8ed82c
[Fix] Revert optionals error (#15008)
* Fix Optional Error in Object Mapper

* Revert Optional Changes

* return null instead of 0.0
2024-02-02 20:18:51 +05:30
Pere Miquel Brull
90dc3e830c
MINOR - Bump testcontainers & mysql test version (#14906)
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-01-28 16:45:34 -08:00
Sriharsha Chintalapani
a9e193608c
Fix #14318: Add support of JSON logs (#14783) 2024-01-21 22:59:28 -08:00
Mohit Yadav
0851b87442
Use 8.2.0 version of mysql (#14801) 2024-01-19 23:41:46 +05:30
Suresh Srinivas
c4b9d1bd21
Dependency updates (#14767) 2024-01-17 19:49:39 -08:00
Sriharsha Chintalapani
10e2f69add
MINOR - exclude logback (#14615)
* exclude logback

* exclude logback
2024-01-08 08:41:24 +01:00
Sriharsha Chintalapani
34ff582cf2
Allow user login using username and reassign roles based on users' ldap groups when login (#14550)
* Allow users to login with their user name in LDAP mode, and assign roles to user according to their LDAP group while login

* Fix #12503: Ldap feature: Allow user login using username and reassign roles based on users' ldap groups when login

---------

Co-authored-by: wentian <zb-wentian@kingmed.com.cn>
Co-authored-by: EricWent <135603556+EricWent@users.noreply.github.com>
2024-01-05 10:46:07 +05:30
Suresh Srinivas
17e86ad08f
Update maven plugins and dependencies (#14583)
* Update maven plugin versions

* Upgrade maven dependencies
2024-01-04 19:25:05 -08:00
Sriharsha Chintalapani
7b934b10ae
fix: upgrade org.apache.santuario:xmlsec from 2.2.6 to 2.3.4 (#14509)
Snyk has created this PR to upgrade org.apache.santuario:xmlsec from 2.2.6 to 2.3.4.

See this package in Maven Repository:
https://mvnrepository.com/artifact/org.apache.santuario/xmlsec/

See this project in Snyk:
https://app.snyk.io/org/harshach/project/97e6fbe6-21ff-4c69-b9c5-388bb60a9097?utm_source=github&utm_medium=referral&page=upgrade-pr

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2023-12-31 11:10:24 -08:00
Sriharsha Chintalapani
0303b44b9c
OpenMetadata Operations Command Line Utility (#14504)
* Add new OpenMetadataSetup command line application to migrate/deploy and re-index

* Add new OpenMetadataSetup command line application to migrate/deploy and re-index

* Add deployPipelines option

* Add reIndex option

* add subcommands

* add provision to store upgrade metrics

* rename bootstrap script

* fix styling checks

* Add changelog and store metrics into SERVER_CHANGE_LOG

* Cast jsonb

* Cast jsonb

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-12-29 08:22:58 +01:00
Suresh Srinivas
3950cdb58a
Update dependencies and plugins (#14309) 2023-12-07 19:29:16 -08:00
Chirag Madlani
40bf4a4ab3
Revert "Revert "fixes #13387: replaced ReDoc with RapiDoc for better user experience (#13388)" (#14017)" (#14122)
This reverts commit 04fbe8c01448530380c4604f2eee23c6ed3c7859.
2023-11-27 16:50:37 +05:30
Sriharsha Chintalapani
845a2e585e
Minor: Fix maven profiles to run tests for MySQL and Postgres using maven profiles (#14094)
* Fix maven profiles, run the default profile once

* Fix maven profiles, run the default profile once

* Minor: Fix systemPropertyVariables for maven profiles, fixes postgres tests run

* Minor: Fix systemPropertyVariables for maven profiles, fixes postgres tests run
2023-11-23 22:32:55 -08:00
Suresh Srinivas
0443402812
Update dependencies (#14031) 2023-11-19 19:20:33 -08:00
Shailesh Parmar
04fbe8c014
Revert "fixes #13387: replaced ReDoc with RapiDoc for better user experience (#13388)" (#14017)
This reverts commit 0c37986ad60f2396702233b2ad0bcc264922e771.
2023-11-17 14:29:17 +01:00
Mohit Yadav
8abaeb5b3d
Add Test Jar for Collate Testing (#14010)
* Add Test Jar for Collate Testing

* Add Extension Path for Tests
2023-11-17 15:11:48 +05:30
Mohit Yadav
3f8a931e39
Bump Pom Version to 1.3.0-SNAPSHOT (#14008)
* Bump Pom Version to 1.3.0-SNAPSHOT

* chore: Fix Makefile recipe

* fix: Prepare Main Branch for Next Feature Release

* fix: Syntax issue

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-11-17 11:33:47 +05:30
Anatoliy Shulika
0c37986ad6
fixes #13387: replaced ReDoc with RapiDoc for better user experience (#13388)
* ISSUE-13387: replaced ReDoc with RapiDoc for better user experience

* fixed specUrl

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-11-16 07:56:38 -08:00
Sriharsha Chintalapani
4ec585dca4
fix: openmetadata-service/pom.xml to reduce vulnerabilities (#13888)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-IONETTY-1042268

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2023-11-09 10:25:37 -08:00
Pere Miquel Brull
ec5a35c08d
Fix mvn postgres CI (#13747) 2023-10-27 11:37:18 -07:00
Sriharsha Chintalapani
a054c56f02
Upgrade xmlsec version for onelogin/saml (#13731)
Co-authored-by: Dhruvin Maniar <dhruvinmaniar123@users.noreply.github.com>
2023-10-26 10:13:32 -07:00
Mohit Yadav
528ea335f7
Separate Status and Logs Endpoints (#13709)
* Separate Status and Logs Endpoints

* fix: change endpoints from UI

* Handle Quartz to Unix Conversion in Backend

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
2023-10-26 02:00:27 +05:30
Pere Miquel Brull
41a2aeb1af
[Docs] - Prepare 1.2 docs (#13706)
* Update workflow imports

* LineageRequest without desc

* Usage Workflow

* Usage Workflow

* 1.2 docs publish

* 1.2 docs publish
2023-10-25 16:30:51 +02:00
Suresh Srinivas
672bf03951
Update dependencies and plugins for release 1.3.0 (#13676) 2023-10-22 18:34:33 -07:00
Sriharsha Chintalapani
eb9c7f681d
fix remaining vulnerabilities in server image (#13662) 2023-10-20 09:24:21 -07:00
Sriharsha Chintalapani
c3b65f5807
Fix vulnerabilities in server dependencies (#13659) 2023-10-19 19:33:33 -07:00
Suresh Srinivas
790b4c2287
Update dependencies and plugins for release 1.3.0 (#13658) 2023-10-19 17:14:29 -07:00
Mohit Yadav
dac6e7b478
Remove Shade Plugin causing transitive deps failure (#13516) 2023-10-11 00:21:48 +05:30
Mohit Yadav
4be007a968
Created Uber Jar for openmetada-service (#13512) 2023-10-10 18:39:32 +05:30
Sriharsha Chintalapani
35c4e18663
Upgrade search versions (#13479)
* working on maven shading

* update es in docker

* add maven shade for elasticsearch/opensearch dependencies

* add submodule for es/os dependencies

* upgrade docker elasticsearch to latest 8.10.2

* upgrade docker elasticsearch to latest 8.10.2

* Update Postgres dev to use OpenSearch

* Fix container service dependency

* revert conf change

---------

Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-08 11:55:34 -07:00
Sriharsha Chintalapani
eb84c524aa
Fix repository registration (#13425)
* Fix failing repository registration in external

* Fix failing repository registration in external

* Scan classPath to find Repository annotation

* Remove additioanl config, register any classes with Repository or Collection annotation

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-10-03 21:03:56 -07:00
Sriharsha Chintalapani
8578aee04f
Search refactor (#13397)
* Refactor Search

* Refactor Search

* Fix propgation bugs

* Fix propgation bugs

* Fix glossary term search

* Fix glossary term search

* Only issue search requests if the client is configured properly

* Only issue search requests if the client is configured properly

* Fix glossary index

* add documentation for sharded tables (#13361)

* Cost analysis agg (#13408)

* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: added unusedReport to DI

* feat: added aggregated unused report

* feat: ran linting

* feat: reverted compose file changes

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>

* Add Java 17 support (#12895)

* Add Java 17 support

* Change Test HTTP client provider

* Create Tests HTTP Client once

* Create Tests HTTP Client once

* fix(CI): Update CI to use jdk 17 and dockerfiles as well

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>

* Refactor Search

* Refactor Search

* Fix propgation bugs

* Fix propgation bugs

* Fix glossary term search

* Fix glossary term search

* Only issue search requests if the client is configured properly

* Only issue search requests if the client is configured properly

* Fix glossary index

* Merge main

* fix style

* deleted field propagation

* Fix style

* close the test client

* Changing to jersey connector

* Fix Authentication Exception headers

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-10-03 14:56:53 -07:00
Sriharsha Chintalapani
03ca972887
Add Java 17 support (#12895)
* Add Java 17 support

* Change Test HTTP client provider

* Create Tests HTTP Client once

* Create Tests HTTP Client once

* fix(CI): Update CI to use jdk 17 and dockerfiles as well

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-10-03 01:35:50 -07:00
Artiom Darie
7d2f8dc2bb
Fixes issue-11740: Added support for the om service to connect to AWS RDS using IAM roles (#11913)
* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Changed intial implementation accordingly. Added better flexibility for different auth prodvider impl

* ISSUE-11740: Clean up unnecessary classes

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Code formatting

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Moved docs to 1.2 version

---------

Co-authored-by: artiom.darie <artiom.darie@adswizz.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-09-27 09:59:24 -07:00
Pere Miquel Brull
351df08957
fix: openmetadata-service/pom.xml to reduce vulnerabilities (#13224)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-ORGECLIPSEJETTY-5902998

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2023-09-19 08:30:58 -07:00
Sriharsha Chintalapani
02094179e6
Fix #12899: UI to use Tier TAG displayName if provided (#13232)
* Fix #12899: UI to use Tier TAG displayName if provided

* fix python test
2023-09-17 14:28:10 -07:00
Suresh Srinivas
a22d7bb87f
Update dependencies (#13158) 2023-09-12 15:48:33 -07:00
Pere Miquel Brull
3afb5212ec
[Docs] - Use a.b.x as version pattern (#13161)
* [Docs] - Use a.b.x as version pattern

* Fix image path
2023-09-12 19:41:21 +02:00
07Himank
1960c68355
doc issue fix (#13148) 2023-09-12 16:27:33 +05:30