186 Commits

Author SHA1 Message Date
Imri Paran
d91273a30d
Fix 20325: Trigger external apps with config (#20397)
* wip

* feat: trigger external apps with override config

- Added in openmetadata-airflow-apis functionality to trigger DAG with feature.
- Modified openmetadata-airflow-apis application runner to accept override config from params.
- Added overloaded runPipeline with `Map<String,Object> config` to allow triggering apps with configuration. We might want to expand this to all ingestion pipelines. For now its just for apps.
- Implemented an example external app that can be used to test functionality of external apps. The app can be enabled by setting the `ENABLE_APP_HelloPipelines=true` environment variable.

* fix class doc for application

* fixed README for airflow apis

* fixes

* set HelloPipelines to disabeld by default

* fixed basedpywright errros

* fixed app schema

* reduced airflow client runPipeline to an overload with null config
removed duplicate call to runPipeline in AppResource

* Update openmetadata-docs/content/v1.7.x-SNAPSHOT/developers/applications/index.md

Co-authored-by: Matias Puerta <matias@getcollate.io>

* deleted documentation file

---------

Co-authored-by: Matias Puerta <matias@getcollate.io>
2025-05-06 17:41:24 +07:00
Pere Miquel Brull
d901dd2948
FIX #16284 - Toggle if we want to raise workflow errors (#20969)
* FIX #16284 - Toggle if we want to raise workflow errors

* schema

* schema

* move prop

* fix

* move prop

* improve error handling

* Update openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/IngestionPipelineRepository.java

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/IngestionPipelineRepository.java

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add the `Raise on Error` option to the ingestion schedule step

* Revert "Update openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/IngestionPipelineRepository.java"

This reverts commit 985b73513a59695c6bb39ad41c2d273bbf4e5d22.

* Update the tests

* Fix sonar issue

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-04-29 08:19:13 +02:00
Ashish Gupta
73aaa34b75
update the snapshot to 1.8.0 (#20925) 2025-04-24 10:46:36 +05:30
Akash Jain
0f6d0523d8
feat: Bump Versions to 1.7.0-SNAPSHOT on Main Branch (#20847)
* feat: Bump Versions to 1.7.0-SNAPSHOT on Main Branch

* fix(script): Add a condition for "-SNAPSHOT" is version update script
2025-04-16 15:21:01 +05:30
Pere Miquel Brull
bcde18b387
DEPRECATE - Remove support for Python 3.8 (#20553) 2025-04-08 07:36:07 +02:00
Mayur Singal
7760663b22
MINOR: Change ingestion licence header (#20549) 2025-04-03 10:39:47 +05:30
Imri Paran
97fad806a2
Fixes 19755: Publish app config with status (#19754)
* feat(app): add config to status

add config to the reported status of the ingestion pipeline

* added separate pipeline service client call for external apps

* fix masking of pydantic model

* - overload model_dump to mask secrets instead of a separate method
- moved tests to test_custom_pydantic.py

* fix: execution time

* fix: mask secrets in dump json

* fix: for python3.8

* fix: for python3.8

* fix: use mask_secrets=False when dumping a model for create

* format

* fix: update mask_secrets=False for workflow configurations

* fix: use context directly when using model_dump_json

* fix: default behavior when dumping json

* format

* fixed tests
2025-02-25 16:51:49 +00:00
Keshav Mohta
df193bd0c4
Feature: Reverse Metadata Ingestion (#19723) 2025-02-25 18:41:08 +05:30
datapopcorn
515d9b4869
ISSUE-19254: Fix pipeline unable to run in current day/week/month error (#19523) 2025-01-29 12:30:02 +05:30
Pere Miquel Brull
e56f477a4a
Fix #19147 - Executable Test Suites (#19221)
* backend

* format & tests

* rename backend

* migrations and ingestion

* format & tests

* format & tests

* tests

* format & tests

* tests

* updated ui side of changes

* addressing comment

* fixed failing unit test

* fix test list

* added e2e test, and fixed existing test

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2025-01-07 17:59:54 +01:00
imishchuk-tsgs
cd2408b56f
Fixes 18349: proper version comparison (#18352)
* fix version check

* make checks happy

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-12-07 18:04:41 +01:00
Pere Miquel Brull
c68a45e7d8
Create new Auto Classification Workflow (#18610) 2024-11-19 08:10:45 +01:00
harshsoni2024
4f57f4e710
GEN-309: OpenAPI Connector (#17754)
* configure api service metadata

* add rest api service

* fix test con. pyformat changes

* add models, fix test con.

* improve test con.

* add docs, side doc

* fix model data parse, url error fix

* add tests

* fix pytest errors

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-09-11 13:36:53 +05:30
Ayush Shah
9245c8965f
fix: Bump Ingestion and Dockerfile versions (#17624) 2024-08-29 12:03:30 +05:30
Pere Miquel Brull
4cddee89ad
FIX - Ingestion Airflow Image constraints (#17296)
* FIX - Ingestion constraints

* FIX - Ingestion constraints

* FIX - Ingestion constraints
2024-08-06 06:44:27 +02:00
Mayur Singal
ef67474310
Data Insights Refactor (#16532)
---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Pablo Takara <pjt1991@gmail.com>
Co-authored-by: Chira Madlani <chirag@getcollate.io>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-07-30 17:38:43 +05:30
Sriharsha Chintalapani
fe107aa3cb
Issue #17012: Multi User/Team Ownership (#17013)
* Add multiple owners

* Multi Ownership

* Issue #17012: Multi User/Team Ownership

* Issue #17012: Multi User/Team Ownership

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 1

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 2

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 3

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 4

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 5

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 6

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 7

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 8

* Add Migrations for Owner Thread

* update ingestion for multi owner

* fix pytests

* fixed checkstyle

* Add Alert Name to Publishers (#17108)

* Add Alert Name to Publishers

* Fix Test

* Add Bound to Setuptools (#17105)

* Minor: fixed testSummaryGraph issue (#17115)

* feat: updated multi pipeline ui as per new mock (#17106)

* feat: updated multi pipeline ui as per new mock

* translation sync

* fixed failing unit test

* fixed playwright test

* fixed viewService click issue

* sorted pipeline based on test case length

* Added domo federated dataset support (#17061)

* fix usernames (#17122)

* Doc: Updated Doris & Redshift Docs (#17123)

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>

* Fix #12677: Added Synapse Connector - docs and side docs (#17041)

* Fix #17098: Fixed case sensitive partition column name in Bigquery (#17104)

* Fixed case sensitive partiion col name bigquery

* update test

* #13876: change placement of comment and close button in task approval workflow (#17044)

* change placment of comment and close button in task approval workflow

* minor change

* playwright test for the close and comment function

* supported ref in activityFeedEditor

* fix playwright test

* added playwright test for data steward

* fix the test for the data streward user

* fix the close button not showing if task has no suggestions and icon fixes

* fix sonar issue

* change glossary and add suggestion button to dropdown button

* fix the glossary failure due to button change

* icon change for add tag and description

* fix glossary cypress failure due to button chnages

* changes as per comments

* MINOR: docs links fix (#17125)

* alation link fix

* dbt yaml config source link fix

* bigquery doc fix

* Explore tree feedbacks (#17078)

* fix explore design

* update switcher icon

* show menu when search query exists

* fix selection of active service

* fix type error

* fix tests

* fix tests

* fix tests

* MINOR: Databricks view TableType fix (#17124)

* Minor: fixed AUT test (#17128)

* Fix #16692: Override Lineage Support for View & Dashboard Lineage (#17064)

* #17065: fix the tags not rendering in selector after selection in edit tags task (#17107)

* fix the tags not rendering in selector after selection in edit tags taks

* added playwright test

* minor changes

* minor fix

* fix the tags not updating in edit and accept tag

* fix explore type changes for collate (#17131)

* MINOR: changed log level to debug (#17126)

* changed log level to debug

* fixed type

* changed type to optional

* Get feed and count data of soft deleted user (#17135)

* Doc: Adding OIDC Docs (#17139)

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>

* Doc: Updating Profiler Workflow Docs URL (#17140)

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>

* fix playwright and cypress (#17138)

* Minor: fixed edit modal issue for sql test case (#17132)

* Minor: fixed edit modal issue for sql test case

* fixed test

* Minor: Added whats new content for 1.4.6 release (#17148)

* MINOR [GEN-799]: add option to disable manual trigger using scheduleType (#17031)

* fix: raise for triggering system app

* added scheduleType ScheduledOrManual

* minor: remove "service" field from required properties in createAPIEndpoint schema (#17147)

* initial commit multi ownership

* update glossary and other entities

* update owners

* fix version pages

* fix tests

* Update entity_extension to move owner to array (#17200)

* fix tests

* fix api page errors

* fix owner label design

* locales

* fix owners in elastic search source

* fix types

* fix tests

* fix tests

* Updated CustomMetric owner to entityReferenceList. (#17211)

* Fix owners field in search mappings

* fix search aggregates

* fix inherited label

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 9

* Fix QUeries

* Fix Mysql Queries

* Typo

* fix tests

* fix tests

* fix tests

* fix advanced search constants

* fix service ingestion tests

* fix tests

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: k.nakagaki <141020064+nakaken-churadata@users.noreply.github.com>
Co-authored-by: Prajwal214 <167504578+Prajwal214@users.noreply.github.com>
Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: harshsoni2024 <64592571+harshsoni2024@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Imri Paran <imri.paran@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Siddhant <86899184+Siddhanttimeline@users.noreply.github.com>
2024-07-30 08:06:39 +02:00
Ayush Shah
e9f66e26b4
Bump Setuptools version (#17225) 2024-07-30 10:20:05 +05:30
Ayush Shah
1962ffad52
Fixes #17221: Setuptools Version Fix, Resolve Ingestion Setup Failure (#17216) 2024-07-29 15:29:25 +05:30
IceS2
c522f14178
MINOR: Refactor output_handlers to a WorkflowOutputHandler class (#17149)
* Refactor output_handlers to a WorkflowOutputHandler class

* Add old methods as deprecated to avoid breaking changes

* Extract WorkflowInitErrorHandler from workflow_output_handler

* Fix static checks

* Fix tests

* Fix tests

* Update code based on comments from PR

* Update comment
2024-07-29 09:20:34 +02:00
Ayush Shah
b676f72b89
chore: Update setuptools version in Makefile and pyproject.toml (#17143) 2024-07-24 10:44:20 +05:30
Pere Miquel Brull
8d739563f2
FIX CL-548 - Encrypt JWT Token w/ Secrets Manager (#16861)
* encrypt jwt

* encrypt jwt

* Handle token revokes with SM

* #10192 - Validate both include and excludes filters

* #10192 - Validate both include and excludes filters

* docs
2024-07-11 09:16:48 +02:00
Chirag Madlani
6c1a12a95f
chore: Update main branch for 1.5.0 Release (#16860) 2024-07-01 16:46:16 +05:30
Sriharsha Chintalapani
5571851e53
Pluggable API/Features Limits (#16782)
* Limits

* Limits

* - Mismatched Types

* Update Limits config response

* Update Limits feature response

* Limits

* Limits

* - Mismatched Types

* Update Limits config response

* Update Limits feature response

* Limits: add entity resource enforcer

* Limits: fix rebase

* update limits enforcement

* Add OperationContext to limits

* chore: Bump versions to `1.4.0`

* chore: Bump Ingestion Versions to `1.4.0.1` for Release

* chore: Bump Ingestion Versions to `1.4.0.1`  in Dockerfiles for Release

* Remove Retry From Abstract Event Consumer (#16405)

(cherry picked from commit f8ed079731cc238dc136306fe018c5df35dd2f3b)

* Fix Migrations: Add postgres migrations (#16403)

(cherry picked from commit 9416a7ac5fa8fd9695063b108501790d813e8e6e)

* Add Null Check for isAdmin (#16407)

* Remove Retry From Abstract Event Consumer

* - Add Check for null Or Empty in isAdmin

* - Fix Test

(cherry picked from commit fe2db2d63c5495b6c288d4252a19ab77481b6de0)

* Fix OpenLineage ingestor (#16416)

* Fix OpenLineage ingestor

* py format

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>

* Minor: added whats new for 1.4.1 (#16420)

* Minor: added whats new for 1.4.1

* added note in to whats new

* Fix SSL issue (#16412)

* chore: Bump Versions for `1.4.1` Release

* chore(release): Prepare Branch for `1.4.2`

* [MINOR] partition migration issue with redshift servics (#16452)

* fix: partition migration issue with redshift servics

* chore: typo in sql comment

(cherry picked from commit 451d73593e813151c24f2c1d17efb3dcdebb71c8)

* minor(ui): update what's new for 1.4.2 (#16457)

(cherry picked from commit d55981adfd2321de706e4a043828bb473a4b05f1)

* fix:  ingestion for dbt > 1.8.0 resource_type is not an enum (#16415)

* fix: resource_type is not an enum

* feat: add log to display finis

* improve readability

* use getattr to be compatible

* format

* Add Cache Query Param for Limits

* Only Parse view  query (#16470)

* add limit check during user creation via PUT

* add limit check during user creation via PUT

* MINOR: Kafka Setup SSL Arg Fix (#16469)

* Fix#16404 - Show Node level lineage by default (#16445)

* default to node layer

* update cypress

* code cleanup

* fix cypress

(cherry picked from commit f0cda8464f34a21f45f18fa557e980fb2f105d8e)

* Invalidate count of data asset after hard delete. add limit exception to ingestion client

* - Remove Change Description from Lineage (#16488)

(cherry picked from commit 9e5c5529a84dfc781382b3a3b6abd80ee41f11f5)

* - Non Indexable fields should be remvoed at the end (#16499)

(cherry picked from commit f0b0f7a9426ca601d3bfee3989d4ce47e732a7af)

* fix announcement not redirect from landing page (#16506)

* fix announcement not redirect from landing page

* minor changes

* change in cypress test

(cherry picked from commit ee7cddd169a3a1fb1e598e80035c2fc15a5a129b)

* Fix Schema Field Null Issue (#16510)

(cherry picked from commit 022772943f1b33f6230cb35547d1da6acfaf6cfa)

* feat(ui): limits integration with application (#16206)

* feat(ui): limits integration with application

* support pipelineSchedules via limit api

* enforce limit to all the modules

* update banner styling

* update

* support disable option for ManageButton

* limit version

* fix spotlight

* update tests

* Add name and version history to resource limits
Refactor the getEntityIcon function and add new icon mappings

* limit version

* hide access token tab

* fix version for all the entity

* fix tests

* fix DQ tests

* Add fallback for the icon

* Revert the fallback icon changes

* Apply the limit to the add ingestion button in the service details page

* Fix the data quality tab add test button not working

* fix banner styling

* minor fix

* Fix ingestion component unit test

* Add InlineAlert component

* update entityNameLabels mapping object

* Fix the incorrect link in LimitBanner

* update pricing page url

* Create the GlobalSettingsClassBase

* Update URLs for pricing page and upgrade options

* fix global settings uncaught error

* add parameters to the resource limit API

* implement inline alerts for service and alert creation form

* update PRIVILEGES for docker

* fix layout issues

* fix tests

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>

* Add token limitations

* Add token limitations

* Add appType as part of schema in ingestion pipeline (#16519)

* #16489: fix the redirect issue to new tab for tags and glossary (#16512)

* fix the redirect issue to new tab for tags and glossary

* fix the redirect on cancel icon and unit test issue

* changes as per comments

(cherry picked from commit 8d312f0853609cfef260739cf789d459838a3421)

* Fix  #16229 - Tag and Service filters for test cases (#16484)

* fix: added test case support for tags (inherit from table/column)]"

* feat: add tag and service filter for test cases

* feat: add tier query param

* fix: tests

(cherry picked from commit 6b00dde90285924445567ee7c396c89f0fcf3f1d)

* fix: None type is not iterable (#16496)

(cherry picked from commit 656da03b14ca24171cf7924b9dd33663e6bed423)

* minor(ui): refresh token for OIDC SSO (#16483)

* minor(ui): refresh token for OIDC SSO

* remove frame window timeout issue

* increase iFrame timeout for oidc

(cherry picked from commit 1a6c4c972052836a9b3cfa273b7ea1aa3202eafe)

* feat(ui): support tag & tier filter for test case (#16502)

* feat(ui): support tag & tier filter for test case

* fix tag filter

* allow single select for tier

* added service name filter

* update cypress for tags, tier & service

* add specific add for filters

* fix tier api call

(cherry picked from commit 5b71d79e8ac2d08a154882dfe71b9b3a0f73bffc)

* minor: sanitize activity feed editor content (#16533)

* Add appType as part of schema in ingestion pipeline (#16519)

* Fixed quicksight conn (#16537)

* fix: saml auth for new user not created (#16543)

* fix: saml auth for new user not created

* doc: add comment

* Fix#16491 -  fix lineage edge description update (#16538)

* fix lineage edge description update

* fix tests

(cherry picked from commit dff0aa8dbedcd4064ad63765cadda65bb998772e)

* CYPRESS: fix announcement cypress (#16536)

* fix announcement cypress

* changes as per comments

* fix the cypress failure

(cherry picked from commit fcb87b5866ba06aa7a6db516677e311c24053db7)

* [MINOR] Fix Test Failure for EventRegistration

* [MINOR] Fix Test Failure for EventRegistration

* [MINOR] Fix Test Failure for EventRegistration

[MINOR] Fix Test Failure for EventRegistration

* Fix Event Handlers registration Issue (#16544)

* Fix Event Handlers Issue

* Review Comments

(cherry picked from commit d374e48b7938e8ad3514dc5cf8dff619a12595e3)

* [MINOR] Fix Test Failure for EventRegistration

(cherry picked from commit 4563ad4fd10f9790c21fe744d8fc131ebd028ac8)

* Fix Topic Schema missing messageSchema (#16545)

(cherry picked from commit b612dd90c07f564d38392b1ccfe0de1505a4867b)

* Add limits exception cache in rest client

* MINOR: Ignore Cluster Information from columns (#16495)

* minor: improve the block editor initial content history (#16540)

* Minor: fixed data quality page type issue (#16556)

* #16521: fix issue in userProfilePage for roles. teams and displayName (#16527)

* fix update on roles and backlink them in user profile page

* fix teams, displayName and profile pic issue

* sonar fix

* fix cypress issue

* minor changes

(cherry picked from commit 98945cb2db87ebb325d3a72131f049abffcba345)

* Empty quick filters (#16402)

* initial commit for empty quick filters

* update progress

* fix field title

* cleanup

* add tests

* unit tests

* fix encoding of search query

* add cypress tests

* add cypress

* fix flaky cypress

* fix review comments

* revert tooltip changes

* fix tests

* fix tests

(cherry picked from commit 5930cd7a7a4bef73f6850848c85118eb64843e2d)

* Fix #16278 : Search to display Draft glossaryTerms on Explore page (#16462)

* Fix #16278 : Search to display Draft glossaryTerms as well on Explore page

* add term status quick filter

* change aggregation key for status field

* change aggregation key for status field

* add lowercase_normalizer in status filed for aggregate api

* add cypress tests

* fix cypress

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
(cherry picked from commit ae5e9d61cc9e6a39d65972987de9149a421395b1)

* [FIX] GlossaryTerm reviewers should be user or team only (#16372)

* add teams as reviewer

* Check Users to be reviewers

* Reviewers can be a team or user

* Fix check by id or name

* Review can be team or user both

* Validate Reviewers

* add multi select control

* - Fix Reviewers

* - Centralize Reviewer Relationship to EntityRepository

* - Sort

* add team as reviewer for glossary terms

* locales

* cleanup

* - Update Reviewer should remove existing reviewers

* fix selectable owner control

* fix code smells

* fix reviewer issue

* add glossary cypress

* fix patch issue on reviewers set to null

* update cypress tests

* fix cypress

* fix cypress

* fix reviewers in glossary task and supported cypress

* fix pytest

* Fix

* fix cypress

* fix code smells

* Inherited Reviewers need to be present always

* filter out inherited users

* fix cypress

* fix backend tests failure

* fix backend tests failure -checkstyle

* restrict owner to accept task in case of reviewer present

* fix pytest

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
(cherry picked from commit 9ec3d94e3b8445e63a7d77239c92c92a32536bf2)

* Add testSuite tags, domain field and check for TestCase limits

* fix owner not showing after refersh in teams page (#16567)

(cherry picked from commit 119fcf8959732a980b75e1f795a9f2dc5288cd27)

* [ISSUE-16503] Fix createUser to use EntityResource (#16549)

* Fix createUser to use EntityResource

* fix broken tests

* Fix Tests - 3

(cherry picked from commit aeb020ae3b0cbab3a2ee5995c61480cdd1eae405)

* what's new for 1.4.2 (#16568)

(cherry picked from commit c86468d9929e433922886852381269b46d69c832)

* address feedbacks

* fix error for bots page

* update banner text

* allow force fetch limit

* fix ingestion schedule

* Revert "Merge branch '1.4.2' into limits"

This reverts commit 8e965207a23ba527d0f5ba91463c1869077bf091, reversing
changes made to 4d16531965fb0d489a4afdebd45ab5b7f3d1eb5c.

* Merge 1.4.2 (#16578)

* fix explore page conflicts

* fix tests

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Chira Madlani <chirag@getcollate.io>

* fix subheader

* Updating glossary reviewers should propagate reviewers in glossary term (#16580)

* highlight inherited reviewer in glossary

* locales

* use glossary name for search query

* fix glossary version cypress

* add union datatype for subfields

* Adding reviewer to glossary also adds them as an assignee to the task

* add glossary approval cypress

---------

Co-authored-by: sonikashah <sonikashah94@gmail.com>
(cherry picked from commit 4c8bf1cac14074df87dafe7a719e2795b0a29895)

* Update documentation for Search Index apis (#16539)

(cherry picked from commit d3123c49143652015c416d271d9fd0f9cfa9e324)

* cypress: fixed flakiness and announcment cypress (#16579)

* fetch latest limit for create / delete operations

* guard datAsset limit got topic, dashboard, mlmodel etc

* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias (#16589)

* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias

* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias

(cherry picked from commit 8723b8c36afe31410c31d1ebbdafe7b1770921fa)

* cypress: fixed cypress AUT for mysql (#16446)

* cypress: fixed cypress AUT for mysql

* minor fix

* skip announcment redirection cypress

* Minor: Ensure correct index mapping in Elasticsearch for clusterAlias (#16598)

(cherry picked from commit 04543722a6f6e2b1eaf7a451ebb1c176862bc346)

* Fix Postgres Application listing (#16600)

* Fix Postgres Application listing

* Fix Listing

(cherry picked from commit 77dfe1f6af53d187ff7a61fdb1e1416de7178f5a)

* fix limit related issue

* Fix Automations limits invalidation during the uninstall

* cypress: fixed 1.4.2 AUT cypress (#16602)

* cypress: fixed 1.4.2 AUT cypress

* fix cypress around announcement,user,glossary, lineage and mydata

* searchIndexApplication fix and minor changes

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>

* test: add updateJWTTokenExpiryTime util (#16606)

(cherry picked from commit 8c173bed6a279cb0a648bd30632ea6ebdf4a2a90)

* OSS changes for adding automator cypress tests (#16611)

* Fix Test Suite Filter (#16615)

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
(cherry picked from commit 3db41f08e27f388495040e5b23cc7bee5ae665f1)

* MINOR: Fix Profiler for SSL Enabled Source (#16613)

* Add Test Suite SSL (#16619)

* MINOR: Fix ssl connection in usage & lineage (#16625)

* Fix owner notification (#16629)

* - Fix Task notification not getting sent to owners

* - Fix Task notification not getting sent to owners

(cherry picked from commit cc2d581eb0524604b6dcf0523e9ca96e0b8a6ce3)

* chore(release): Prepare Branch for `1.4.3`

* - Fix User Signup (#16667)

(cherry picked from commit b4cba8a850ecd7a25aeff6ca7dea0dc432d43d86)

* - Fix User Signup - p2

(cherry picked from commit d9ae6f6db9891f8e9bf7ad49c561a71dd50103da)

* - Update What's new (#16669)

- fix vulnerability

(cherry picked from commit 1dcb1bd46f9da49764f4c61a7ac5048dd2fa956b)

* Minor: Fix incorrect alert on signup page (#16666)

* Fix Application enforceLimits during install

* Wrap the add test button with limits wrapper for column profile tab

* fix errors

* fix tests

* fix pylint

* fix tests

* fix limits

* pylint

* fix schedule options

* fix glossary spec failure

* Add domain & tags to testSuite

* Update airflow-apis-tests-3_9.yml

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Chira Madlani <chirag@getcollate.io>
Co-authored-by: Pablo Takara <pjt1991@gmail.com>
Co-authored-by: Akash-Jain <15995028+akash-jain-10@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Antoine Balliet <antoine.balliet@gorgias.com>
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-01 14:59:25 +05:30
IceS2
34f6f81a20
MINOR: added exclude_defaults=False for profiler and deploy on airflow-apis (#16734) 2024-06-21 12:05:13 +05:30
Mayur Singal
e3fa340c8f
MINOR: Pydantic fixes for redshift & kafka (#16638) 2024-06-14 14:08:59 +05:30
Pere Miquel Brull
cb72a22b59
Fix - e2e tests for pydantic V2 (#16551)
* Fix - e2e tests for pydantic V2

* add correct default

* add correct default

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* fix apis

* format
2024-06-06 19:36:17 -07:00
Pere Miquel Brull
d8e2187980
#15243 - Pydantic V2 & Airflow 2.9 (#16480)
* pydantic v2

* pydanticv2

* fix parser

* fix annotated

* fix model dumping

* mysql ingestion

* clean root models

* clean root models

* bump airflow

* bump airflow

* bump airflow

* optionals

* optionals

* optionals

* jdk

* airflow migrate

* fab provider

* fab provider

* fab provider

* some more fixes

* fixing tests and imports

* model_dump and model_validate

* model_dump and model_validate

* model_dump and model_validate

* union

* pylint

* pylint

* integration tests

* fix CostAnalysisReportData

* integration tests

* tests

* missing defaults

* missing defaults
2024-06-05 21:18:37 +02:00
Akash Jain
015d71c9de
chore: Update main branch for 1.5.0 Release (#16401) 2024-05-23 18:46:15 +05:30
Pere Miquel Brull
62259a25cf
MINOR - Airflow dag_tag PK (#16314)
* MINOR - Airflow dag_tag PK

* MINOR - Airflow dag_tag PK
2024-05-17 07:31:06 +02:00
Pere Miquel Brull
44181b1ad3
FIX - Airflow tagging long names (#16304)
* FIX - Airflow tagging long names

* FIX - Airflow tagging long names

* FIX - Airflow tagging long names
2024-05-16 15:33:42 +02:00
Pere Miquel Brull
db3bff19af
MINOR - Add labels in OM Airflow DAGs (#16233)
* MINOR - Add labels in OM Airflow DAGs

* update test
2024-05-13 12:45:20 +02:00
Akash Jain
95bb221025
feat: Prepare main for next release cycle (#15139) 2024-02-12 16:56:50 +05:30
Teddy
9a4a9df836
Fix #14895 - Get Metadata from Parquet Schema (#14956)
* linting: fix python linting

* fix: get column types from parquet schema for parquet files

* style: python linting

* fix: remove displayType check in test as variation depending on OS
2024-02-01 09:02:52 +01:00
Sriharsha Chintalapani
2e95fcb98d
Fix #14786: Suggestions API (#14821)
* Fix #14786: Suggestions API

* Handle suggestions in ometa

* Minor: Optimise Databricks Client (#14776)

* MINOR - Fix SP topology context & Looker usage context (#14816)

* MINOR - Fix SP topology context & Looker usage context

* MINOR - Fix SP topology context & Looker usage context

* Fix tests

* Fixes #14598: Fix Tags / Labels ingestion on includeTags as False (#14782)

* fix(ui): password error message for char limits (#14808)

* fix(ui): password error message for char limits

* fix java side code

* Fixes #13556: Support for Salesforce table description ingestion (#14733)

* ISSUE-13556: Add suport for Salesforce table description ingestion

* ISSUE-13556: Remove unnecessary blank line

* ISSUE-13556: Fix to get description for each table

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>

* MINOR - Better handling of Ingestion Pipeline Status (#14792)

* MINOR - Better handling of Ingestion Pipeline Status

* format

* format

* MINOR: Added table validation for cost analysis data (#14793)

* Added validation for cost analysis source

* centralized life cycle logic

* CYPRESS: simplify side navigation click in cypress (#14818)

* simplify side navigation click in cypress

* make sidbar item uses common enum

* fix cypress failure of outside import

* fix(#14326): tier dropdown is not working in advance search (#14780)

* improvement in advance search based on custom property

* fix a reading undefined property issue

* wip: advance search based on tier

* some code cleanup and improvement

* some fixes

* fix: ui flicker when advanceSearched is apply and refresh the page

* some cleanup

* no need to call customproperty api call, if entity not suppport customProperties

* minor change

* fix: autocomplete not working in tier search option in advance search modal

* added unit test for advance search provider component

* some cleanup

* added testcase for open modal

* added testcase for resetAllFilters method

* removed unwanted code

* added e2e test for testing tier advance search

* fix: e2e search flow for single field

* fix: string field not working after giving listValues in TierSearch

* fix: group query e2e test fix

* used asyncFetch way to get the tierOptions synchronously

* some cleanup

* remove unwanted lines

* some cleanup

* fix: selected option show option value instead of option title

* fix(minor): update skip icon for executions (#14809)

* Fixes #14803: ignore capitalization when confirming deletes  (#14804)

* ignore case when confirming deletes

* Test confirmation of deletes works when case differs 

Added test case for 'delete' as the confirmation text.

* minor(config): update openmetadata-ui code reviewers (#14823)

* Add Tests

* Add list/accept/reject apis

* initial ui changes

* localisation

* show suggestion for empty description

* ui feedbacks

* Fix permission check for entities without owner

* Fix entityLink and add tests

* Add update suggestion WIP

* Fix test

* Fix PUT and Pagination

* Fix styling

* update test

* Update status

* add OM server connection in apps

* add permissions check

* Fix CI

* Remove TODO

* Fix feedResourceTest

* fix unit tests

* add private configs for apps

* add private configs for apps

* fix update application icons

* minor center align icon

* add private configs for apps

* Format

* Fix pydantic gen

* Remove token

* Update name

* Rework private conf

* Fix apps

* Fix apps

* Format

* Format

* show metapilot only if its installed

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: kwgdaig <18678754+kwgdaig@users.noreply.github.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Abhishek Porwal <80886271+Abhishek332@users.noreply.github.com>
Co-authored-by: Carlo Q <carlo@machina.bio>
Co-authored-by: karanh37 <karanh37@gmail.com>
2024-01-31 18:51:09 -08:00
Mayur Singal
96435a10eb
MINOR: Fix pyproject.toml readme (#14925) 2024-01-29 17:02:31 +05:30
Pere Miquel Brull
0332fb9fe5
#13934 - Update test connection no response error msg (#14897)
* #13934 - Update test connection no response error msg

* #13934 - Update test connection no response error msg
2024-01-28 10:25:15 -08:00
Pere Miquel Brull
6d6db4b257
#14859 - Fix signature of status callback (#14894) 2024-01-26 17:44:35 +01:00
Pere Miquel Brull
e8eb2bda3c
MINOR - Hide AutoTagger, add MetaPilot search, and fix apps status calls (#14201)
* MINOR - Hide AutoTagger & MetaPilot search & Fix apps status calls

* Fix method call

* Fix method call

* Fix method call

* "additionalProperties": true

* Use EntityRef

* Use ES sink

* Update UI schema type
2024-01-08 14:57:07 +01:00
Mayur Singal
d82770a91e
Fix #14421: Airflow - Handle pipeline status when dag killed (#14562) 2024-01-05 18:14:26 +05:30
Onkar Ravgan
79444f4a24
Fix 13823: Validate and Parse gracefully IngestionPipeline (#14461)
* fixed ingestion pipeline parsing

* Added validation for automation workflow

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-12-22 09:56:39 +05:30
karanh37
853a83f8b9
fix: allow config update for external app (#14034)
* fix: allow config update for external app

* Prep config

* Add configs

* fix: call configure app on saving of configurations

* fix: add autocomplete widget for json schema forms

* ORganize configuration handling

* Allow config

* Format

* Fix test

* Prep migrations for allowConfiguration

* TODO

* Add pg migrations

* fix: add allowConfiguration support

* Autotagger

* Update install flow

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-11-27 16:55:33 -08:00
Pere Miquel Brull
caaf0e7a1d
Fix #12436 - Migrate to pyproject.toml (#14025)
* test

* Use pyproject.toml

* Fix pylint

* makefile

* makefile

* Fix pylint

* isort

* pyproject

* Airflow apis pyproject

* Remove ingestion core

* isort

* Fix makefile help
2023-11-22 07:10:37 +01:00
Mohit Yadav
3f8a931e39
Bump Pom Version to 1.3.0-SNAPSHOT (#14008)
* Bump Pom Version to 1.3.0-SNAPSHOT

* chore: Fix Makefile recipe

* fix: Prepare Main Branch for Next Feature Release

* fix: Syntax issue

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-11-17 11:33:47 +05:30
Pere Miquel Brull
c742835766
Auto Tagger Application - Preparing the Ingestion Framework (#13862)
* Prepare the skeleton for generic app registration

* Prepare the skeleton for generic app registration

* Handle app runner

* Prepare the skeleton for generic app registration

* Prepare the skeleton for generic app registration

* Allow deployment

* Fix PII APP

* Fix lint

* Fix PII APP

* Fix PII APP

* Prepare config-based external apps

* Prepare config-based external apps

* Fix lint

* Prepare config-based external apps

* Fix DI errors

* Amend comments
2023-11-13 08:58:38 +01:00
Onkar Ravgan
600cbff707
Fixed print_status import in data_insights.py (#13490) 2023-10-09 18:31:58 +05:30
Pere Miquel Brull
aed9e3875f
DQ base workflow (#13454)
* DQ BaseWorkflow

* Test suite runner

* test Suite workflow

* Refactor DQ for BaseWorkflow

* Lint

* Fix source

* Fix source

* Fix source

* Fix source

* Fix test

* Fix test

* Fix test
2023-10-06 18:29:18 +02:00
Pere Miquel Brull
0282574bdd
Create ometa client once and pass it around & improve pycln config (#13310)
* Create ometa client once and pass it around & improve pycln config

* Fix

* Fix

* Fix tests

* Fix maven ci

* Fix tests

* Fix tests

* Fix tests

* Format

* Fix DI
2023-10-04 09:14:03 +02:00
Teddy
6ca71ae323
Issue 12679 - Handle Entity Object Instantiation Error + Refactor Workflow (#13384)
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: ran linting

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-02 12:05:30 +02:00
Mayur Singal
4e633877b3
Fix ElasticSearch Test Connection & Deploy (#13061) 2023-09-08 12:40:48 +05:30