* wip
* feat: trigger external apps with override config
- Added in openmetadata-airflow-apis functionality to trigger DAG with feature.
- Modified openmetadata-airflow-apis application runner to accept override config from params.
- Added overloaded runPipeline with `Map<String,Object> config` to allow triggering apps with configuration. We might want to expand this to all ingestion pipelines. For now its just for apps.
- Implemented an example external app that can be used to test functionality of external apps. The app can be enabled by setting the `ENABLE_APP_HelloPipelines=true` environment variable.
* fix class doc for application
* fixed README for airflow apis
* fixes
* set HelloPipelines to disabeld by default
* fixed basedpywright errros
* fixed app schema
* reduced airflow client runPipeline to an overload with null config
removed duplicate call to runPipeline in AppResource
* Update openmetadata-docs/content/v1.7.x-SNAPSHOT/developers/applications/index.md
Co-authored-by: Matias Puerta <matias@getcollate.io>
* deleted documentation file
---------
Co-authored-by: Matias Puerta <matias@getcollate.io>
* fix(data-quality): table diff
- added handling for case-insensitive columns
- added handling for different numeric types (int/float/Decimal)
- added handling of boolean test case parameters
* add migrations for table diff
* add migrations for table diff
* removed cross type diff for now. it appears to be flaky
* fixed migrations
* use casefold() instead of lower()
* - implemented utils.get_test_case_param_value
- fixed params for case sensitive column
* handle bool test case parameters
* format
* testing
* format
* list -> List
* list -> List
* - change caseSensitiveColumns default to fase
- added migration to stay backward compatible
* - removed migration files
- updated logging message for table diff migration
* changed bool test case parameters default to always be false
* format
* docs: data diff
- added the caseSensitiveColumns parameter
requires: https://github.com/open-metadata/OpenMetadata/pull/18115
* fixed test_get_bool_test_case_param
* enable logging for debugging
* remove node-gyp
* add node-gyp globally before installing deps
* reduce quick type to 10
* revert quicktype and node-gyp changes for CI
* fix unit tests
* GEN-1493 - Fix opensearch pagination
* GEN-1494 - Add CI for py-tests with Postgres and Opensearch
* GEN-1494 - Add CI for py-tests with Postgres and Opensearch
* Fix backend tests and have index alias with clusterAlias appended
* Fix backend tests and have index alias with clusterAlias appended
* Fix failing tests
* alias setup
* fix suggestion not working due to alias
* fix getIndexOrAliasName method for multiple indexes
* update openmetadata.yaml
* update childAliases with clusterAlias
---------
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
* Limits
* Limits
* - Mismatched Types
* Update Limits config response
* Update Limits feature response
* Limits
* Limits
* - Mismatched Types
* Update Limits config response
* Update Limits feature response
* Limits: add entity resource enforcer
* Limits: fix rebase
* update limits enforcement
* Add OperationContext to limits
* chore: Bump versions to `1.4.0`
* chore: Bump Ingestion Versions to `1.4.0.1` for Release
* chore: Bump Ingestion Versions to `1.4.0.1` in Dockerfiles for Release
* Remove Retry From Abstract Event Consumer (#16405)
(cherry picked from commit f8ed079731cc238dc136306fe018c5df35dd2f3b)
* Fix Migrations: Add postgres migrations (#16403)
(cherry picked from commit 9416a7ac5fa8fd9695063b108501790d813e8e6e)
* Add Null Check for isAdmin (#16407)
* Remove Retry From Abstract Event Consumer
* - Add Check for null Or Empty in isAdmin
* - Fix Test
(cherry picked from commit fe2db2d63c5495b6c288d4252a19ab77481b6de0)
* Fix OpenLineage ingestor (#16416)
* Fix OpenLineage ingestor
* py format
---------
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
* Minor: added whats new for 1.4.1 (#16420)
* Minor: added whats new for 1.4.1
* added note in to whats new
* Fix SSL issue (#16412)
* chore: Bump Versions for `1.4.1` Release
* chore(release): Prepare Branch for `1.4.2`
* [MINOR] partition migration issue with redshift servics (#16452)
* fix: partition migration issue with redshift servics
* chore: typo in sql comment
(cherry picked from commit 451d73593e813151c24f2c1d17efb3dcdebb71c8)
* minor(ui): update what's new for 1.4.2 (#16457)
(cherry picked from commit d55981adfd2321de706e4a043828bb473a4b05f1)
* fix: ingestion for dbt > 1.8.0 resource_type is not an enum (#16415)
* fix: resource_type is not an enum
* feat: add log to display finis
* improve readability
* use getattr to be compatible
* format
* Add Cache Query Param for Limits
* Only Parse view query (#16470)
* add limit check during user creation via PUT
* add limit check during user creation via PUT
* MINOR: Kafka Setup SSL Arg Fix (#16469)
* Fix#16404 - Show Node level lineage by default (#16445)
* default to node layer
* update cypress
* code cleanup
* fix cypress
(cherry picked from commit f0cda8464f34a21f45f18fa557e980fb2f105d8e)
* Invalidate count of data asset after hard delete. add limit exception to ingestion client
* - Remove Change Description from Lineage (#16488)
(cherry picked from commit 9e5c5529a84dfc781382b3a3b6abd80ee41f11f5)
* - Non Indexable fields should be remvoed at the end (#16499)
(cherry picked from commit f0b0f7a9426ca601d3bfee3989d4ce47e732a7af)
* fix announcement not redirect from landing page (#16506)
* fix announcement not redirect from landing page
* minor changes
* change in cypress test
(cherry picked from commit ee7cddd169a3a1fb1e598e80035c2fc15a5a129b)
* Fix Schema Field Null Issue (#16510)
(cherry picked from commit 022772943f1b33f6230cb35547d1da6acfaf6cfa)
* feat(ui): limits integration with application (#16206)
* feat(ui): limits integration with application
* support pipelineSchedules via limit api
* enforce limit to all the modules
* update banner styling
* update
* support disable option for ManageButton
* limit version
* fix spotlight
* update tests
* Add name and version history to resource limits
Refactor the getEntityIcon function and add new icon mappings
* limit version
* hide access token tab
* fix version for all the entity
* fix tests
* fix DQ tests
* Add fallback for the icon
* Revert the fallback icon changes
* Apply the limit to the add ingestion button in the service details page
* Fix the data quality tab add test button not working
* fix banner styling
* minor fix
* Fix ingestion component unit test
* Add InlineAlert component
* update entityNameLabels mapping object
* Fix the incorrect link in LimitBanner
* update pricing page url
* Create the GlobalSettingsClassBase
* Update URLs for pricing page and upgrade options
* fix global settings uncaught error
* add parameters to the resource limit API
* implement inline alerts for service and alert creation form
* update PRIVILEGES for docker
* fix layout issues
* fix tests
---------
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
* Add token limitations
* Add token limitations
* Add appType as part of schema in ingestion pipeline (#16519)
* #16489: fix the redirect issue to new tab for tags and glossary (#16512)
* fix the redirect issue to new tab for tags and glossary
* fix the redirect on cancel icon and unit test issue
* changes as per comments
(cherry picked from commit 8d312f0853609cfef260739cf789d459838a3421)
* Fix #16229 - Tag and Service filters for test cases (#16484)
* fix: added test case support for tags (inherit from table/column)]"
* feat: add tag and service filter for test cases
* feat: add tier query param
* fix: tests
(cherry picked from commit 6b00dde90285924445567ee7c396c89f0fcf3f1d)
* fix: None type is not iterable (#16496)
(cherry picked from commit 656da03b14ca24171cf7924b9dd33663e6bed423)
* minor(ui): refresh token for OIDC SSO (#16483)
* minor(ui): refresh token for OIDC SSO
* remove frame window timeout issue
* increase iFrame timeout for oidc
(cherry picked from commit 1a6c4c972052836a9b3cfa273b7ea1aa3202eafe)
* feat(ui): support tag & tier filter for test case (#16502)
* feat(ui): support tag & tier filter for test case
* fix tag filter
* allow single select for tier
* added service name filter
* update cypress for tags, tier & service
* add specific add for filters
* fix tier api call
(cherry picked from commit 5b71d79e8ac2d08a154882dfe71b9b3a0f73bffc)
* minor: sanitize activity feed editor content (#16533)
* Add appType as part of schema in ingestion pipeline (#16519)
* Fixed quicksight conn (#16537)
* fix: saml auth for new user not created (#16543)
* fix: saml auth for new user not created
* doc: add comment
* Fix#16491 - fix lineage edge description update (#16538)
* fix lineage edge description update
* fix tests
(cherry picked from commit dff0aa8dbedcd4064ad63765cadda65bb998772e)
* CYPRESS: fix announcement cypress (#16536)
* fix announcement cypress
* changes as per comments
* fix the cypress failure
(cherry picked from commit fcb87b5866ba06aa7a6db516677e311c24053db7)
* [MINOR] Fix Test Failure for EventRegistration
* [MINOR] Fix Test Failure for EventRegistration
* [MINOR] Fix Test Failure for EventRegistration
[MINOR] Fix Test Failure for EventRegistration
* Fix Event Handlers registration Issue (#16544)
* Fix Event Handlers Issue
* Review Comments
(cherry picked from commit d374e48b7938e8ad3514dc5cf8dff619a12595e3)
* [MINOR] Fix Test Failure for EventRegistration
(cherry picked from commit 4563ad4fd10f9790c21fe744d8fc131ebd028ac8)
* Fix Topic Schema missing messageSchema (#16545)
(cherry picked from commit b612dd90c07f564d38392b1ccfe0de1505a4867b)
* Add limits exception cache in rest client
* MINOR: Ignore Cluster Information from columns (#16495)
* minor: improve the block editor initial content history (#16540)
* Minor: fixed data quality page type issue (#16556)
* #16521: fix issue in userProfilePage for roles. teams and displayName (#16527)
* fix update on roles and backlink them in user profile page
* fix teams, displayName and profile pic issue
* sonar fix
* fix cypress issue
* minor changes
(cherry picked from commit 98945cb2db87ebb325d3a72131f049abffcba345)
* Empty quick filters (#16402)
* initial commit for empty quick filters
* update progress
* fix field title
* cleanup
* add tests
* unit tests
* fix encoding of search query
* add cypress tests
* add cypress
* fix flaky cypress
* fix review comments
* revert tooltip changes
* fix tests
* fix tests
(cherry picked from commit 5930cd7a7a4bef73f6850848c85118eb64843e2d)
* Fix#16278 : Search to display Draft glossaryTerms on Explore page (#16462)
* Fix#16278 : Search to display Draft glossaryTerms as well on Explore page
* add term status quick filter
* change aggregation key for status field
* change aggregation key for status field
* add lowercase_normalizer in status filed for aggregate api
* add cypress tests
* fix cypress
---------
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
(cherry picked from commit ae5e9d61cc9e6a39d65972987de9149a421395b1)
* [FIX] GlossaryTerm reviewers should be user or team only (#16372)
* add teams as reviewer
* Check Users to be reviewers
* Reviewers can be a team or user
* Fix check by id or name
* Review can be team or user both
* Validate Reviewers
* add multi select control
* - Fix Reviewers
* - Centralize Reviewer Relationship to EntityRepository
* - Sort
* add team as reviewer for glossary terms
* locales
* cleanup
* - Update Reviewer should remove existing reviewers
* fix selectable owner control
* fix code smells
* fix reviewer issue
* add glossary cypress
* fix patch issue on reviewers set to null
* update cypress tests
* fix cypress
* fix cypress
* fix reviewers in glossary task and supported cypress
* fix pytest
* Fix
* fix cypress
* fix code smells
* Inherited Reviewers need to be present always
* filter out inherited users
* fix cypress
* fix backend tests failure
* fix backend tests failure -checkstyle
* restrict owner to accept task in case of reviewer present
* fix pytest
---------
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
(cherry picked from commit 9ec3d94e3b8445e63a7d77239c92c92a32536bf2)
* Add testSuite tags, domain field and check for TestCase limits
* fix owner not showing after refersh in teams page (#16567)
(cherry picked from commit 119fcf8959732a980b75e1f795a9f2dc5288cd27)
* [ISSUE-16503] Fix createUser to use EntityResource (#16549)
* Fix createUser to use EntityResource
* fix broken tests
* Fix Tests - 3
(cherry picked from commit aeb020ae3b0cbab3a2ee5995c61480cdd1eae405)
* what's new for 1.4.2 (#16568)
(cherry picked from commit c86468d9929e433922886852381269b46d69c832)
* address feedbacks
* fix error for bots page
* update banner text
* allow force fetch limit
* fix ingestion schedule
* Revert "Merge branch '1.4.2' into limits"
This reverts commit 8e965207a23ba527d0f5ba91463c1869077bf091, reversing
changes made to 4d16531965fb0d489a4afdebd45ab5b7f3d1eb5c.
* Merge 1.4.2 (#16578)
* fix explore page conflicts
* fix tests
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Chira Madlani <chirag@getcollate.io>
* fix subheader
* Updating glossary reviewers should propagate reviewers in glossary term (#16580)
* highlight inherited reviewer in glossary
* locales
* use glossary name for search query
* fix glossary version cypress
* add union datatype for subfields
* Adding reviewer to glossary also adds them as an assignee to the task
* add glossary approval cypress
---------
Co-authored-by: sonikashah <sonikashah94@gmail.com>
(cherry picked from commit 4c8bf1cac14074df87dafe7a719e2795b0a29895)
* Update documentation for Search Index apis (#16539)
(cherry picked from commit d3123c49143652015c416d271d9fd0f9cfa9e324)
* cypress: fixed flakiness and announcment cypress (#16579)
* fetch latest limit for create / delete operations
* guard datAsset limit got topic, dashboard, mlmodel etc
* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias (#16589)
* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias
* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias
(cherry picked from commit 8723b8c36afe31410c31d1ebbdafe7b1770921fa)
* cypress: fixed cypress AUT for mysql (#16446)
* cypress: fixed cypress AUT for mysql
* minor fix
* skip announcment redirection cypress
* Minor: Ensure correct index mapping in Elasticsearch for clusterAlias (#16598)
(cherry picked from commit 04543722a6f6e2b1eaf7a451ebb1c176862bc346)
* Fix Postgres Application listing (#16600)
* Fix Postgres Application listing
* Fix Listing
(cherry picked from commit 77dfe1f6af53d187ff7a61fdb1e1416de7178f5a)
* fix limit related issue
* Fix Automations limits invalidation during the uninstall
* cypress: fixed 1.4.2 AUT cypress (#16602)
* cypress: fixed 1.4.2 AUT cypress
* fix cypress around announcement,user,glossary, lineage and mydata
* searchIndexApplication fix and minor changes
---------
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
* test: add updateJWTTokenExpiryTime util (#16606)
(cherry picked from commit 8c173bed6a279cb0a648bd30632ea6ebdf4a2a90)
* OSS changes for adding automator cypress tests (#16611)
* Fix Test Suite Filter (#16615)
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
(cherry picked from commit 3db41f08e27f388495040e5b23cc7bee5ae665f1)
* MINOR: Fix Profiler for SSL Enabled Source (#16613)
* Add Test Suite SSL (#16619)
* MINOR: Fix ssl connection in usage & lineage (#16625)
* Fix owner notification (#16629)
* - Fix Task notification not getting sent to owners
* - Fix Task notification not getting sent to owners
(cherry picked from commit cc2d581eb0524604b6dcf0523e9ca96e0b8a6ce3)
* chore(release): Prepare Branch for `1.4.3`
* - Fix User Signup (#16667)
(cherry picked from commit b4cba8a850ecd7a25aeff6ca7dea0dc432d43d86)
* - Fix User Signup - p2
(cherry picked from commit d9ae6f6db9891f8e9bf7ad49c561a71dd50103da)
* - Update What's new (#16669)
- fix vulnerability
(cherry picked from commit 1dcb1bd46f9da49764f4c61a7ac5048dd2fa956b)
* Minor: Fix incorrect alert on signup page (#16666)
* Fix Application enforceLimits during install
* Wrap the add test button with limits wrapper for column profile tab
* fix errors
* fix tests
* fix pylint
* fix tests
* fix limits
* pylint
* fix schedule options
* fix glossary spec failure
* Add domain & tags to testSuite
* Update airflow-apis-tests-3_9.yml
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Chira Madlani <chirag@getcollate.io>
Co-authored-by: Pablo Takara <pjt1991@gmail.com>
Co-authored-by: Akash-Jain <15995028+akash-jain-10@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Antoine Balliet <antoine.balliet@gorgias.com>
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Split ExternalSecretsManagerTest to new ExternalSecretsManagerTest and AWSBasedSecretsManagerTest
* implement SecretsManagerFactory to create GCPSecretsManager
* implement GCPSecretsManager
* implements gcp secret manager.
* Fix it for the GCP's rule.
* create a template of GCP
* fix compile error
* implements to use project_id
* add library for the google cloud secret manager
* add test code for using google credential in the docker container
* modify docker-compose.yml for GCP
* add google_crc32c module
* modify ways to get project id
* create a new docker-compose.yml for Google Cloud
* create a new document
* create compose file for gcp secret manager
* fix invalid styles and formats
* downgrade google library to avoid conflicting protoc versions
* Revert "fix(CI): update `run_local_docker.sh` script with cluster alias ES endpoint (#16604)"
This reverts commit baab52076f71f76e8bc48d89564ba733ac77a1e6.
* Revert "chore(CI): Enable Elasticsearch cluster alias by default (#16599)"
This reverts commit a3524e14d3a1244db19199ab1eae3796ba15b499.
* WIP - MINOR - Clean ingestion dependencies
* test
* test
* Clean imports
* add pyiceberg for test
* Revert "add pyiceberg for test"
This reverts commit ab26942736586f089a57a644ffd727aca200db62.
* add pyiceberg for test
* Remove docker dep
* clean local docker sh
* MINOR - AKS Airflow troubleshooting docs
* Fix action
* clean local docker sh
* feat: use native backup tools
1. added mysqldump 8.3 to the ingestion container.
2. documented how to use native tools to back up and restore.
3. added deprecated message on the cli backup and restore.
* added deprecation notice for 1.3 backup
* removed 1.3.x deprecation notice
* added another backup page in 1.3 introducing SQL dump tools
* added --set-gtid-purged=OFF to the mysql dump process
* feat(nosql-profiler): row count
1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.
* added profiler support for mongodb connection
* doc
* use int_admin_ometa in test setup
* - fixed linting issue in gx
- removed unused inheritance
* moved the nosql function into the metric class
* formatting
* validate_compose: raise exception for bad status code.
* fixed import
* format
