* feat(table-diff): added column validation
added column validation for table diff that will be carried out before running the row level diff. If a diff for the column exists, it will short circuit the test and report.
* fixed unit tests
* format
* - resolve column types more robustly
- changed test result metric to include "rows" or "columns"
* fix(data-quality): incompatible columns
gracefully fail when a column of incompatible type is submitted for a test case
* format
* added condition to handle only colum test cases
* fixed tests
* format
* Limits
* Limits
* - Mismatched Types
* Update Limits config response
* Update Limits feature response
* Limits
* Limits
* - Mismatched Types
* Update Limits config response
* Update Limits feature response
* Limits: add entity resource enforcer
* Limits: fix rebase
* update limits enforcement
* Add OperationContext to limits
* chore: Bump versions to `1.4.0`
* chore: Bump Ingestion Versions to `1.4.0.1` for Release
* chore: Bump Ingestion Versions to `1.4.0.1` in Dockerfiles for Release
* Remove Retry From Abstract Event Consumer (#16405)
(cherry picked from commit f8ed079731cc238dc136306fe018c5df35dd2f3b)
* Fix Migrations: Add postgres migrations (#16403)
(cherry picked from commit 9416a7ac5fa8fd9695063b108501790d813e8e6e)
* Add Null Check for isAdmin (#16407)
* Remove Retry From Abstract Event Consumer
* - Add Check for null Or Empty in isAdmin
* - Fix Test
(cherry picked from commit fe2db2d63c5495b6c288d4252a19ab77481b6de0)
* Fix OpenLineage ingestor (#16416)
* Fix OpenLineage ingestor
* py format
---------
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
* Minor: added whats new for 1.4.1 (#16420)
* Minor: added whats new for 1.4.1
* added note in to whats new
* Fix SSL issue (#16412)
* chore: Bump Versions for `1.4.1` Release
* chore(release): Prepare Branch for `1.4.2`
* [MINOR] partition migration issue with redshift servics (#16452)
* fix: partition migration issue with redshift servics
* chore: typo in sql comment
(cherry picked from commit 451d73593e813151c24f2c1d17efb3dcdebb71c8)
* minor(ui): update what's new for 1.4.2 (#16457)
(cherry picked from commit d55981adfd2321de706e4a043828bb473a4b05f1)
* fix: ingestion for dbt > 1.8.0 resource_type is not an enum (#16415)
* fix: resource_type is not an enum
* feat: add log to display finis
* improve readability
* use getattr to be compatible
* format
* Add Cache Query Param for Limits
* Only Parse view query (#16470)
* add limit check during user creation via PUT
* add limit check during user creation via PUT
* MINOR: Kafka Setup SSL Arg Fix (#16469)
* Fix#16404 - Show Node level lineage by default (#16445)
* default to node layer
* update cypress
* code cleanup
* fix cypress
(cherry picked from commit f0cda8464f34a21f45f18fa557e980fb2f105d8e)
* Invalidate count of data asset after hard delete. add limit exception to ingestion client
* - Remove Change Description from Lineage (#16488)
(cherry picked from commit 9e5c5529a84dfc781382b3a3b6abd80ee41f11f5)
* - Non Indexable fields should be remvoed at the end (#16499)
(cherry picked from commit f0b0f7a9426ca601d3bfee3989d4ce47e732a7af)
* fix announcement not redirect from landing page (#16506)
* fix announcement not redirect from landing page
* minor changes
* change in cypress test
(cherry picked from commit ee7cddd169a3a1fb1e598e80035c2fc15a5a129b)
* Fix Schema Field Null Issue (#16510)
(cherry picked from commit 022772943f1b33f6230cb35547d1da6acfaf6cfa)
* feat(ui): limits integration with application (#16206)
* feat(ui): limits integration with application
* support pipelineSchedules via limit api
* enforce limit to all the modules
* update banner styling
* update
* support disable option for ManageButton
* limit version
* fix spotlight
* update tests
* Add name and version history to resource limits
Refactor the getEntityIcon function and add new icon mappings
* limit version
* hide access token tab
* fix version for all the entity
* fix tests
* fix DQ tests
* Add fallback for the icon
* Revert the fallback icon changes
* Apply the limit to the add ingestion button in the service details page
* Fix the data quality tab add test button not working
* fix banner styling
* minor fix
* Fix ingestion component unit test
* Add InlineAlert component
* update entityNameLabels mapping object
* Fix the incorrect link in LimitBanner
* update pricing page url
* Create the GlobalSettingsClassBase
* Update URLs for pricing page and upgrade options
* fix global settings uncaught error
* add parameters to the resource limit API
* implement inline alerts for service and alert creation form
* update PRIVILEGES for docker
* fix layout issues
* fix tests
---------
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
* Add token limitations
* Add token limitations
* Add appType as part of schema in ingestion pipeline (#16519)
* #16489: fix the redirect issue to new tab for tags and glossary (#16512)
* fix the redirect issue to new tab for tags and glossary
* fix the redirect on cancel icon and unit test issue
* changes as per comments
(cherry picked from commit 8d312f0853609cfef260739cf789d459838a3421)
* Fix #16229 - Tag and Service filters for test cases (#16484)
* fix: added test case support for tags (inherit from table/column)]"
* feat: add tag and service filter for test cases
* feat: add tier query param
* fix: tests
(cherry picked from commit 6b00dde90285924445567ee7c396c89f0fcf3f1d)
* fix: None type is not iterable (#16496)
(cherry picked from commit 656da03b14ca24171cf7924b9dd33663e6bed423)
* minor(ui): refresh token for OIDC SSO (#16483)
* minor(ui): refresh token for OIDC SSO
* remove frame window timeout issue
* increase iFrame timeout for oidc
(cherry picked from commit 1a6c4c972052836a9b3cfa273b7ea1aa3202eafe)
* feat(ui): support tag & tier filter for test case (#16502)
* feat(ui): support tag & tier filter for test case
* fix tag filter
* allow single select for tier
* added service name filter
* update cypress for tags, tier & service
* add specific add for filters
* fix tier api call
(cherry picked from commit 5b71d79e8ac2d08a154882dfe71b9b3a0f73bffc)
* minor: sanitize activity feed editor content (#16533)
* Add appType as part of schema in ingestion pipeline (#16519)
* Fixed quicksight conn (#16537)
* fix: saml auth for new user not created (#16543)
* fix: saml auth for new user not created
* doc: add comment
* Fix#16491 - fix lineage edge description update (#16538)
* fix lineage edge description update
* fix tests
(cherry picked from commit dff0aa8dbedcd4064ad63765cadda65bb998772e)
* CYPRESS: fix announcement cypress (#16536)
* fix announcement cypress
* changes as per comments
* fix the cypress failure
(cherry picked from commit fcb87b5866ba06aa7a6db516677e311c24053db7)
* [MINOR] Fix Test Failure for EventRegistration
* [MINOR] Fix Test Failure for EventRegistration
* [MINOR] Fix Test Failure for EventRegistration
[MINOR] Fix Test Failure for EventRegistration
* Fix Event Handlers registration Issue (#16544)
* Fix Event Handlers Issue
* Review Comments
(cherry picked from commit d374e48b7938e8ad3514dc5cf8dff619a12595e3)
* [MINOR] Fix Test Failure for EventRegistration
(cherry picked from commit 4563ad4fd10f9790c21fe744d8fc131ebd028ac8)
* Fix Topic Schema missing messageSchema (#16545)
(cherry picked from commit b612dd90c07f564d38392b1ccfe0de1505a4867b)
* Add limits exception cache in rest client
* MINOR: Ignore Cluster Information from columns (#16495)
* minor: improve the block editor initial content history (#16540)
* Minor: fixed data quality page type issue (#16556)
* #16521: fix issue in userProfilePage for roles. teams and displayName (#16527)
* fix update on roles and backlink them in user profile page
* fix teams, displayName and profile pic issue
* sonar fix
* fix cypress issue
* minor changes
(cherry picked from commit 98945cb2db87ebb325d3a72131f049abffcba345)
* Empty quick filters (#16402)
* initial commit for empty quick filters
* update progress
* fix field title
* cleanup
* add tests
* unit tests
* fix encoding of search query
* add cypress tests
* add cypress
* fix flaky cypress
* fix review comments
* revert tooltip changes
* fix tests
* fix tests
(cherry picked from commit 5930cd7a7a4bef73f6850848c85118eb64843e2d)
* Fix#16278 : Search to display Draft glossaryTerms on Explore page (#16462)
* Fix#16278 : Search to display Draft glossaryTerms as well on Explore page
* add term status quick filter
* change aggregation key for status field
* change aggregation key for status field
* add lowercase_normalizer in status filed for aggregate api
* add cypress tests
* fix cypress
---------
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
(cherry picked from commit ae5e9d61cc9e6a39d65972987de9149a421395b1)
* [FIX] GlossaryTerm reviewers should be user or team only (#16372)
* add teams as reviewer
* Check Users to be reviewers
* Reviewers can be a team or user
* Fix check by id or name
* Review can be team or user both
* Validate Reviewers
* add multi select control
* - Fix Reviewers
* - Centralize Reviewer Relationship to EntityRepository
* - Sort
* add team as reviewer for glossary terms
* locales
* cleanup
* - Update Reviewer should remove existing reviewers
* fix selectable owner control
* fix code smells
* fix reviewer issue
* add glossary cypress
* fix patch issue on reviewers set to null
* update cypress tests
* fix cypress
* fix cypress
* fix reviewers in glossary task and supported cypress
* fix pytest
* Fix
* fix cypress
* fix code smells
* Inherited Reviewers need to be present always
* filter out inherited users
* fix cypress
* fix backend tests failure
* fix backend tests failure -checkstyle
* restrict owner to accept task in case of reviewer present
* fix pytest
---------
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
(cherry picked from commit 9ec3d94e3b8445e63a7d77239c92c92a32536bf2)
* Add testSuite tags, domain field and check for TestCase limits
* fix owner not showing after refersh in teams page (#16567)
(cherry picked from commit 119fcf8959732a980b75e1f795a9f2dc5288cd27)
* [ISSUE-16503] Fix createUser to use EntityResource (#16549)
* Fix createUser to use EntityResource
* fix broken tests
* Fix Tests - 3
(cherry picked from commit aeb020ae3b0cbab3a2ee5995c61480cdd1eae405)
* what's new for 1.4.2 (#16568)
(cherry picked from commit c86468d9929e433922886852381269b46d69c832)
* address feedbacks
* fix error for bots page
* update banner text
* allow force fetch limit
* fix ingestion schedule
* Revert "Merge branch '1.4.2' into limits"
This reverts commit 8e965207a23ba527d0f5ba91463c1869077bf091, reversing
changes made to 4d16531965fb0d489a4afdebd45ab5b7f3d1eb5c.
* Merge 1.4.2 (#16578)
* fix explore page conflicts
* fix tests
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Chira Madlani <chirag@getcollate.io>
* fix subheader
* Updating glossary reviewers should propagate reviewers in glossary term (#16580)
* highlight inherited reviewer in glossary
* locales
* use glossary name for search query
* fix glossary version cypress
* add union datatype for subfields
* Adding reviewer to glossary also adds them as an assignee to the task
* add glossary approval cypress
---------
Co-authored-by: sonikashah <sonikashah94@gmail.com>
(cherry picked from commit 4c8bf1cac14074df87dafe7a719e2795b0a29895)
* Update documentation for Search Index apis (#16539)
(cherry picked from commit d3123c49143652015c416d271d9fd0f9cfa9e324)
* cypress: fixed flakiness and announcment cypress (#16579)
* fetch latest limit for create / delete operations
* guard datAsset limit got topic, dashboard, mlmodel etc
* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias (#16589)
* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias
* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias
(cherry picked from commit 8723b8c36afe31410c31d1ebbdafe7b1770921fa)
* cypress: fixed cypress AUT for mysql (#16446)
* cypress: fixed cypress AUT for mysql
* minor fix
* skip announcment redirection cypress
* Minor: Ensure correct index mapping in Elasticsearch for clusterAlias (#16598)
(cherry picked from commit 04543722a6f6e2b1eaf7a451ebb1c176862bc346)
* Fix Postgres Application listing (#16600)
* Fix Postgres Application listing
* Fix Listing
(cherry picked from commit 77dfe1f6af53d187ff7a61fdb1e1416de7178f5a)
* fix limit related issue
* Fix Automations limits invalidation during the uninstall
* cypress: fixed 1.4.2 AUT cypress (#16602)
* cypress: fixed 1.4.2 AUT cypress
* fix cypress around announcement,user,glossary, lineage and mydata
* searchIndexApplication fix and minor changes
---------
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
* test: add updateJWTTokenExpiryTime util (#16606)
(cherry picked from commit 8c173bed6a279cb0a648bd30632ea6ebdf4a2a90)
* OSS changes for adding automator cypress tests (#16611)
* Fix Test Suite Filter (#16615)
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
(cherry picked from commit 3db41f08e27f388495040e5b23cc7bee5ae665f1)
* MINOR: Fix Profiler for SSL Enabled Source (#16613)
* Add Test Suite SSL (#16619)
* MINOR: Fix ssl connection in usage & lineage (#16625)
* Fix owner notification (#16629)
* - Fix Task notification not getting sent to owners
* - Fix Task notification not getting sent to owners
(cherry picked from commit cc2d581eb0524604b6dcf0523e9ca96e0b8a6ce3)
* chore(release): Prepare Branch for `1.4.3`
* - Fix User Signup (#16667)
(cherry picked from commit b4cba8a850ecd7a25aeff6ca7dea0dc432d43d86)
* - Fix User Signup - p2
(cherry picked from commit d9ae6f6db9891f8e9bf7ad49c561a71dd50103da)
* - Update What's new (#16669)
- fix vulnerability
(cherry picked from commit 1dcb1bd46f9da49764f4c61a7ac5048dd2fa956b)
* Minor: Fix incorrect alert on signup page (#16666)
* Fix Application enforceLimits during install
* Wrap the add test button with limits wrapper for column profile tab
* fix errors
* fix tests
* fix pylint
* fix tests
* fix limits
* pylint
* fix schedule options
* fix glossary spec failure
* Add domain & tags to testSuite
* Update airflow-apis-tests-3_9.yml
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Chira Madlani <chirag@getcollate.io>
Co-authored-by: Pablo Takara <pjt1991@gmail.com>
Co-authored-by: Akash-Jain <15995028+akash-jain-10@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Antoine Balliet <antoine.balliet@gorgias.com>
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* get table ddl for athena tables
* changes in method to get all table ddls
* external table/container lineage for athena
* column lineage for external table lineage
* unittest for athena
* pyformat changes
* add external table lineage unit test
* fix unittest with pydantic v2 changes
* fix unittest formating
* fix code smell
* feat: add tableDiff test case
This changed introduces a "table diff" test case which
compares two tables and fails if they are not identical.
The similarity is made based on a specific "key" (because the test only makes sense when performed on ordered collections).
1. Added the `tableDiff` test definition.
2. Implemented a "runtime" parameters feature which injects additional parameters for the test at runtime.
3. Integration tests (because of course).
This feature was not tested end-to-end yet because "array" data
* pydantic v2
* format
* format
* format and added data diff to setup.py
* format
* fixed param issue which has type ARRAY
* fixed runtime_parameter_setter
* moved models to parent directory
* handle errors in table diff
* fixed issue with edit test case
* format
* added more details to pytest skip
* format
* refactor: Improve createTestCaseParameters function in DataQualityUtils
* fixed unit test
* removed unused fixture
* removed validator.py
* fixed tests
* added validate kwarg to tests_mixin
* removed "postgres" data diff extra as they interfere with psycopg2-binary
* fixed tests
* pinned tenacity for tests
* reverted tenacity pinning
* added ui support for test diff
* fixed dq cypress and added edit flow
* organized the test case
* added dialect support
* fixed tests
* option style fix
* fixed calculation for passing/failing rows
* restrict the tableDiff test to limited services
* set where to None if blank string
* fixed where clause
* fixed tests for where clause
* use displayName in place of name in edit form
* added docs for RuntimeParameterSetter
* fixed cypress
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
* Fix Metabase E2E Test
* Add 'debug' input to python e2e tests
* Fix 'debug' default to be 'false'
* Standardized all Metabase IDs to MetabaseStrId
* Fix Metabase expected filtered sink mix value
* Fix wrong parameter being passed to the config
* Fix powerBI e2e tests
* Fix one Redash e2e test
* Fix checkstyle
* add teams as reviewer
* Check Users to be reviewers
* Reviewers can be a team or user
* Fix check by id or name
* Review can be team or user both
* Validate Reviewers
* add multi select control
* - Fix Reviewers
* - Centralize Reviewer Relationship to EntityRepository
* - Sort
* add team as reviewer for glossary terms
* locales
* cleanup
* - Update Reviewer should remove existing reviewers
* fix selectable owner control
* fix code smells
* fix reviewer issue
* add glossary cypress
* fix patch issue on reviewers set to null
* update cypress tests
* fix cypress
* fix cypress
* fix reviewers in glossary task and supported cypress
* fix pytest
* Fix
* fix cypress
* fix code smells
* Inherited Reviewers need to be present always
* filter out inherited users
* fix cypress
* fix backend tests failure
* fix backend tests failure -checkstyle
* restrict owner to accept task in case of reviewer present
* fix pytest
---------
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
