* tests: refactor
refactor tests and consolidate common functionality in integrations.conftest
this enables writing tests more concisely.
demonstrated with postgres and mssql.
will migrate more
* format
* removed helpers
* changed scope of fictures
* changed scope of fixtures
* added profiler test for mssql
* fixed import in data_quality test
* json safe serialization
* format
* set MARS_Connection
* use SerializableTableData instead of TableData
* deleted file test_postgres.py
* fixed tests
* added more test cases
* format
* changed name test_models.py
* removed the logic for serializing table data
* wip
* changed mapping in common type map
* changed mapping in common type map
* reverted TableData imports
* reverted TableData imports
* reverted TableData imports
* feat(table-diff): added column validation
added column validation for table diff that will be carried out before running the row level diff. If a diff for the column exists, it will short circuit the test and report.
* fixed unit tests
* format
* - resolve column types more robustly
- changed test result metric to include "rows" or "columns"
* Limits
* Limits
* - Mismatched Types
* Update Limits config response
* Update Limits feature response
* Limits
* Limits
* - Mismatched Types
* Update Limits config response
* Update Limits feature response
* Limits: add entity resource enforcer
* Limits: fix rebase
* update limits enforcement
* Add OperationContext to limits
* chore: Bump versions to `1.4.0`
* chore: Bump Ingestion Versions to `1.4.0.1` for Release
* chore: Bump Ingestion Versions to `1.4.0.1` in Dockerfiles for Release
* Remove Retry From Abstract Event Consumer (#16405)
(cherry picked from commit f8ed079731cc238dc136306fe018c5df35dd2f3b)
* Fix Migrations: Add postgres migrations (#16403)
(cherry picked from commit 9416a7ac5fa8fd9695063b108501790d813e8e6e)
* Add Null Check for isAdmin (#16407)
* Remove Retry From Abstract Event Consumer
* - Add Check for null Or Empty in isAdmin
* - Fix Test
(cherry picked from commit fe2db2d63c5495b6c288d4252a19ab77481b6de0)
* Fix OpenLineage ingestor (#16416)
* Fix OpenLineage ingestor
* py format
---------
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
* Minor: added whats new for 1.4.1 (#16420)
* Minor: added whats new for 1.4.1
* added note in to whats new
* Fix SSL issue (#16412)
* chore: Bump Versions for `1.4.1` Release
* chore(release): Prepare Branch for `1.4.2`
* [MINOR] partition migration issue with redshift servics (#16452)
* fix: partition migration issue with redshift servics
* chore: typo in sql comment
(cherry picked from commit 451d73593e813151c24f2c1d17efb3dcdebb71c8)
* minor(ui): update what's new for 1.4.2 (#16457)
(cherry picked from commit d55981adfd2321de706e4a043828bb473a4b05f1)
* fix: ingestion for dbt > 1.8.0 resource_type is not an enum (#16415)
* fix: resource_type is not an enum
* feat: add log to display finis
* improve readability
* use getattr to be compatible
* format
* Add Cache Query Param for Limits
* Only Parse view query (#16470)
* add limit check during user creation via PUT
* add limit check during user creation via PUT
* MINOR: Kafka Setup SSL Arg Fix (#16469)
* Fix#16404 - Show Node level lineage by default (#16445)
* default to node layer
* update cypress
* code cleanup
* fix cypress
(cherry picked from commit f0cda8464f34a21f45f18fa557e980fb2f105d8e)
* Invalidate count of data asset after hard delete. add limit exception to ingestion client
* - Remove Change Description from Lineage (#16488)
(cherry picked from commit 9e5c5529a84dfc781382b3a3b6abd80ee41f11f5)
* - Non Indexable fields should be remvoed at the end (#16499)
(cherry picked from commit f0b0f7a9426ca601d3bfee3989d4ce47e732a7af)
* fix announcement not redirect from landing page (#16506)
* fix announcement not redirect from landing page
* minor changes
* change in cypress test
(cherry picked from commit ee7cddd169a3a1fb1e598e80035c2fc15a5a129b)
* Fix Schema Field Null Issue (#16510)
(cherry picked from commit 022772943f1b33f6230cb35547d1da6acfaf6cfa)
* feat(ui): limits integration with application (#16206)
* feat(ui): limits integration with application
* support pipelineSchedules via limit api
* enforce limit to all the modules
* update banner styling
* update
* support disable option for ManageButton
* limit version
* fix spotlight
* update tests
* Add name and version history to resource limits
Refactor the getEntityIcon function and add new icon mappings
* limit version
* hide access token tab
* fix version for all the entity
* fix tests
* fix DQ tests
* Add fallback for the icon
* Revert the fallback icon changes
* Apply the limit to the add ingestion button in the service details page
* Fix the data quality tab add test button not working
* fix banner styling
* minor fix
* Fix ingestion component unit test
* Add InlineAlert component
* update entityNameLabels mapping object
* Fix the incorrect link in LimitBanner
* update pricing page url
* Create the GlobalSettingsClassBase
* Update URLs for pricing page and upgrade options
* fix global settings uncaught error
* add parameters to the resource limit API
* implement inline alerts for service and alert creation form
* update PRIVILEGES for docker
* fix layout issues
* fix tests
---------
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
* Add token limitations
* Add token limitations
* Add appType as part of schema in ingestion pipeline (#16519)
* #16489: fix the redirect issue to new tab for tags and glossary (#16512)
* fix the redirect issue to new tab for tags and glossary
* fix the redirect on cancel icon and unit test issue
* changes as per comments
(cherry picked from commit 8d312f0853609cfef260739cf789d459838a3421)
* Fix #16229 - Tag and Service filters for test cases (#16484)
* fix: added test case support for tags (inherit from table/column)]"
* feat: add tag and service filter for test cases
* feat: add tier query param
* fix: tests
(cherry picked from commit 6b00dde90285924445567ee7c396c89f0fcf3f1d)
* fix: None type is not iterable (#16496)
(cherry picked from commit 656da03b14ca24171cf7924b9dd33663e6bed423)
* minor(ui): refresh token for OIDC SSO (#16483)
* minor(ui): refresh token for OIDC SSO
* remove frame window timeout issue
* increase iFrame timeout for oidc
(cherry picked from commit 1a6c4c972052836a9b3cfa273b7ea1aa3202eafe)
* feat(ui): support tag & tier filter for test case (#16502)
* feat(ui): support tag & tier filter for test case
* fix tag filter
* allow single select for tier
* added service name filter
* update cypress for tags, tier & service
* add specific add for filters
* fix tier api call
(cherry picked from commit 5b71d79e8ac2d08a154882dfe71b9b3a0f73bffc)
* minor: sanitize activity feed editor content (#16533)
* Add appType as part of schema in ingestion pipeline (#16519)
* Fixed quicksight conn (#16537)
* fix: saml auth for new user not created (#16543)
* fix: saml auth for new user not created
* doc: add comment
* Fix#16491 - fix lineage edge description update (#16538)
* fix lineage edge description update
* fix tests
(cherry picked from commit dff0aa8dbedcd4064ad63765cadda65bb998772e)
* CYPRESS: fix announcement cypress (#16536)
* fix announcement cypress
* changes as per comments
* fix the cypress failure
(cherry picked from commit fcb87b5866ba06aa7a6db516677e311c24053db7)
* [MINOR] Fix Test Failure for EventRegistration
* [MINOR] Fix Test Failure for EventRegistration
* [MINOR] Fix Test Failure for EventRegistration
[MINOR] Fix Test Failure for EventRegistration
* Fix Event Handlers registration Issue (#16544)
* Fix Event Handlers Issue
* Review Comments
(cherry picked from commit d374e48b7938e8ad3514dc5cf8dff619a12595e3)
* [MINOR] Fix Test Failure for EventRegistration
(cherry picked from commit 4563ad4fd10f9790c21fe744d8fc131ebd028ac8)
* Fix Topic Schema missing messageSchema (#16545)
(cherry picked from commit b612dd90c07f564d38392b1ccfe0de1505a4867b)
* Add limits exception cache in rest client
* MINOR: Ignore Cluster Information from columns (#16495)
* minor: improve the block editor initial content history (#16540)
* Minor: fixed data quality page type issue (#16556)
* #16521: fix issue in userProfilePage for roles. teams and displayName (#16527)
* fix update on roles and backlink them in user profile page
* fix teams, displayName and profile pic issue
* sonar fix
* fix cypress issue
* minor changes
(cherry picked from commit 98945cb2db87ebb325d3a72131f049abffcba345)
* Empty quick filters (#16402)
* initial commit for empty quick filters
* update progress
* fix field title
* cleanup
* add tests
* unit tests
* fix encoding of search query
* add cypress tests
* add cypress
* fix flaky cypress
* fix review comments
* revert tooltip changes
* fix tests
* fix tests
(cherry picked from commit 5930cd7a7a4bef73f6850848c85118eb64843e2d)
* Fix#16278 : Search to display Draft glossaryTerms on Explore page (#16462)
* Fix#16278 : Search to display Draft glossaryTerms as well on Explore page
* add term status quick filter
* change aggregation key for status field
* change aggregation key for status field
* add lowercase_normalizer in status filed for aggregate api
* add cypress tests
* fix cypress
---------
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
(cherry picked from commit ae5e9d61cc9e6a39d65972987de9149a421395b1)
* [FIX] GlossaryTerm reviewers should be user or team only (#16372)
* add teams as reviewer
* Check Users to be reviewers
* Reviewers can be a team or user
* Fix check by id or name
* Review can be team or user both
* Validate Reviewers
* add multi select control
* - Fix Reviewers
* - Centralize Reviewer Relationship to EntityRepository
* - Sort
* add team as reviewer for glossary terms
* locales
* cleanup
* - Update Reviewer should remove existing reviewers
* fix selectable owner control
* fix code smells
* fix reviewer issue
* add glossary cypress
* fix patch issue on reviewers set to null
* update cypress tests
* fix cypress
* fix cypress
* fix reviewers in glossary task and supported cypress
* fix pytest
* Fix
* fix cypress
* fix code smells
* Inherited Reviewers need to be present always
* filter out inherited users
* fix cypress
* fix backend tests failure
* fix backend tests failure -checkstyle
* restrict owner to accept task in case of reviewer present
* fix pytest
---------
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
(cherry picked from commit 9ec3d94e3b8445e63a7d77239c92c92a32536bf2)
* Add testSuite tags, domain field and check for TestCase limits
* fix owner not showing after refersh in teams page (#16567)
(cherry picked from commit 119fcf8959732a980b75e1f795a9f2dc5288cd27)
* [ISSUE-16503] Fix createUser to use EntityResource (#16549)
* Fix createUser to use EntityResource
* fix broken tests
* Fix Tests - 3
(cherry picked from commit aeb020ae3b0cbab3a2ee5995c61480cdd1eae405)
* what's new for 1.4.2 (#16568)
(cherry picked from commit c86468d9929e433922886852381269b46d69c832)
* address feedbacks
* fix error for bots page
* update banner text
* allow force fetch limit
* fix ingestion schedule
* Revert "Merge branch '1.4.2' into limits"
This reverts commit 8e965207a23ba527d0f5ba91463c1869077bf091, reversing
changes made to 4d16531965fb0d489a4afdebd45ab5b7f3d1eb5c.
* Merge 1.4.2 (#16578)
* fix explore page conflicts
* fix tests
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Chira Madlani <chirag@getcollate.io>
* fix subheader
* Updating glossary reviewers should propagate reviewers in glossary term (#16580)
* highlight inherited reviewer in glossary
* locales
* use glossary name for search query
* fix glossary version cypress
* add union datatype for subfields
* Adding reviewer to glossary also adds them as an assignee to the task
* add glossary approval cypress
---------
Co-authored-by: sonikashah <sonikashah94@gmail.com>
(cherry picked from commit 4c8bf1cac14074df87dafe7a719e2795b0a29895)
* Update documentation for Search Index apis (#16539)
(cherry picked from commit d3123c49143652015c416d271d9fd0f9cfa9e324)
* cypress: fixed flakiness and announcment cypress (#16579)
* fetch latest limit for create / delete operations
* guard datAsset limit got topic, dashboard, mlmodel etc
* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias (#16589)
* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias
* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias
(cherry picked from commit 8723b8c36afe31410c31d1ebbdafe7b1770921fa)
* cypress: fixed cypress AUT for mysql (#16446)
* cypress: fixed cypress AUT for mysql
* minor fix
* skip announcment redirection cypress
* Minor: Ensure correct index mapping in Elasticsearch for clusterAlias (#16598)
(cherry picked from commit 04543722a6f6e2b1eaf7a451ebb1c176862bc346)
* Fix Postgres Application listing (#16600)
* Fix Postgres Application listing
* Fix Listing
(cherry picked from commit 77dfe1f6af53d187ff7a61fdb1e1416de7178f5a)
* fix limit related issue
* Fix Automations limits invalidation during the uninstall
* cypress: fixed 1.4.2 AUT cypress (#16602)
* cypress: fixed 1.4.2 AUT cypress
* fix cypress around announcement,user,glossary, lineage and mydata
* searchIndexApplication fix and minor changes
---------
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
* test: add updateJWTTokenExpiryTime util (#16606)
(cherry picked from commit 8c173bed6a279cb0a648bd30632ea6ebdf4a2a90)
* OSS changes for adding automator cypress tests (#16611)
* Fix Test Suite Filter (#16615)
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
(cherry picked from commit 3db41f08e27f388495040e5b23cc7bee5ae665f1)
* MINOR: Fix Profiler for SSL Enabled Source (#16613)
* Add Test Suite SSL (#16619)
* MINOR: Fix ssl connection in usage & lineage (#16625)
* Fix owner notification (#16629)
* - Fix Task notification not getting sent to owners
* - Fix Task notification not getting sent to owners
(cherry picked from commit cc2d581eb0524604b6dcf0523e9ca96e0b8a6ce3)
* chore(release): Prepare Branch for `1.4.3`
* - Fix User Signup (#16667)
(cherry picked from commit b4cba8a850ecd7a25aeff6ca7dea0dc432d43d86)
* - Fix User Signup - p2
(cherry picked from commit d9ae6f6db9891f8e9bf7ad49c561a71dd50103da)
* - Update What's new (#16669)
- fix vulnerability
(cherry picked from commit 1dcb1bd46f9da49764f4c61a7ac5048dd2fa956b)
* Minor: Fix incorrect alert on signup page (#16666)
* Fix Application enforceLimits during install
* Wrap the add test button with limits wrapper for column profile tab
* fix errors
* fix tests
* fix pylint
* fix tests
* fix limits
* pylint
* fix schedule options
* fix glossary spec failure
* Add domain & tags to testSuite
* Update airflow-apis-tests-3_9.yml
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Chira Madlani <chirag@getcollate.io>
Co-authored-by: Pablo Takara <pjt1991@gmail.com>
Co-authored-by: Akash-Jain <15995028+akash-jain-10@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Antoine Balliet <antoine.balliet@gorgias.com>
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* get table ddl for athena tables
* changes in method to get all table ddls
* external table/container lineage for athena
* column lineage for external table lineage
* unittest for athena
* pyformat changes
* add external table lineage unit test
* fix unittest with pydantic v2 changes
* fix unittest formating
* fix code smell
* feat: add tableDiff test case
This changed introduces a "table diff" test case which
compares two tables and fails if they are not identical.
The similarity is made based on a specific "key" (because the test only makes sense when performed on ordered collections).
1. Added the `tableDiff` test definition.
2. Implemented a "runtime" parameters feature which injects additional parameters for the test at runtime.
3. Integration tests (because of course).
This feature was not tested end-to-end yet because "array" data
* pydantic v2
* format
* format
* format and added data diff to setup.py
* format
* fixed param issue which has type ARRAY
* fixed runtime_parameter_setter
* moved models to parent directory
* handle errors in table diff
* fixed issue with edit test case
* format
* added more details to pytest skip
* format
* refactor: Improve createTestCaseParameters function in DataQualityUtils
* fixed unit test
* removed unused fixture
* removed validator.py
* fixed tests
* added validate kwarg to tests_mixin
* removed "postgres" data diff extra as they interfere with psycopg2-binary
* fixed tests
* pinned tenacity for tests
* reverted tenacity pinning
* added ui support for test diff
* fixed dq cypress and added edit flow
* organized the test case
* added dialect support
* fixed tests
* option style fix
* fixed calculation for passing/failing rows
* restrict the tableDiff test to limited services
* set where to None if blank string
* fixed where clause
* fixed tests for where clause
* use displayName in place of name in edit form
* added docs for RuntimeParameterSetter
* fixed cypress
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
* Added Test Case for Matching Enum
1. Implemented the test case using the `matchEnum` parameter.
2. Added integration tests.
3. Added migrations.
* fix tests
* fixed tests
* format
* fixed tests
* clear search cache before running ingestion
* format
* changed scopt of aws fixture
* moved migrations to 1.5.0
* MINOR - Add Integration Test for S3 Storage
* MINOR - Add Integration Test for S3 Storage
* MINOR - Add Integration Test for S3 Storage
* format
* format
* issue-15719: unit test for superset db source
* issue-15719: use testcontainers for superset_api client test
* issue-15719: superset-api yield data changes
* fix failed test cases due to testcontainer version
* issue-15719: postgres container version fix
* issue-15719: setup & teardown with testcontainers
* issue-15719: remove more patch code
* feat: add global metric configuration for the profiler
* style: ran java linting
* fix: renamed disable to disabled
* style: ran java linting
* feat: ometa sdk for profiler setting
* test: ingestion profiler global config tests
* fix: update metric name to use MetricType Enum
* fix: allow bot to retrieve settings
* fix: exclude GX artifacts
* feat: implement global profiler setting logic for ingestion side
* fix: exclude metrics if Metric is empty
* style: ran python linting
* style: ran python linting
* fix: skip empty metrics
* style: ran python linting
* fix: moved GET profiler config to seperate endpoint in system resource
* fix: moved compute metric filter to MetricFilter + renamed container
* fix: test failures
* fix: profiler test case
* feat(nosql-profiler): row count
1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.
* added profiler support for mongodb connection
* doc
* use int_admin_ometa in test setup
* - fixed linting issue in gx
- removed unused inheritance
* moved the nosql function into the metric class
* feat(profiler): add dynamodb row count
* feat(profiler): add dynamodb row count
* formatting
* fixed import
* format
* dded dynamodb row count
* format
* removed unused factory file
* removed "validate"
* migrations
* removed validations
* format
* linting
* fixed: test_amundsen.py
* Update schemaChanges.sql
