* Improve parallelization in SearchIndexApp
* Improve parallelization in SearchIndexApp
* add countdown latch
* typo
* Status issues - #1
* Improve search indexing
* cleanup the code
* schemas
* Status Updates Via WebSockets
* Add Updates
* Make Logs Debug
* show duration in table
* add searchIndexApp params to openmetadata-ops.sh reindex commandline
* Fix code style
* Add Parallel Entity Level Reader
* Make code more readable
* Send App Run Record instead of jobData
* Sned App Run Record WebSocket for Data Insights
* ui: use socket connection to get the real-time updates of application data
* Send WebSocket Final Update
* ui: remove failedRecords, totalRecords and successRecordsfrom entities record
* Invert Condition
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
* fix: list records from latest to earliest.
* refactor: remove listing unprocessed events from listEvents API.
* refactor: EventSubscriptionDiagnosticInfo.
* fix getSuccessfullySentChangeEventsForAlert
* migrations.
* offset-based pagination
* add startingOffset in EventSubscriptionDiagnosticInfo
* fix missing paginationOffset
* Record successfully sent change events in a separate table, clean up associated successful and failed records on alert deletion.
* Remove oldest 30 records if record count reaches 100 for successful_sent_change_events.
* list unprocessed changeEvents based on alert subscription rules, ensuring only relevant unprocessed events are listed.
* refactor: enhance eventSubscriptionDiagnosticInfo
Added new fields to the eventSubscriptionDiagnosticInfo schema to support detailed diagnostics:
- `relevantUnprocessedEventsCount`: Count of relevant unprocessed events based on alert filtering rules.
- `relevantUnprocessedEventsList`: List of relevant unprocessed events based on alert filtering rules.
- `totalUnprocessedEventsCount`: Count of all unprocessed events.
- `totalUnprocessedEventsList`: List of all unprocessed events.
* Refactor: delete 30 oldest records per alert when the count reaches 100 records
* fix: table creation issue in PostgreSQL
* fix tests.
* scheduleCleanupJob
* remove in-line deletion of records.
* fix postgres table structure issue.
* feat(statistics-profiler): use statistics tables to profile trino tables
- implemented the collaborative root class
- added the "useStatistics" profiler parameter
- added the "supportsStatistics" database connection property
- implemented the ProfilerWithStatistics and StoredStatisticsSource to add this functionality to specific profilers
- implemented TrinoStoredStatisticsSource for specific trino statistics logic
* added ABC to terminal classes in collaborative root
* fixed docstring for TestSuiteInterface
* reverted unintended changes
* typo
* Add flake.nix
* Add lockfile for flake
* Update nix environment and document usage
* Add schema for exasol connector
* Add Exasol definitions to databaseService
* Fix error in exasol connector schema
* Add additional connection options/settings to exasol connector
* Add exasol-connector to ui
* Add depdencies for exasol-connector
* Update notes
* Update ingestion code
* Add Basic Documentation for Exasol Connector
* Update flake file
* Add developer notes
* Add python script which can be used as entry point for debugging in ide
* Add config file which can be used for debugging (manual execution)
* Update debug script
* Update developer notes
* Remove old developer notes
* Add .venv to gitignore
* Update dev notes
* Update development notes
* Update ExasolSource
* Establish basic connection to Exasol DB from connector
* Update exasol connector connection settings
* Add service_spec for exasol plugin
* Remove development files
* Remove unused module
* Applied code formatter
* Update exasol dependency constraint(s)
* Add unit test for exasol connection url(s)
* Fixed test expectations for exasol connection url test(s)
* Adjust the test query for the Exasol connection test
* create EventSubscriptionDiagnosticInfo
* API for failedEvents
* migrations
* add index on source
* list all failedEvents optionally filtered by source
* refactor
* refactor
* getSuccessfullySentChangeEventsForAlert API
* increase defaultValue of limit to 100
* resolve conflicts
* listEvents API with 'failed', 'successful,' and 'unprocessed' query parameters
* fix description
* eventSource as enums
* refactor name.
---------
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
* fix: entity link accepted characters
* fix: match all characters but `<>|::`
* fix: remove unnecessary negative lookbehind
* fix the test case search not working on add ingestion page in case of special characters in FQN
* style: ran java linting
* fix: failing testst
* style: ran java linting
---------
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
* Add ER diag APIs and sample data
* fix pylint
* formatting fixes2
* fixed es client return
* fixed os client return
* supported TableDetailPage tabs as classBase for supporting collate only tabs
* Added schema Apis
* change the base class to .ts and move the component in the util files
* beautify function arguments
* Added optimizations
* Ingestion changes
* svg dimension change
* supported class base tab in databaseSchema
* supported classBase action button in schema table name column
* added further keys data for constraint modal
* fix sonar issue
* remove old method to override edit action on column and shifted to DisplayNameModal for fields
* supported table right panel component to further extends on collate side
* minor fix around duplicate constraint
* added support to update table constraints and column constraints in the UI
* code optimization and minor fixes
* review comments and multi col fix
* added queryFilter option in NodeSuggestion and tableConstrainst to fetch and use only in service tables
---------
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
* MINOR - Renaming MetaPilot to CollateAI
* MINOR - Renaming MetaPilot to CollateAI
* MINOR - Renaming MetaPilot to CollateAI
* MINOR - Renaming MetaPilot to CollateAI
* fix
* moving limits migrations for 1.5.9
* Fix#18007: Disabled Classifications or Tags shouldn't be visible in UI
* added playwright test for disabled tags should not be visible while search
* replace testing tag to new generated one to avoid flakyness
* added test for checking tags are re-enabling it from disabled state
* fix the playwright test for the wrong column selector
---------
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
* Minor: Add an API to list all the properties and custom properties for a given entity type
* Minor: Add an API to list all the properties and custom properties for a given entity type
* add custom properties
* add custom properties
* add custom properties
* fix: custom properties folder name in generation to match expected package name (i.e. customProperties -- uppercase P)
* fix: allow non admin/bot to read profiler global config with the right permission
* style: ran java linting
* fix: custom properties import casing
* Table custom property enhancements
* ui: remove rowCount
* minor style file changes
* minor change to show no data placeholder
* add icons for property types
* fix type filtering issue
* fix spacing issue for custom properties card
* refactor comments and style file
* fix test
* add playwright test for adding/updating table type property in entity
* add sesrchField for search property by name
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
* Feature : Add table-type custom property
* ui: remove enumWithDescriptions property
* Refactor AddCustomProperty component to remove unused EnumWithDescriptionsConfig import and property
* ui: add support for creating and updating the table-type property
* add rowCount field in table-type and clean logic of validation
* add rowCount field in table-type and clean logic of validation
* add rowCount field in table-type and clean logic of validation
* ui: add support for adding value to table type property part 1
* use rowCount from the API
* default row-count to 10
* fix tooltip to read rowcount from the props
* fix empty row issue
* rename from table-type to table-cp
* Refactor table type property to handle empty rows and display in modal
* Refactor table type property to handle empty rows and display in modal
* use locales for messages
* minor change
* ui: use column layout for rendering custom properties cards
* remove enumwithDescriptions from backend and add tests for table type
* fix unit test
* fix playwright
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
* GEN-1322: API Entity - Remove Beta
* minor: add doc for the metadata pipeline
* api service refactor
* api service refactor backend changes
* add apiconnection in test service connection
* pytest fix
* fix java file formatting
* Fix casing of REST in ApiServiceRest.spec.ts
* Refactor REST to Rest in API classes
* minor change
* minor change
* minor change
* fix cashing for API to Api
* add playwright test for api service ingestion
* fix: playwright test
---------
Co-authored-by: harshsoni2024 <harshsoni2024@gmail.com>
* Add support for custom headers and JSON payload configuration in webhook
* move testDestination API to EventSubscriptionResource
* support POST and PUT http ops for generic webhook requests
* fix name
* remove json field from webhook
* fix getEmailTestMessage
* refactor exception handling
* Fix OpenSearch Content Size Issue
entity content is too long [204857600] for the configured buffer limit [104857600]
* Change Type to Long
* Add Payload Size to take dynamic entry
* Migrations for 1.5.6
* Mark Stale Entries Stopped
* Format checkstyle
* Fix failure
---------
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
* fix import issue
* feat: compute test case resolution metrics
* feat: added index key to resolution index
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
* Add new ‘metaEnum’ Custom Property to allow adding Enum Keys with Description
* replace JsonNodeFactory method with JsonUtils
* rename property from metaEnum to enumWithDescriptions, and other method optimizations
* ui: add support for creating enumWithDescription property
* minor locale changes
* ui: add edit support for created enumWithDescription property
* Refactor enum description field layout in AddCustomProperty and EditCustomPropertyModal
* add support for adding values to enumWithDescription custom property type
* Refactor custom property input IDs in AddCustomProperty and EditCustomPropertyModal components
* Refactor custom property table rendering logic and UI components
* Refactor custom property table rendering logic and UI components
* Refactor custom property table rendering logic and UI components
* add basic card layout
* Refactor CustomPropertyTable component to improve UI and functionality
* update playwright test part 1
* Refactor PropertyValue component to conditionally render right panel styles
* fix: entity reference property update
* Refactor CustomPropertyTable component to conditionally render right panel styles
* fix: flaky test
* Refactor CustomPropertyTable test to use updated test IDs and remove unnecessary code
* fix flaky test
* improve the playwright test
* add more test
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
