* fix: made databricks httpPath required and added a migration file for the same
* fix: added sql migration in postDataMigration file and fix databricks tests
* fix: added httpPath in test_source_connection.py and test_source_parsing.py files
* fix: added httpPath in test_databricks_lineage.py
* fix: table name in postgres migration
* Add last login time
* Add change to entitySpecificUpdate
* remove last login time from createOrUpdate
* moved UpdateLastLogin to user repository
* Added last login Time on login (#20419)
* updated last login time
* removed condition
* added unit tests
---------
Co-authored-by: Dhruv Parmar <83108871+dhruvjsx@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
* Refactor SQL column processing and enhance BigQuery project ID handling
* Introduced a new `process_column` function in `sql_column_handler.py` to streamline column processing logic.
* Updated `BigquerySource` to improve project ID retrieval from service connections, ensuring compatibility with various credential types.
* Added handling for nested columns in BigQuery schema processing.
* Enhanced error handling and logging for better debugging during project ID setup.
* Add support for GCP Application Default Credentials in BigQuery ingestion
* Enhanced `BigquerySource` to include handling for GCP Application Default Credentials (ADC).
* Updated JSON schema for GCP credentials to define `gcpADC` and its properties.
* Improved logging for credential setup in `set_google_credentials` function.
* Added comments and TODOs for future enhancements related to project ID fetching from the resource manager.
* Update .gitignore to include cursor rules files
* Added .cursorrules and .cursor/ to the .gitignore to prevent tracking of cursor rule files in the repository.
* This change helps maintain a cleaner repository by excluding unnecessary files from version control.
* refactor: Bigquery Credentials to allow multiple project ids
* fix: Handle unknown array data types in SQL column processing
* feat: allow user to turn off rnadomized sample
* style: ran python linting
* fix: models default value for randomizedSample
* style: ran linting
* doc: move config to advanced
* Fix#19667: OpenSearch Connector
* Fix#19667: OpenSearch Connector
* do not ingest any system level indexes
* fix pyformat
* Add AWS auth
* Use common schema and fix ssl config in client
* Add openseach connector docs and update schema
* Remove api key auth type and complete docs checklist
* Remove unnecessary httpx dependency and pyformat
* Add compatible version of httpx for elasticsearch
* Fix pylint fails and py-tests validation error
---------
Co-authored-by: Mohit Tilala <tilalamohit123@gmail.com>
Co-authored-by: Mohit Tilala <63147650+mohittilala@users.noreply.github.com>
* playwright: skip DI, superset & postgresql test
* skip the run part instead of skipping full test for DI settings
* Fix the postgres Ingestion playwright failure
* Fix the superset playwright
---------
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
* MINOR - Add AgentType to apps
* MINOR - Add AgentType to apps
* update the types according to the schema
* Fix the search lineage request schema
* Add pipelineStatuses field in the ingestionPipelineIndex
* Add MetadataAgentsListWidget
* Localization changes
* added list for app runs by service id
* use `$.services` path
* propogate services field
* Fix the ci type errors
* java format
* use `getJobDetail().getJobDataMap()` instead of `getMergedJobDataMap()`
* Fix the playwright and unit tests
* Fix the service version page playwright failures
* Fix sonarcloud
* Fix the unit tests
* Fix the errors
---------
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: sushi30 <imri.paran@gmail.com>
* Expose Workflow Stage Status and Bring OneDayApp to OSS
* Fix Migrations
* Change Bot and add comment
* Fix checkstyle
* Initial idea for triggering app manually
* Remove merge conflics rows
* Remove merge conflics rows
* Allow to Trigger DayOneApp Manually. Change its Trigger to NoOp
* add docs
* back to main
* Actually take into account the active status
* Fix the generated types
---------
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
* MINOR - Add Test Case Automator
* MINOR - Add Test Case Automator
* MINOR - Add Test Case Automator
* MINOR - Add Test Case Automator
* simplify tc body for automator
* simplify tc body for automator
* simplify tc body for automator
* remove all
* dp
* Initial unification of create and run tasks
* Unify Create and Run Ingestion Pipeline Tasks
---------
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
* - typo update
* - Update lineage model and api
(cherry picked from commit c5cce2b0c7c1be7ba654bfc2677da0d05215a895)
* - Adding patch
* - spotless fix
* Add Migration For createdAt , createdBy, updatedAt and updatedBy
* Moved classes from Elastic Client to Lineage Graph Builder
* Update to use correct depth
* remove toCondition
* Fix Downstream Lineage
* moved not so cool things here and there
* spotless-failures
* call with direction
* add upstream depth with + 1 for 0th node
* Fix Exports
* Fix Exports from and size
* parse lineage in ui
* Add per entity pagination
* Added per entity pagination
* Fix Async Export
* Fix Java Merges
* Fix merge conflict
* Fix missing upstream nodes
* lineage ui initial commit
* Fix Pagination for upstream
* Fix Values
* fix load more loading
* cleanup
* fix loading of nodes
* fix load more
* Fix Direction Values
* update expand collapse logic
* Fix Deletion Issue
* Fix Pipeline Lineage
* show pipeline as a node
* Use Fqn Hash to find entities is es for lineage
* update lineage
* Add Lineage for Pipeline to look for as edge and node
* Layer and size conditions for upstream and downstream
* Add include source fields in lineage request
* redesign controls component
* add lineage fixes
* fix tests
* Fix node
* Fix Some Failing Test on Lineage
* Java Spotless Fix and added missing relation check on migration
* Missing conflict changes
* fix conflicts
* added missing keys
* update tests
* Add Service Information in service indexes
* Add Domain Lineage
* Add domain lineages
* fix tests
* Fix Metric Entity Service Error
* Maintain Asset Count for entities adding service and domain lineage
* Update to Doc Id and add details to search
* Cleanup Service and Domain automatically
* fix types
* Update review comments
* fix icon sizes
* Update Services Index wit upstream lineage
* add service view
* Fix Service Name missingFix Service Name resolution in LineageRepository
Update the method to correctly retrieve the service name for the "fromEntity" by specifying the FIELD_SERVICE path. This resolves an issue where the service name was not properly populated.
* - Add docUniqueId to lineage data and update related queries
- Added lenient Object Mapper for search index to entities mapping
* - Fix LineageResourceTest failing due to changes on Lineage Storage
* - Update Lineage Details
---------
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
* Allowing to trigger an app with a payload
* added option to pass custom payload to triggered job
* feat(apps): add custom config to trigger
- added custom config for app triggers
* added handling for configuration validation on apps
* - removed `oneOf` from app configuration scheme
- generated ts files for apps
* added CONFIG_OVERRIDE_KEY
* format
* fixed app config key
* format
* format
* fixed python app runner to work with new schema
---------
Co-authored-by: sushi30 <imri.paran@gmail.com>
* Add the Platform Insights for the service details page
* Add new localization translations
* Fix the styling
* Add Data Quality widget in the service details page insights tab
* Remove the unused localization keys
* Fix the border radius
* Update the border color
* Add Total Data Assets widget
* localization changes
* Fix the Tier insights chart
* Add Tier and PII Distribution charts
* Localization changes
* Fix the calls for special fqn
* Limit the DataQualityWidget only to the DB services
* Add the MostUsedAssetsWidget
* Add missing loaders to the widgets
* Localization changes
* Update the API call for most used assets
* Add most expensive queries widget
* Localization changes
* Move the widgets logic to classBase for better customization
* Fix the font size
* add charts backend
* fix initialisation
* fix migration
* pii chart
* Add assets_with_tier DI system chart
* Fix the service insights charts
* localization changes
* spotless
* filter to be regex instead of literal compare
* Fix stylings for Most Used Assets widget
* Fix Unit tests
* Remove the New widgets
* Delete unused utils
* Fix the ServiceEntity tests
* Move SystemChartType to enum file
* Worked on comments
* Refactor ServiceInsightsTab and TotalDataAssetsWidget for improved code organization
* Refactor PlatformInsightsWidget and improve type safety
- Added PlatformInsightsWidgetProps interface for better type definition
- Updated PlatformInsightsWidget component to use the new interface
- Simplified component props typing
- Minor code cleanup in TotalDataAssetsWidget
- Updated Fqn utility method with a more precise regex replacement
* Refactor ServiceInsightsTab to use serviceDetails prop and remove Fqn utility method
- Updated ServiceInsightsTab to accept serviceDetails prop instead of using useFqn hook
- Removed Fqn.getNameWithoutQuotes method
- Updated PlatformInsightsWidget and TotalDataAssetsWidget to use serviceName prop
- Added ServiceInsightWidgetCommonProps interface for shared prop typing
- Simplified service name handling across components
* Refactor ServiceInsightsTabUtils to use local i18n import
- Updated import for internationalization to use local i18next utility
- Removed direct import of 't' from 'i18next'
- Imported i18n from local utility and destructured 't' method
* Improve type safety in TotalDataAssetsWidget component
- Added Readonly type to ServiceInsightWidgetCommonProps prop interface
- Enhanced type safety for component props
* add postgres migrations
---------
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
