* Reverted the ES config step for Data Insights pipeline
* es sink fixes for ingestion
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
* Add fine-grained health validation for Airflow APIs
* Add ingestion version response
* Improve messaging
* Format
* Format
* Update response
* ui:update the hook to return airflow status based on response status field
* ui: add unit test for useAirflowStatus hook
* chore: only show test connection if airflow is available
* feat: add airflow message banner
* chore: update icon and background color
* chore: update typography to text
* test: add unit test
* address comments
* chore: show banner on service detail page
* fix: update test suite api workflow to reflect new implementation
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
* Deprecate location and old storage service
* Format
* Fix test
* Refactor
* Clean location
* Rename object store to storage
* Rename object store to storage
* Rename object store to storage
* Format
* Format
* Refactor object store for storage
* Refactor object store for storage
* Rename object store to storage
* Fix test
* Fix test
* Format
* chore(ui): change Objectstore to Storage
* Fixes
* Fix test
* Remove storage service from Glue cypress
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
* Prepare the new test connection ingestion logic
* Update test assert
* Update Test Connection for SQA Sources
* Correct return type and method doc
* Handle decryption
* Non SQA Database Sources
* Add the run_automation script in ingestion-base
* Dashboard Test Connection Changes
* Pipeline, Messagin, MlModel & Metadata Sources
* ui: test connect flow-1
* Unmask connection parameters before sending to Ariflow
* ui: test connect flow-2
* Address review comments and pylint
* pytest fix
* ui: test connect flow-3 (refactoring and style fix)
* ui: test connect flow-4 (fix test connection status logic)
* sync local file
* ui: test connect flow-5 (fix lowercase issue and styling)
* ui: test connect flow-5 (show toast notifications)
* test: add unit test
* ui: test connect flow-5 (update service page test connection button)
* Databrick fix & pytest fix
* pylint
* Update test
* Fix merge
* S3 Test connection
* add style for mandatory step
* sync locales
* chore: add service name in workflow request
* Unmask using original service connection parameters
* Fix test connection unmasking
* Wrap inspector function to eliminate error outside test conn
* Fix linting
* fix:cy test
* Fix linting
* address comment
* refactor and fix connection type casing issue
---------
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* feat(profiler): renamed module to
* feat(profiler): added dbt-artifacts-parser to test setup.py
* feat(profiler): refactor workflow and interface
* feat(profiler): linting
* feat(profiler): removed old profiler modules
* feat(profiler): added support for value and integer range partition
* feat(profiler): fixed linting
* feat(profiler): added partitionning support for datalake profiler
* feat(profiler): removed `ProfilerInterfaceArgs` class
* feat(profiler): address comments
* feat(profiler): Added `OTHER` as an `IntervalType` for UI type generation
* Update airflow loggers and rename ometa loggers
* ANSI print to logger
* Remove colored logging from tests
* Merge ometa_logger into the one used in loggers class
* linting
* linting
Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
* Added support for 'Metadata Service' connectors
* Fixed failing unit tests
* Removed delete button for 'Metadata Service' connector of type OpenMetadataServer
* Code optimization for SelectServiceType component
* Fixed errors arrived due to backend changes related to OpenMetadata Services
* init reindex
* Added additional step for adding Metadata to ES configs while creating ingestion pipeline for Metadata Service
* Add ES reindex
* Add ES reindex
* Format
* Format
* Rename service
* Pick service name from IngestionPipeline
* fix ui conflicts
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
* Removed part of the secrets manager implementation in the server side
* Removed part of the secrets manager implementation in the openmetadata python library side
* Remove deprecated test
* Address pylint checks
* Address new pylint checks
* Address PR comments
* Fix import on airflows apis
* Clear singleton instances for running Ometa secrets manager test
* Fix#6141: Ingestion Pipeline Status Updates
* List Pipeline Status API & Improvements
* Rename State field to PipelineState in UI
* Convert Pipeline Status array to single object
* fix braking UI
* Rebase Fixes
* Profiler, TestStuite & DataInsigts Pipeline
* py_format
* fix logs page not loading
add pipelineStatus endpoint
* fix recent run changes
* Fix Tests
* address review comments for ui
* fix failing checks
* fix unit tests
* fix cypress test
* remove loader test as not using it
* wait for API when we click on the tab
* fix cypress waitFor api
* fix failed cypress tests
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
* added data insight module to refine and ingest data
* Added json schema for data insight workflow
* Added mixin for data insight
* Added tests for data insight workflow and methods
* Fixed python style
* Fixed python linting
* Fixed unit test
* Added data insight worflow + workflow logging
* Fix python style
* Renamed analytic -> data_insight + added test to make + fixed style
* Fixed java style
* Fixed code smells + python style
* Ignore ES linting for now
* Fixed comments from review
* fix python formatting
* Fixe PipelineType attribute for data insight
* initial setup for data quality form
* added stepper and form component
* added select/add test suite step form
* added form for table test and cron step
* added data quality table test form flow
* bug fix for profiler
* added column test form
* render right panel information dynamically
* updated test as per new changes
* updated data test id
* Fixed ingestionPipeline
* Fixed pytest + python format
* miner fix, added sql editor
* removed filter for duplicate test check
* miner fix
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
* Added database filter in workflow
* Removed association between profiler and data quality
* fixed tests with removed association
* Fixed sonar code smells and bugs
* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample
* moved status to workflow
* Fixed tests
* removed test logic from profiler sink
* Added logic to return sample from workflow sample value
* Added profiler examples
* Updated documentation for profiler
* Fixed code smells
* commited changed to profiler
* initial commit of the revamp workflow
* Fixed python formating
* cleaned up profiler submodule by removing test related files and functions
* Added airflow DAG logic for testSuite workflow
* Fixed code smells + added airflow ingestion tests + fixed comments
* Implementation of AWS SSM as secrets manager
* Remove dead code
* Minor fixes
* Allow using default credentials in AWS client
* Fixed py style
* Fixed tests imports
* Minor changes
* Add patch to failing test
* Fix how we were storing dbtConfigSource
* Address PR comments
