* fix(data-quality): table diff
- added handling for case-insensitive columns
- added handling for different numeric types (int/float/Decimal)
- added handling of boolean test case parameters
* add migrations for table diff
* add migrations for table diff
* removed cross type diff for now. it appears to be flaky
* fixed migrations
* use casefold() instead of lower()
* - implemented utils.get_test_case_param_value
- fixed params for case sensitive column
* handle bool test case parameters
* format
* testing
* format
* list -> List
* list -> List
* - change caseSensitiveColumns default to fase
- added migration to stay backward compatible
* - removed migration files
- updated logging message for table diff migration
* changed bool test case parameters default to always be false
* format
* docs: data diff
- added the caseSensitiveColumns parameter
requires: https://github.com/open-metadata/OpenMetadata/pull/18115
* fixed test_get_bool_test_case_param
* enable logging for debugging
* remove node-gyp
* add node-gyp globally before installing deps
* reduce quick type to 10
* revert quicktype and node-gyp changes for CI
* fix unit tests
* Fix backend tests and have index alias with clusterAlias appended
* Fix backend tests and have index alias with clusterAlias appended
* Fix failing tests
* alias setup
* fix suggestion not working due to alias
* fix getIndexOrAliasName method for multiple indexes
* update openmetadata.yaml
* update childAliases with clusterAlias
---------
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
* Revert "fix(CI): update `run_local_docker.sh` script with cluster alias ES endpoint (#16604)"
This reverts commit baab52076f71f76e8bc48d89564ba733ac77a1e6.
* Revert "chore(CI): Enable Elasticsearch cluster alias by default (#16599)"
This reverts commit a3524e14d3a1244db19199ab1eae3796ba15b499.
* WIP - MINOR - Clean ingestion dependencies
* test
* test
* Clean imports
* add pyiceberg for test
* Revert "add pyiceberg for test"
This reverts commit ab26942736586f089a57a644ffd727aca200db62.
* add pyiceberg for test
* Remove docker dep
* clean local docker sh
* MINOR - AKS Airflow troubleshooting docs
* Fix action
* clean local docker sh
* feat: use native backup tools
1. added mysqldump 8.3 to the ingestion container.
2. documented how to use native tools to back up and restore.
3. added deprecated message on the cli backup and restore.
* added deprecation notice for 1.3 backup
* removed 1.3.x deprecation notice
* added another backup page in 1.3 introducing SQL dump tools
* added --set-gtid-purged=OFF to the mysql dump process
* feat: SAS Viya connector
* refactor SASCatalog to SAS
* add SAS logo to UI and connection documentation
* doc changes
* modify ingestion logic
* revert original changes
* added support for dataflow & perfect logic for reports/datatables
* add filter doc
* more updates to perfect ingestion for each asset type
* fix a bug with table lineage not created properly
* Delete ingestion/pipelines/sasCatalog.yaml
* precomit fix
* Conversion to database connector
* minor fixes
* make custom properties type generic
* Add SAS javaEnum
* add dummy variable for sas.yaml
---------
Co-authored-by: lizmc <liz.mcintosh@sas.com>
Co-authored-by: Shiyang Xiao <Shiyang.Xiao@sas.com>
* Fix Alert Intermittent Issues for some entities
* Fix Failing test for disabled publisher
* Fix Failing test due to deleted entity
* Fix Tests
---------
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
* Added diffrent clients for elasticSearch and openSearch and did the interface implementation
* removed some comments and some changes
* Add Elastic Search Test Container
* upgarded es version
* added test cases for elastic search
* added test case for, when tag is deleted cehck its relationships also get deleted
* changed package of open,elastic to openSearch and elasticSearch
* added support for skip search test case if index is not present of that entity
* Refactor bunch of files
* unnecessary semicolon
* refactor test case indexing
* fix checkstyle
* checkstyle
* added hardcoded configs and changed package name
* fixed no response from hits
* stopped elasticsearch container and some fixes
* closed elasticsearch container
* revert
* closed client after used in test
* style
* edit es config in junit
* checking if using already es version release memory for maven ci
* revert docker file changes
* remove es container for testing the ci
* commented out all test cases for elasticsearch
* fix
* creating elastic search with reusable enabled
* trying to fix ci error
* start container only if it is not running
* revert docker changes and done changes for elastic search container
* changed var name of sort_order for es
* lower the version of es client
* fixes
* all latest version of es including docker
* Use Single sort order
* Fix Keyword issue
* Immediate refresh policy for entity created
* increase sleep time
* disable test
* added reindexing logic in run_local_docker.sh
* enabled test case
---------
Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
* Adding the different docker-compose file openmetadata and ingestion
* Added two different env files for mysql and postgres
* Updated the docker file path
* Updated the path of docker folder structure
* Fix docker
* Updating the PR with necessary changes required
---------
Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
* Mask sensitive info from API responses
* Rename converter classes
* Add missing Java classes from JSON schemas and class converters
* Update test service connection schema
* Update datalakeConnection JSON schema and fix some tests
* Fix AlertsRuleEvaluatorResourceTest and minor error in run_local_docker.sh
* Fix Pipeline and Database service tests
* Minor refactor
* Fix CsvUtilTest
* Fix EventMonitorFactoryTest
* Fix CloudWatchEventMonitorTest
* Update datalake metadata
* Update bigquery metadata
* Fix test connection functionality
* Fix OMeta service api test
* Update gcsValues title and revert changes in GH actions
* Mask sensitive enabled by default for local docker
* Add missing tests
* Address PR comments
* Address PR comments
* fix ui breaks on gcsValues.json
* Address PR comments
* Minor refactor
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
* Added the default volumes for ElasticSearch
* Added docs for ES Volumes
* Updated as per the comments
* Updated as per the comments
* Added the docs for bare-metal upgrade
* Apply suggestions from code review
* Update docker/local-metadata/docker-compose.yml
* Pylint check addressed
Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: Nahuel <nahuel@getcollate.io>
* Added host volumes for Database service
* Formated the python file
* Done changes according to the comments on PR
* Changes done as per the comments on Draft PR
* Fix PyLint
* Changes done as per the comments on Draft PR
* Updated the path of volume for Database
* Added cleanup state in workflows
* Added cleanup state in workflows
Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* Removed the volume for Database
* Removed the volume for Database
* Added named volume
* Added named volume
Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
* Added default volumes
* Ran make py_format for formatting
* Added sample data in mount directories
* Updated as per the comments
* Done changes for --ingest_sample_data flag
Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
* Fix sample data DAG
* Fix callback imports
* Use --set-default-enum-member in generate
* Format
* Add faulty merge hard_delete
* Fix airflow lineage, improve naming and fix lineage tests
* Add mysql url test
* Add mysql url test
* Update CI name
* Fix test ometa endpoint
* Format
* Fix metadata config
* Issue-3685: Variable based separator used for fullyQualifiedName instead of hardcoded . for Python and make the sepearator to :
* Fix failing test
* Use colon for run_local_docker validation
* Update tests FQDN
* Update tests FQDN
Co-authored-by: Sachin-chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
* Simplify sample data DAG
* Remove mkdir
* Generate sources before running compose
* Generate sources to install models to ingestion image
* Add python-on-whales for docker --start
* Remove python-on-whales from base
* Install venv
* Setup python
* Fix typo
* Clean setup
* Update ingestion local image to be barebone on connector dependencies
* Prepare ingestion connectors base image
* Add system dependencies
* Prepare docker CLI
* Add docker provider
* Prepare entrypoint for the image
* Remove DBT pipeline as per Issue 1658
* Add TODO for ingestion build
* Bind docker socket
* Update comment
* Update README
* Use DockerOperator in sample data
* Build images with latest tag
* Prepare symlink to pass the volume to the DockerOperator
* Update README
* Prepare Base image for CI
* COPY multiple files into dir
* COPY multiple files into dir
* Remove DBT source as is now part of table ingestion
* Build docker base in run_local_docker
* Force image build in local docker
* Use service as default field query param for entities with required service
* Return service in List
* Naming convention
* Update Repository setFields to include service
* Revert resource changes
* Revert resource changes
* Update tests to check we always return the service, even if not specified
* Return service for database
* added docker file for local build and ingestion
* Docker Modified for local setup
* Ingestion script optimized
* Added script to directly run Local Docker
Co-authored-by: Parth Panchal <parthpanchal@Parths-MacBook-Air.local>
