Teddy
989f2911c2
Fixes #7810 - Allow to only pass min or max ( #8474 )
...
* ISSUE-7810 Added default values for min and max
For all data validations on columns:-
min_bound is set to float("-inf"), if there is no next value
max_bound is set to float("inf"), if there is no next value
* Fixed PR errors by removing tuple + added tests
Co-authored-by: demi <deepak1212365@gmail.com>
2022-11-01 13:26:51 +01:00
Ayush Shah
2d7d89754c
Remove Click and Add ArgParse ( #8182 )
2022-10-31 18:12:26 +05:30
Pere Miquel Brull
6d404ccdc1
Domo Pipeline - Use pipeline ID with display name ( #8374 )
...
* Use pipeline ID with display name
* handle id
* lint
* update test for ID
2022-10-26 21:22:26 +02:00
Teddy
83b048c07c
Fixes #8206 -- Implement data insight refinement for entity report data ( #8351 )
...
* added data insight module to refine and ingest data
* Added json schema for data insight workflow
* Added mixin for data insight
* Added tests for data insight workflow and methods
* Fixed python style
* Fixed python linting
* Fixed unit test
* Added data insight worflow + workflow logging
* Fix python style
* Renamed analytic -> data_insight + added test to make + fixed style
* Fixed java style
* Fixed code smells + python style
* Ignore ES linting for now
* Fixed comments from review
* fix python formatting
* Fixe PipelineType attribute for data insight
2022-10-26 11:18:08 +02:00
Nahuel
1583adee29
Fix lineage to support 'merge into' queries starting with comments ( #8355 )
2022-10-25 10:29:59 -07:00
Nahuel
36b12bd6f1
Fix lineage issues with merge_into and copy grants queries ( #8335 )
2022-10-24 17:22:22 +02:00
Suresh Srinivas
229b56e7b3
Fixes #8236 - Initialize bots from JSON data files ( #8304 )
2022-10-21 08:38:52 -07:00
Milan Bariya
1565aa7733
FIX: Trino&Presto catalogs as databases ( #8189 )
...
* FIX: Trino&Presto catalogs as databases
* FIX: Trino&Presto catalogs as databases
* Change based on comments
2022-10-18 20:00:17 +05:30
NiharDoshi99
84a89d7181
Added DOMO Pipeline ( #8133 )
2022-10-18 17:52:28 +05:30
Milan Bariya
5b3c877dd1
FIX:connection failed because of having '@' character in uname and pass ( #8172 )
2022-10-17 16:57:55 +05:30
Pere Miquel Brull
d48fd468d4
Tag PATCH + Cleanup of helpers methods ( #8150 )
...
* cleanup
* lint
* Add tag patch
* Fix rename
* Dont kill tests
2022-10-15 14:56:30 +02:00
Milan Bariya
e7a3770240
Add: databricks usage and lineage added ( #7386 )
2022-10-14 07:06:28 +05:30
NiharDoshi99
0e44f7d35d
Fixed: Intergrated DOMO Connector APIs ( #7989 )
2022-10-13 10:55:48 +05:30
Teddy
f883863b8a
Fixes #7490 - Split Profiler and TestSuite Interface ( #8032 )
...
* Clean up test suite workflow and interface
* Fixed tests
* Split profiler and testSuite interfaces
* Cleaned up workflows and runners
* Fixed code formatting
* - remove old code
- remove `table` attribute used for testing and used mock instead
* Fixed execution bugs from refactor
* Fixed static type checking for profiler/api/workflow.py
* Fixed linting
* Added __init__ files
2022-10-11 15:57:25 +02:00
Ashish Gupta
8ddab79dc1
UI :- Remove rc-tree and react-table dependencies ( #8049 )
...
* Remove rc-tree and react-table dependencies
* fix unit test issue
* File Formatted
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-10-11 15:42:58 +05:30
Teddy
3b7f576d04
Fixed linting for Profiler ( #7922 )
...
* - Fixed linting
- Added logic to skip partition check when engine is not BQ
- Added ingestion partition logic to testSuite
* Fixed python formating
* Fixed test for BQ partition
2022-10-11 09:36:36 +02:00
Pere Miquel Brull
f031293af0
Fix Pylint ingestion/bulksink and ingestion/stage ( #8030 )
...
* Pylint ingestion bulksink
* Pylint stage
* Remove test
* Remove stage
2022-10-10 14:06:43 +02:00
Nahuel
5c499d2a7a
Add CLI E2E tests for MySQL ( #8041 )
...
* Add CLI E2E tests for MySQL
* Fix setup.py and pylint
* Add missing doc and update code after pylint refactor
2022-10-10 11:36:20 +02:00
Teddy
e244a241c3
added logic to clean up quotes from table constraints ( #7959 )
2022-10-05 16:09:33 +02:00
Milan Bariya
cd43f1fdcf
Fix: connector for hive has double dash ( #7953 )
2022-10-05 12:01:13 +02:00
Teddy
15f7c4aa41
Fix param name for median test ( #7942 )
...
* Fixed param name for median test
* Fixed unite test for median DQ
2022-10-05 06:32:28 +02:00
Teddy
63b5233fd3
Added util command to overwrite dag_generated_config path in DAG file ( #7831 )
...
* Added util command to overwrite dag_generated_config path in DAG file
* Added logic to only write file on change
2022-10-03 09:24:47 +02:00
Nahuel
593ca3a4a0
Fix bot creation and edition logic ( #7796 )
...
* Fix bot creation and edition logic
* Fix minor error creating user
* Fix failing test
* Minor fixes
* Add missing tests for new flows
* Fix put_failIfBotHasARelationshipToAnotherUser test
* Changes after manual testing
* Move where auth_provider is retrieved in the secret manager
2022-10-01 11:25:56 +02:00
Teddy
962866a30e
Fixes #7280 - Implements support for BYTEA DType ( #7749 )
...
* Added support for postgres bytea data type
* Added BYTEA support for profiler and converts bytes to hex on results read
2022-09-28 13:28:35 +02:00
Nahuel
df909703eb
Fix#6615: Fix failing test still in CI ( #7725 )
...
* Fix failing test still some tests are failing
* Fix bug in service entity repository
* Fix all tests except Ingestion Pipeline integration tests
* Fix Ingestion Pipeline integration tests
* Fix failing Python tests
2022-09-26 20:39:42 +02:00
Ayush Shah
56c71c4101
Fix Ingestion Time Partitioned Tables on Profiler ( #7506 )
2022-09-26 19:41:40 +05:30
Onkar Ravgan
35efd49256
Added control for DBT descriptions ( #7653 )
...
* Added control for DBT descriptions
* Fixed tests
* Added UI changes
* fixed maven ci tests
* Java formatting changes
* ui review fixes
* Fixed pytests
* Fixed python integration tests
* fixed airflow tests
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-09-26 16:19:47 +05:30
NiharDoshi99
8adcf2ee3d
Fixed: Dagster to work with its API ( #7579 )
2022-09-23 17:46:40 +05:30
Teddy
f2bf5194bb
Fixes #7623 -- Added logic to encode and decode entityLink ( #7670 )
...
* Encode entityLink string when processing request
* Added logic to decode column type from entityLink
* mvn code formating
* Extracted unquote step into its own function
2022-09-23 09:42:33 +02:00
Teddy
c34281251f
Fixes #7509 - implement import path migration script ( #7515 )
...
* Added script to migarte import
* Added tests
* Updated doc to include info about import migration
* renamed test resource files
2022-09-22 14:43:25 +02:00
Ayush Shah
517e9a4aac
Remove unused imports ( #7657 )
2022-09-22 16:42:10 +05:30
Sriharsha Chintalapani
adaa8e7493
Fix #6917 : startTs & endTs filters providing paginated data ( #7617 )
...
* Fix #6917 : startTs & endTs filters providing paginated data
* Updated ui side changes
* Fixed OM tests end to end test
* Fix #6917 : startTs & endTs filters providing paginated data
* Fix #6917 : startTs & endTs filters providing paginated data
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-09-21 13:33:47 -07:00
Pere Miquel Brull
1578fa7f1b
Fix #7121 - Support Spark metastore DB connection ( #7520 )
...
* Fix #7121 - Support Spark metastore DB connection
* appname
* Update docs
* test validation
* Address PR comments
Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-09-20 16:47:57 +02:00
Nahuel
2a6c6134f4
Fix#7272: Improve logging when initializing workflow from CLI ( #7522 )
...
* Improve logging when initializing workflow from CLI
* Fix broken tests
2022-09-19 08:00:00 -07:00
Teddy
1ba6e284fe
Fixes #7118 by cleaning up test names ( #7494 )
...
* Cleaned up tests names and add registry name tests
* Updated documentation for test types supported by OM
2022-09-16 07:04:56 +02:00
Teddy
3fdbe6ecfb
Added logic to report column level metric computation failure for profiler ( #7465 )
2022-09-15 17:39:29 +02:00
NiharDoshi99
5eb45948bd
Fixed: PostGIS geometry type ( #7382 )
...
Fixed: PostGIS geometry type (#7382 )
2022-09-15 10:35:15 +02:00
Sriharsha Chintalapani
656b50dd3a
Fix #7469 : Refactor OpenMetadata code modules ( #7474 )
2022-09-14 23:14:02 -07:00
Teddy
71679137a0
Fixes #6009 - Adds SQA MetaData object instance per DB ( #7388 )
...
* Added SQA MetaData object instance per DB to allow same schema/table name across multiple db
* Update ingestion/src/metadata/orm_profiler/orm/converter.py
updated type hint
2022-09-12 18:05:36 +02:00
Teddy
0677b5258f
Fixes #7299 by updating GE submodule to new test API ( #7332 )
...
* Update GE logic to the new test API
* Cleaned up part of the code from
update to TestSuiteMixin
* Added + fixed tests
* Deleted unused files for GE tests
* fixed python tests
2022-09-11 20:07:19 -07:00
Nahuel
b829a2cbf3
Fix: Improvements on secret manager implementation ( #7282 )
...
* Change local secret manager by noop
* Update openmetadata-secure-test.yaml
2022-09-07 09:18:59 +02:00
Sriharsha Chintalapani
d29a230726
Fix #7104 : Add support for get resource permission by FQN ( #7270 )
2022-09-06 12:30:52 -07:00
Onkar Ravgan
9be3df4a1e
Integrated DBT tests ( #7122 )
...
* Integrated DBT tests
* Fixed py tests
* Review comments addressed
* Changed test suite name default value
* Added fixes
* update ui changes for new field
* send data along with api
* update name
* fixes for UI
* fixes for UI
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-09-05 22:25:31 +05:30
Teddy
9dbcb3911b
Fix minor column data quality test bugs ( #7111 )
...
* Fixed test name issue + filtered out partition details for non BQ tables
* Exclude non BQ table from partition processing
* Fixed test + formating
2022-09-01 13:47:00 +02:00
Teddy
ef41382cb1
Fixes #7094 by fixing minior bugs in table tests ( #7095 )
2022-08-31 21:35:33 +02:00
Teddy
a39c4db8e7
Add partial support for BQ partitioned table ( #7066 )
...
* Added support for BQ time based partition (not ingestion)
* Fixed minor errors in test suite workflow
2022-08-30 11:39:15 -07:00
Mayur Singal
791245c0e8
Redshift Mark Partitioned Tables ( #7063 )
2022-08-30 18:40:51 +02:00
Pere Miquel Brull
d2d3151cca
Simplify lineage logs ( #7062 )
2022-08-30 17:25:40 +02:00
Mayur Singal
5a3fcafb3b
Snowflake Mark Partitioned Tables ( #7021 )
...
Snowflake Mark Partitioned Tables (#7021 )
2022-08-30 14:35:09 +02:00
Nahuel
4c6f0c7bda
Fix#6027: Fix logging in metadata package modules except ingestion and orm_profiler ( #6970 )
...
* Fix logging in metadata package modules except ingestion and orm_profiler
* Reduce log level to _get_columns_with_constraints method
2022-08-29 06:46:06 +02:00
