Mayur Singal
a3ceea6b63
Hive Source Fix ( #3919 )
...
* Hive Source Fix
* Added auth option test & unified unit test
2022-04-08 22:29:48 +05:30
Milan Bariya
4d15d0f91e
ISSUE-3811: Add unit test for clickhouse ( #3949 )
2022-04-08 19:26:02 +05:30
dvecerdea
230fc256d6
Refactor Trino Connector and allow proxy setup in trino connection config ( #3810 )
...
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2022-04-07 12:50:37 -07:00
Ayush Shah
7c3a6082a6
Refactor - Redshift Connection ( #3920 )
2022-04-07 08:52:38 -07:00
Mayur Singal
3651efd7f5
Databricks Source Fix ( #3917 )
...
Databricks Source Fix (#3917 )
2022-04-07 16:40:50 +02:00
Pere Miquel Brull
93525aea0d
Fix #3823 & Fix generate default Enum field ( #3909 )
...
* Fix sample data DAG
* Fix callback imports
* Use --set-default-enum-member in generate
* Format
* Add faulty merge hard_delete
* Fix airflow lineage, improve naming and fix lineage tests
* Add mysql url test
* Add mysql url test
* Update CI name
* Fix test ometa endpoint
* Format
* Fix metadata config
2022-04-07 18:22:50 +05:30
Pere Miquel Brull
bd4071bd64
Fix #3826 & #3886 - Profiler workflow & filter pattern ( #3893 )
...
Fix #3826 & #3886 - Profiler workflow & filter pattern (#3893 )
2022-04-06 17:05:00 +02:00
Pere Miquel Brull
63533eb388
Fix for connectors based on refactoring of schemas V2 ( #3870 )
...
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-04-05 18:33:25 -07:00
Pere Miquel Brull
b3480693e4
Fix #3824 - OMeta Schema and JSON Connections ( #3861 )
...
Fix #3824 - OMeta Schema and JSON Connections (#3861 )
2022-04-05 21:20:39 +02:00
Pere Miquel Brull
e2539c5e83
Fix #3844 - First iteration for deprecating MetadataServerConfig ( #3853 )
...
* Style
* deprecate MetadataServerConfig
* Remove audience from Okta
2022-04-05 18:02:45 +05:30
Onkar Ravgan
01b246f586
ISSUE-3485: Added DynamoDB Unit Test ( #3488 )
...
* ISSUE-3485: Added DynamoDB Unit Test
* Added more tests and optimized code
* Re-added list check condition
* Added file sink test
* Fixed file path
* added assert for table and column names
* Modified exception block and optimized test cases
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-03-31 20:38:17 +05:30
codingwithabhi
2006b2b8ac
bigquery-unittest-added ( #3479 )
...
* bigquery-unittest-added
* mock-added-in-create_bigquery_client
* complex-struct-added
* makefile-updated-to-install-all-packages-for-unittest
* bigquery-unittest-updated-via-file-sink
2022-03-31 14:27:51 +05:30
Sriharsha Chintalapani
f79b251509
Fix #3771 : Main branch schemas and ingestion is failing; Fix #2992 : Remove ChangeEvents log from ElasticSearch Indexes ( #3772 )
2022-03-30 18:26:57 -07:00
Pere Miquel Brull
c22381fdc8
Fix #3747 - FQDN model in ometa api ( #3748 )
...
Fix #3747 - FQDN model in ometa api (#3748 )
2022-03-30 08:54:47 +02:00
Pere Miquel Brull
b3087d08b9
Fix #3522 - Add timeout to profiler ( #3707 )
...
Fix #3522 - Add timeout to profiler (#3707 )
2022-03-30 08:54:27 +02:00
Pere Miquel Brull
eb906589fd
Fix #3525 - Profiler breaks on Postgres data ( #3583 )
...
Fix #3525 - Profiler breaks on Postgres data (#3583 )
2022-03-22 15:55:44 +01:00
Pere Miquel Brull
16e82d45de
Fix #3371 - Run Profiler and Tests on a % of the data ( #3424 )
...
Fix #3371 - Run Profiler and Tests on a % of the data (#3424 )
2022-03-16 06:05:59 +01:00
Pere Miquel Brull
130bbb0c5c
Fix #3104 - Remove unused imports with pycln ( #3370 )
...
Fix #3104 - Remove unused imports with pycln (#3370 )
2022-03-14 06:59:15 +01:00
Pere Miquel Brull
94d7500216
Fix #3248 & #3251 - Update metrics and column profile ( #3262 )
...
Fix #3248 & #3251 - Update metrics and column profile (#3262 )
2022-03-08 11:44:39 +01:00
Pere Miquel Brull
434aa30a92
Fix #3221 - columnValuesToBeUnique vs. columnValuesToBeNotNull JSONs ( #3255 )
...
Fix #3221 - columnValuesToBeUnique vs. columnValuesToBeNotNull JSONs (#3255 )
2022-03-08 09:15:20 +01:00
Pere Miquel Brull
954908efec
Fix #3217 - Test case props should be unique ( #3220 )
...
Fix #3217 - Test case props should be unique (#3220 )
2022-03-07 14:43:42 +01:00
Pere Miquel Brull
2444b884bf
Fix #3150 - Missing count & value length tests ( #3193 )
...
Fix #3150 - Missing count & value length tests (#3193 )
2022-03-07 07:19:13 +01:00
Pere Miquel Brull
4a752e3ab2
Fix #3151 - Ingestion profiler should use ORM Profiler ( #3192 )
2022-03-06 15:43:43 -08:00
Pere Miquel Brull
4116233697
Fix #3105 - ColumnValuesToMatchRegex & other fixes ( #3149 )
...
Fix #3105 - ColumnValuesToMatchRegex & other fixes (#3149 )
2022-03-04 18:11:49 +01:00
Pere Miquel Brull
bd7b91b448
Fix #3112 - col profile safety & sample data ( #3142 )
...
Fix #3112 - col profile safety & sample data (#3142 )
2022-03-04 13:14:11 +01:00
Pere Miquel Brull
e96ac838ff
Fix #3084 - Implement missing tests ( #3117 )
...
Fix #3084 - Implement missing tests
2022-03-04 06:59:47 +01:00
Pere Miquel Brull
71207de362
Fix #2875 - Profiler API Sink ( #3011 )
...
Fix #2875 - Profiler API Sink
2022-03-02 16:46:28 +01:00
Alberto Miorin
fe5618c8f1
Fix #3037 : metadata --version doesn't work ( #3038 )
2022-03-01 12:19:36 +01:00
Pere Miquel Brull
990608522a
Fix #2981 - Update Profile to match TableProfile ( #2982 )
2022-02-25 09:26:30 -08:00
Pere Miquel Brull
a4b383fa83
Fix #2897 - Profiler CLI ( #2967 )
...
Fix #2897 - Profiler CLI
2022-02-24 08:03:50 +01:00
Pere Miquel Brull
1fb0e7c489
Fix #2878 & #2877 - Implement Metrics and Validate Composed Metrics ( #2926 )
...
Fix #2878 & #2877 - Implement Metrics and Validate Composed Metrics
2022-02-24 07:08:39 +01:00
Pere Miquel Brull
1224d20a36
Fix #2894 - Profiler Processor & Metrics ( #2900 )
...
Fix #2894 - Profiler Processor & Metrics (#2900 )
2022-02-22 08:09:02 +01:00
Pere Miquel Brull
e55579aaa8
Fix #2845 - Init Profiler Workflow ( #2862 )
...
* Fix list typing
* Add sqlite service
* Add sqlite service
* Add sqlite service
* Refactor validation into class
* Refactor validation into class
* Prepare table simple profiler
* Add note
* test ORM conversion
* Prepare workflow config utilities
* Prepare workflow skeleton
* Use new core Validation
* Refactor workflow config parsing
* Add comment
* Simplify workflow validations
* Fix table metric check
* Add init for convenience, otherwise interpreter cries when trying to __call__ the get result
* Fix table metric check
* Format
* Format
* Fix table list and metrics init
* Prepare profiler workflow integration tests
* Bump version
* Fix pycharm imports
* format
2022-02-20 17:55:12 +01:00
Pere Miquel Brull
f304d290b4
Fix #2751 - Init ORM Profiler ( #2831 )
...
* ORM Profiler skeleton
* Fix table name within service
* Add license
* Prepare custom types
* Fix converter
* Compute stddev only on numeric
* Prepare smart registries
* Update tests
* Update results retrieval
* Fix composed metrics result
* Format
* Add missing type
* Add _label decorator
* clean readme
* clean readme
* Filter types when profiler runs not allowed metric types
* Fix null ratio
* Add proper type
* RuleMetric skeleton
* Prepare table metrics
* Update simple profiler
* Format
* Define test expression grammar and node visiting
* Unify metric registry
* Prepare validation core
* Add grammar lib
* Add safe get
* Format
* Allow decimals in grammar
* Test validation conversion
* Fix validation conversion and test
* Rename to row_number
* Update READMEs
* Format
* Row number naming
* Fix rename
2022-02-18 07:48:38 +01:00
Sriharsha Chintalapani
50799227e4
Fix #2817 : Remove indexing of Glossary as another Elasticsearch index ( #2818 )
2022-02-16 20:10:26 -08:00
Suresh Srinivas
28ba1a3c04
Fixes #2760 - Add entities for Glossary and initial API ( #2761 )
...
* Fixes #2760 - Add entities for Glossary and initial API
* Fixing merge issues
2022-02-15 20:54:46 -08:00
Teddy
b5ddada930
Fix #1526 : Added OMetaTag mixin
2022-02-06 12:42:13 -08:00
Mayur Singal
5a6e7d743e
ISSUE-545: Added tests for QueryProcessor ( #2510 )
...
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-04 08:20:20 -08:00
Robert Pack
67a1c23482
Fix #2348 : Harmonize request titles in json schemas
2022-01-31 16:29:56 -08:00
Sriharsha Chintalapani
f390bac375
Fix 2270: Column Type Parser ( #2271 )
...
* Fix 2270: Column Type Parser
* Fix 2270: Column Type Parser
* Added checks to allow arrayDataType and dataTypeDisplay
* Modified - arrayDataType and dataTypeDisplay
* Update sql_source.py
* Update sql_source.py
* file formatted
* Modified according to column_type_parser.py
* modified / refactored / deleted the files
* Added Tests, modified sql_source
* file formatted
* Added missing datatypes
* Added Tests
* Added Tests
* Added Tests - refactored expected output into a json file
* file formatted
* Sample Data Updated
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-22 23:32:58 +05:30
codingwithabhi
ddf4ae261b
Bigquery credential configuration updated ( #2041 )
...
* bigquery-credentials-in-bigquery-json
* ui-changes-reverted
* credentials-moved-to-options
* Update bigquery.json
* Update bigquery.py
* delete-temp-code-added
* delete-temp-code-added
* removed-init-from-config
* Update datatypes_test.py
* Update column_helpers.py
* code-formatted
Co-authored-by: Abhishek <abhishek@Abhisheks-MacBook-Air.local>
Co-authored-by: = <=>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-21 17:23:02 +05:30
Pere Miquel Brull
d3b6c7cf27
[issue-1973] - Python API from Sklearn to MlModel ( #2119 )
...
* Move staticmethods to utils
* Use functions from utils
* Convert sklearn to MlModel
* merge main
2022-01-10 09:36:08 +01:00
Pere Miquel Brull
48ebcffbd0
[issue-1976] - Ingestion SonarCloud ( #2085 )
...
* Update path
* Prepare sonar properties
* Prepare coverage recipes
* Add coverage
* Simplify pytest
* Organise integration tests
* Update path
* Use setup instead of reqs
* Update recipes
* Fix PR event to target
* Update event_name
* Prepare sonar
* Run tests & sonar
* Use sonarcloud host
* Fix compose
* Use ingestion token
2022-01-07 10:28:38 +01:00
Alberto Miorin
c8de136cca
First part of #2091 : Python code in branch main is always formatted correctly ( #2093 )
2022-01-07 06:17:48 +01:00
Ayush Shah
5d6f385a75
Added Filter Params for Table and Schema ( #1954 )
...
* Added Filter Params for table and Schema
* Bigquery Doc changes
* Doc Changes for databases
* Filter Pattern Changes
* Table Filter Pattern Example Changes
* Filter Pattern Example Changes
2021-12-29 09:13:09 -08:00
Pere Miquel Brull
1f01081698
Clean DBT Model ( #1888 )
2021-12-22 17:17:05 +01:00
Pere Miquel Brull
ab620e95af
[issue-1750] - Expandvars transforming values ( #1830 )
...
* Prepare tests for basic config cases
* Add test cases
* Use os expandvars
* Add missing commas
* Add missing commas
* Remove dataclasses backport
2021-12-18 15:35:12 -08:00
Alberto Miorin
93276079ec
Fix #1729 : Connector Trino works with username. ( #1732 )
...
Implementation details
I have decided to rename schema_name to database and make it mandatory. Without database there is an error while scanning all available tables. The connector doesn't support multiple databases at the moment. It has to be tested with passwords. Trino requires SSL if you use passwords. It has to be tested with impersonation. I have removed quote_plus because I don't think it's needed.
- [x] Support username
- [ ] There is an integration test
- [ ] Support impersonation
- [ ] Support passwords
- [ ] Support tokens
- [ ] Support multiple databases
2021-12-13 22:05:28 +01:00
Ayush Shah
b592f86941
Added Unit tests for datatypes ( #1405 )
...
* Added Unittests for datatypes
* OpenMetadata Datatypes added in unittest
* Added few more datatype support
* SqlTypes updated
2021-12-08 00:31:51 +05:30
Alberto Miorin
8621ee3fa0
Fix #1533 , #1522 : ( #1575 )
...
- Glue works with columns containing dots.
- Glue works with temporary credentials.
- Fix bug in TableRepository, the EntityType was database instead of table.
- only region_name is mandatory in Glue config.
- first integration test for Glue. The CI integration with AWS is still missing.
2021-12-06 16:03:05 -08:00
