yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-07-13 20:18:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
10,673 Commits 677 Branches 144 Tags
Commit Graph

984 Commits

Author SHA1 Message Date
Pere Miquel Brull
b3087d08b9
Fix #3522 - Add timeout to profiler (#3707) 
Fix #3522 - Add timeout to profiler (#3707)
 2022-03-30 08:54:27 +02:00
Snyk bot
3173646484
fix: ingestion/tests/integration/source/mlflow/Dockerfile to reduce vulnerabilities (#3618) 2022-03-23 18:26:41 -07:00
Pere Miquel Brull
eb906589fd
Fix #3525 - Profiler breaks on Postgres data (#3583) 
Fix #3525 - Profiler breaks on Postgres data (#3583)
 2022-03-22 15:55:44 +01:00
Pere Miquel Brull
548a0ab722
Fix #3437 & #3186 - Airflow lineage Task Group & Tests (#3460) 2022-03-21 10:29:49 -07:00
Ayush Shah
538ee87acc
Fix tests - added recursion delete and error handling (#3560) 2022-03-21 10:00:39 -07:00
Pere Miquel Brull
16e82d45de
Fix #3371 - Run Profiler and Tests on a % of the data (#3424) 
Fix #3371 - Run Profiler and Tests on a % of the data (#3424)
 2022-03-16 06:05:59 +01:00
Pere Miquel Brull
130bbb0c5c
Fix #3104 - Remove unused imports with pycln (#3370) 
Fix #3104 - Remove unused imports with pycln (#3370)
 2022-03-14 06:59:15 +01:00
Pere Miquel Brull
94d7500216
Fix #3248 & #3251 - Update metrics and column profile (#3262) 
Fix #3248 & #3251 - Update metrics and column profile (#3262)
 2022-03-08 11:44:39 +01:00
Pere Miquel Brull
434aa30a92
Fix #3221 - columnValuesToBeUnique vs. columnValuesToBeNotNull JSONs (#3255) 
Fix #3221 - columnValuesToBeUnique vs. columnValuesToBeNotNull JSONs (#3255)
 2022-03-08 09:15:20 +01:00
Pere Miquel Brull
954908efec
Fix #3217 - Test case props should be unique (#3220) 
Fix #3217 - Test case props should be unique (#3220)
 2022-03-07 14:43:42 +01:00
Pere Miquel Brull
2444b884bf
Fix #3150 - Missing count & value length tests (#3193) 
Fix #3150 - Missing count & value length tests (#3193)
 2022-03-07 07:19:13 +01:00
Pere Miquel Brull
4a752e3ab2
Fix #3151 - Ingestion profiler should use ORM Profiler (#3192) 2022-03-06 15:43:43 -08:00
Francisco J. Jurado Moreno
a6e98b3fdb
Fix2113 - Database Service API Integration tests (#3185) 2022-03-06 14:59:32 +01:00
Pere Miquel Brull
4116233697
Fix #3105 - ColumnValuesToMatchRegex & other fixes (#3149) 
Fix #3105 - ColumnValuesToMatchRegex & other fixes (#3149)
 2022-03-04 18:11:49 +01:00
Pere Miquel Brull
bd7b91b448
Fix #3112 - col profile safety & sample data (#3142) 
Fix #3112 - col profile safety & sample data (#3142)
 2022-03-04 13:14:11 +01:00
Snyk bot
834bb8e483
fix: ingestion/tests/integration/source/mlflow/Dockerfile to reduce vulnerabilities (#3134) 
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-DEBIAN11-AOM-1290331
- https://snyk.io/vuln/SNYK-DEBIAN11-AOM-1298721
- https://snyk.io/vuln/SNYK-DEBIAN11-AOM-1300249
- https://snyk.io/vuln/SNYK-DEBIAN11-UNZIP-2396444
- https://snyk.io/vuln/SNYK-DEBIAN11-UNZIP-2396445
 2022-03-04 08:34:24 +01:00
Snyk bot
8132b91c30
fix: ingestion/tests/integration/lineage/airflow/requirements.txt to reduce vulnerabilities (#3130) 
The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-APACHEAIRFLOW-2413442
- https://snyk.io/vuln/SNYK-PYTHON-APACHEAIRFLOW-2414089
- https://snyk.io/vuln/SNYK-PYTHON-WTFORMS-40581
 2022-03-04 07:49:58 +01:00
Pere Miquel Brull
e96ac838ff
Fix #3084 - Implement missing tests (#3117) 
Fix #3084 - Implement missing tests
 2022-03-04 06:59:47 +01:00
Pere Miquel Brull
71207de362
Fix #2875 - Profiler API Sink (#3011) 
Fix #2875 - Profiler API Sink
 2022-03-02 16:46:28 +01:00
Alberto Miorin
fe5618c8f1
Fix #3037: metadata --version doesn't work (#3038) 2022-03-01 12:19:36 +01:00
Pere Miquel Brull
990608522a
Fix #2981 - Update Profile to match TableProfile (#2982) 2022-02-25 09:26:30 -08:00
Pere Miquel Brull
4d928bab80
Fix #2965 - Additional Properties set to False (#2976) 2022-02-24 08:38:56 -08:00
Pere Miquel Brull
a4b383fa83
Fix #2897 - Profiler CLI (#2967) 
Fix #2897 - Profiler CLI
 2022-02-24 08:03:50 +01:00
Pere Miquel Brull
1fb0e7c489
Fix #2878 & #2877 - Implement Metrics and Validate Composed Metrics (#2926) 
Fix #2878 & #2877 - Implement Metrics and Validate Composed Metrics
 2022-02-24 07:08:39 +01:00
Pere Miquel Brull
1224d20a36
Fix #2894 - Profiler Processor & Metrics (#2900) 
Fix #2894 - Profiler Processor & Metrics (#2900)
 2022-02-22 08:09:02 +01:00
Pere Miquel Brull
e55579aaa8
Fix #2845 - Init Profiler Workflow (#2862) 
* Fix list typing

* Add sqlite service

* Add sqlite service

* Add sqlite service

* Refactor validation into class

* Refactor validation into class

* Prepare table simple profiler

* Add note

* test ORM conversion

* Prepare workflow config utilities

* Prepare workflow skeleton

* Use new core Validation

* Refactor workflow config parsing

* Add comment

* Simplify workflow validations

* Fix table metric check

* Add init for convenience, otherwise interpreter cries when trying to __call__ the get result

* Fix table metric check

* Format

* Format

* Fix table list and metrics init

* Prepare profiler workflow integration tests

* Bump version

* Fix pycharm imports

* format
 2022-02-20 17:55:12 +01:00
Pere Miquel Brull
f304d290b4
Fix #2751 - Init ORM Profiler (#2831) 
* ORM Profiler skeleton

* Fix table name within service

* Add license

* Prepare custom types

* Fix converter

* Compute stddev only on numeric

* Prepare smart registries

* Update tests

* Update results retrieval

* Fix composed metrics result

* Format

* Add missing type

* Add _label decorator

* clean readme

* clean readme

* Filter types when profiler runs not allowed metric types

* Fix null ratio

* Add proper type

* RuleMetric skeleton

* Prepare table metrics

* Update simple profiler

* Format

* Define test expression grammar and node visiting

* Unify metric registry

* Prepare validation core

* Add grammar lib

* Add safe get

* Format

* Allow decimals in grammar

* Test validation conversion

* Fix validation conversion and test

* Rename to row_number

* Update READMEs

* Format

* Row number naming

* Fix rename
 2022-02-18 07:48:38 +01:00
Sriharsha Chintalapani
50799227e4
Fix #2817: Remove indexing of Glossary as another Elasticsearch index (#2818) 2022-02-16 20:10:26 -08:00
Suresh Srinivas
28ba1a3c04
Fixes #2760 - Add entities for Glossary and initial API (#2761) 
* Fixes #2760 - Add entities for Glossary and initial API

* Fixing merge issues
 2022-02-15 20:54:46 -08:00
Pere Miquel Brull
76f4ccd590
Fix #2735 - Simplify Airflow properties extraction (#2749) 
* Simplify lineage properties extraction

* Add network name

* Format
 2022-02-14 16:53:42 +01:00
Pere Miquel Brull
1b2ea54d4f
Fix #2316 - Append Tasks & Add Status in Airflow (#2738) 
* Add license

* Add date to timestamp helper

* Prepare pipeline status operation

* Update and clean tasks from client

* Update tasks from client

* Check if pipeline is empty

* Keep all current pipeline info

* Append and clean tasks

* format

* Add status information

* Test pipelineStatus update

* Update task on clear

* Log status on callback

* Update lineage and status docs

* Update lineage docs

* Format

* Logic to handle DAG status

* Lint and format

* Update lineage tests
 2022-02-13 17:51:25 +01:00
Teddy
b5ddada930
Fix #1526: Added OMetaTag mixin 2022-02-06 12:42:13 -08:00
Vivek Ratnavel Subramanian
e6343a79d7
Fix #2555: PUT APIs should only allow admin and owners to update a resource in secure env (#2608) 2022-02-04 12:39:08 -08:00
Mayur Singal
5a6e7d743e
ISSUE-545: Added tests for QueryProcessor (#2510) 
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-02-04 08:20:20 -08:00
Robert Pack
67a1c23482
Fix #2348: Harmonize request titles in json schemas 2022-01-31 16:29:56 -08:00
Pere Miquel Brull
05c8d1ebb6
Fix #2153 - Lineage date parsing (#2387) 
* Check for start_date

* Fix date keys and remove repr

* Add date management tests
 2022-01-24 12:29:16 -08:00
Sriharsha Chintalapani
f390bac375
Fix 2270: Column Type Parser (#2271) 
* Fix 2270: Column Type Parser

* Fix 2270: Column Type Parser

* Added checks to allow arrayDataType and dataTypeDisplay

* Modified - arrayDataType and dataTypeDisplay

* Update sql_source.py

* Update sql_source.py

* file formatted

* Modified according to column_type_parser.py

* modified / refactored / deleted the files

* Added Tests, modified sql_source

* file formatted

* Added missing datatypes

* Added Tests

* Added Tests

* Added Tests - refactored expected output into a json file

* file formatted

* Sample Data Updated

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-22 23:32:58 +05:30
Sriharsha Chintalapani
10e71a7e09
Fix #2340: Ingestion pipeline deployment refactor (#2341) 
* Fix #2340: Ingestion pipeline deployment refactor
 2022-01-21 22:06:14 -08:00
codingwithabhi
ddf4ae261b
Bigquery credential configuration updated (#2041) 
* bigquery-credentials-in-bigquery-json

* ui-changes-reverted

* credentials-moved-to-options

* Update bigquery.json

* Update bigquery.py

* delete-temp-code-added

* delete-temp-code-added

* removed-init-from-config

* Update datatypes_test.py

* Update column_helpers.py

* code-formatted

Co-authored-by: Abhishek <abhishek@Abhisheks-MacBook-Air.local>
Co-authored-by: = <=>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-21 17:23:02 +05:30
Sriharsha Chintalapani
0d3ded0742
Data Profiler Integration (#2235) 
* Fix 2234: Data profiler integration
 2022-01-18 20:25:43 -08:00
Pere Miquel Brull
a9c840778a
[issue-2116] - Add target property to MlModel (#2134) 
* Add target property to MlModel

* Use pull_request_target for forks

* Simplify str

* Update ingestion-core setup

* Bump ingestion-core version

* Update install
 2022-01-18 16:27:35 +01:00
Pere Miquel Brull
2728041d0f
[issue-2213] - Airflow lineage refactor (#2218) 
* Fix ometa logging

* Refactor airflow lineage

* Refactor airflow lineage

* Add airflow lineage integration tests

* Fix circular dependency
 2022-01-14 13:00:33 -08:00
Pere Miquel Brull
d3b6c7cf27
[issue-1973] - Python API from Sklearn to MlModel (#2119) 
* Move staticmethods to utils

* Use functions from utils

* Convert sklearn to MlModel

* merge main
 2022-01-10 09:36:08 +01:00
Pere Miquel Brull
178315d68a
[issue-2112] - Add helper to get EntityReference in OMeta API (#2114) 
* Use entity list from mixin

* Add entity reference helper

* Add tests for retrieving the entity reference

* Add missing space

* Fix shadowing

* Use get entity ref

* Use get entity ref
 2022-01-10 09:07:39 +01:00
Teddy
dbcc5e9dd2
ISSUE-1048: Add Version Mixin (#2018) 
* Added entity version mixins logics to ometa API

* remove logging in line 385 used for testing

* Fixed black error + ran isort

* remove extra underscore in

* Added integration tests for OpenMetadata versions methods

* Fixed linting errors in versionMixin.py
 2022-01-07 10:37:56 +01:00
Pere Miquel Brull
48ebcffbd0
[issue-1976] - Ingestion SonarCloud (#2085) 
* Update path

* Prepare sonar properties

* Prepare coverage recipes

* Add coverage

* Simplify pytest

* Organise integration tests

* Update path

* Use setup instead of reqs

* Update recipes

* Fix PR event to target

* Update event_name

* Prepare sonar

* Run tests & sonar

* Use sonarcloud host

* Fix compose

* Use ingestion token
 2022-01-07 10:28:38 +01:00
Alberto Miorin
c8de136cca
First part of #2091: Python code in branch main is always formatted correctly (#2093) 2022-01-07 06:17:48 +01:00
Pere Miquel Brull
eb01c55676
Update naming (#2080) 2022-01-06 14:42:50 +01:00
Ayush Shah
5d6f385a75
Added Filter Params for Table and Schema (#1954) 
* Added Filter Params for table and Schema

* Bigquery Doc changes

* Doc Changes for databases

* Filter Pattern Changes

* Table Filter Pattern Example Changes

* Filter Pattern Example Changes
 2021-12-29 09:13:09 -08:00
Pere Miquel Brull
1f01081698
Clean DBT Model (#1888) 2021-12-22 17:17:05 +01:00