yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-11-01 02:56:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,027 Commits 873 Branches 177 Tags
Commit Graph

394 Commits

Author SHA1 Message Date
Ayush Shah
3a6217a7e9
Fixed data profiling and raw data type issue (#2474) 
* Fixed data profiling and raw data type issue

* Update metadata_rest.py

* Update sql_source.py
 2022-01-27 18:59:35 +05:30
Mayur Singal
a8bdf844dd
make black (#2475) 
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-01-27 18:58:30 +05:30
Sriharsha Chintalapani
c36ae72ccd
Issue #2462: Fix profiler group_by query (#2463) 2022-01-27 14:26:32 +05:30
Sriharsha Chintalapani
4cd5c2bea2
Issue #2457: remove logging utils (#2458) 2022-01-26 10:26:17 -08:00
Ayush Shah
02f3678f2a
Fixed Mysql profiler + removed additional dependency for profiler (#2447) 
* Fixed Mysql profiler + removed additional dependency for profiler

* Update sql_source.py

* Update sql_source.py

* Update setup.py
 2022-01-26 16:45:28 +05:30
Mayur Singal
cfca795edf
ISSUE-2422: Handled None datatype (#2423) 
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-01-25 08:42:37 -08:00
Sriharsha Chintalapani
0e736012a9
Fix #1843: Add Delta Lake Connector (#1844) 
* Fix #1843: Add Delta Lake Connector

* Fix #1843: Add Delta Lake Connector
 2022-01-25 14:35:44 +05:30
Ayush Shah
917ccd7147
Bigquery and Helpers fix - Ingestion (#2408) 
* Bigquery Ingestion fix - error handling and keyerror

* Update helpers.py and column_type_parser

* Update helpers.py
 2022-01-25 14:30:09 +05:30
Pere Miquel Brull
ce5df0eb88
Use path str properly (#2405) 2022-01-25 08:55:44 +01:00
Ayush Shah
b689c2d632
Multiple fixes - Looker, Ingestion fix, pyarrow dependency (#2385) 
* Added pyarrow dependency

* Fix #2378

* Fix Looker

* Fix Looker, secretstr to str
 2022-01-24 12:29:54 -08:00
Pere Miquel Brull
05c8d1ebb6
Fix #2153 - Lineage date parsing (#2387) 
* Check for start_date

* Fix date keys and remove repr

* Add date management tests
 2022-01-24 12:29:16 -08:00
Pere Miquel Brull
6a8fc9e0fb
import on call (#2382) 2022-01-24 12:29:03 -08:00
Pere Miquel Brull
b39b698eb7
Fix #2345 - Make bump & unify (#2359) 
* Add installing instructions

* Unify makefile and automate bump version setup

* Update submake call

* Mark targets as phony

* Add help to recipes

* Update core publish

* Do not trigger install from echo

* Run make from root

* Prepare py_format from isort and black

* Run py_format

* Check black and isort formatting

* Update comment msg
 2022-01-23 09:47:06 -08:00
Ayush Shah
6f5a4eafe7
Bigquery support for env without passing any config (#2160) 
* Bigquery support for env without passing any config

* Bigquery File formatted
 2022-01-22 22:25:48 -08:00
Sriharsha Chintalapani
e5fdf77d8e
Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service (#2357) 
* Fix #2340: Ingestion pipeline deployment refactor - update ingestion

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service
 2022-01-22 21:12:25 -08:00
Pere Miquel Brull
1eb3f1ad41
[issue-2322] - Backup CLI (#2349) 
* Prepare cli module

* Check call

* Prepare backup

* Run pytest on schema changes

* Remove traces

* Add backup plugin

* run generate during CI

* Install after generate

* Address logger hotspot
 2022-01-22 12:08:14 -08:00
ulixius9
8fca53ec52
ISSUE-2048: Bigquery data profiler uses schema name instead of project id (#2335) 
* ISSUE-2048: Bigquery data profiler uses schema name instead of project id

* Update bigquery.py

* Update bigquery.py

* Update profiler_runner.py

* Update profiler.py

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-22 23:33:15 +05:30
Sriharsha Chintalapani
f390bac375
Fix 2270: Column Type Parser (#2271) 
* Fix 2270: Column Type Parser

* Fix 2270: Column Type Parser

* Added checks to allow arrayDataType and dataTypeDisplay

* Modified - arrayDataType and dataTypeDisplay

* Update sql_source.py

* Update sql_source.py

* file formatted

* Modified according to column_type_parser.py

* modified / refactored / deleted the files

* Added Tests, modified sql_source

* file formatted

* Added missing datatypes

* Added Tests

* Added Tests

* Added Tests - refactored expected output into a json file

* file formatted

* Sample Data Updated

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-22 23:32:58 +05:30
Sriharsha Chintalapani
10e71a7e09
Fix #2340: Ingestion pipeline deployment refactor (#2341) 
* Fix #2340: Ingestion pipeline deployment refactor
 2022-01-21 22:06:14 -08:00
codingwithabhi
ddf4ae261b
Bigquery credential configuration updated (#2041) 
* bigquery-credentials-in-bigquery-json

* ui-changes-reverted

* credentials-moved-to-options

* Update bigquery.json

* Update bigquery.py

* delete-temp-code-added

* delete-temp-code-added

* removed-init-from-config

* Update datatypes_test.py

* Update column_helpers.py

* code-formatted

Co-authored-by: Abhishek <abhishek@Abhisheks-MacBook-Air.local>
Co-authored-by: = <=>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-21 17:23:02 +05:30
Matt
2a80caa8fe
Add logic to initialize relationships from seed data during application startup (#2307) 
* Add logic to initialize relationships from seed data during application startup

* Remove ingestion related code for access control policies

* Move PolicyEvaluator init to PolicyResource
 2022-01-20 22:18:03 -08:00
Sriharsha Chintalapani
18f5c288fe Fix #2320: Fix DB migration scripts to upgrade to 0.8 from 0.7 2022-01-20 15:08:03 -08:00
Sriharsha Chintalapani
20e6c4bc86 fix upgrade 2022-01-20 15:08:03 -08:00
ulixius9
d594499fa0
ISSUE-1838: Added table type in glue connector (#2309) 
* ISSUE-1838: Added table type in glue connector

* Initialized table type with regular

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-01-20 07:57:44 -08:00
ulixius9
d6522e2ab4
ISSUE-2272: Ingestion: Roles and Teams apis called only when ingesting users (#2284) 
* ISSUE-2272: Ingestion: Roles and Teams apis called only when ingesting users

* reformatted metadata_rest.py

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-01-20 12:24:35 +05:30
Sriharsha Chintalapani
0a3ba3ddfd
Fix 2276: fix elastic search indexing (#2278) 2022-01-18 23:30:36 -08:00
Sriharsha Chintalapani
0d3ded0742
Data Profiler Integration (#2235) 
* Fix 2234: Data profiler integration
 2022-01-18 20:25:43 -08:00
Sriharsha Chintalapani
b4983764e5
Fix 2232: Presto - connection formatting issues when username/password configured (#2233) 
* Fix 2232: Presto - connection formatting issues when username/password configured

* Fix 2232: Presto - connection formatting issues when username/password configured
 2022-01-17 11:03:17 +05:30
Pere Miquel Brull
ebda556adc
[issue-2223] - Lineage Callback for any operator (#2225) 
* Fix logging

* Reorganise lineage

* Add callback

* Add metadata config

* Fix type smell
 2022-01-15 18:57:48 +01:00
Pere Miquel Brull
2728041d0f
[issue-2213] - Airflow lineage refactor (#2218) 
* Fix ometa logging

* Refactor airflow lineage

* Refactor airflow lineage

* Add airflow lineage integration tests

* Fix circular dependency
 2022-01-14 13:00:33 -08:00
Akash Jain
105d8c1b67
Feat: docker cmd improvements (#1895) 
* revamped docker commands

* update metadata docker commands in comments

* Added Docker Compose Check

* code refactored to docker.py

* fix py-checkstyle lints

* Adding clean functionality; refactor code

* pylint checkstyle fixes

* version string revised
 2022-01-13 20:39:46 -08:00
codingwithabhi
5714c4122b
Move sql queries (#2183) 
* sql-queries-moved-separate-file

* sql-queries-moved-separate-file

* redshift-and-neo4j-queries-added

* redshift-and-neo4j-queries-added

* neo4j-query-imported

* code-formatted

Co-authored-by: = <=>
 2022-01-13 14:16:54 +05:30
Sriharsha Chintalapani
c4d7c76198
Event publishers to ElasticSearch and Slack (#2118) 
* ElasticSearch & Slack Event publishers

* Fix #2165: Support for deleted entities in ElasticSearch
 2022-01-12 16:35:27 -08:00
Matt
f323afe0cd
Support role ingestion from sample data (#2167) 2022-01-11 16:51:18 -08:00
Ayush Shah
f379b35279
Reflection Cache Implementation (#2016) 
* Reflection Cache for Bigquery and Redshift

* Overrided few sqlalchemy packages

* Added Geography Support

* Reformatted files

* DBT models error handling implemented

* Geography type added as a custom sqlalchemy datatype

* GEOGRAPHY and VARIANT added as custom sql types

* Implemented file formatting using black

* Implemented file formatting using black
 2022-01-11 14:58:03 +05:30
Sriharsha Chintalapani
cf6f438531
Fix #1994: Add support for marking dataset entities as deleted (#2141) 
* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted
 2022-01-11 11:42:53 +05:30
Pere Miquel Brull
d3b6c7cf27
[issue-1973] - Python API from Sklearn to MlModel (#2119) 
* Move staticmethods to utils

* Use functions from utils

* Convert sklearn to MlModel

* merge main
 2022-01-10 09:36:08 +01:00
Pere Miquel Brull
178315d68a
[issue-2112] - Add helper to get EntityReference in OMeta API (#2114) 
* Use entity list from mixin

* Add entity reference helper

* Add tests for retrieving the entity reference

* Add missing space

* Fix shadowing

* Use get entity ref

* Use get entity ref
 2022-01-10 09:07:39 +01:00
Pere Miquel Brull
74e9038b92
[issue-997] - More pylint fixes (#2107) 
* Update naming and disable false positives

* Rename for naming convention

* Fix linting issues

* Moved pylint to pylintrc

* Moved pylint to pylintrc

* Rename CI

* Check sonar properties

* Fix pull_request_target checkout
 2022-01-07 19:50:53 +01:00
Teddy
dbcc5e9dd2
ISSUE-1048: Add Version Mixin (#2018) 
* Added entity version mixins logics to ometa API

* remove logging in line 385 used for testing

* Fixed black error + ran isort

* remove extra underscore in

* Added integration tests for OpenMetadata versions methods

* Fixed linting errors in versionMixin.py
 2022-01-07 10:37:56 +01:00
Alberto Miorin
c8de136cca
First part of #2091: Python code in branch main is always formatted correctly (#2093) 2022-01-07 06:17:48 +01:00
Ayush Shah
860f0ae8db
Iterable from Collections not working (#2019) 2022-01-04 06:48:18 -08:00
Ayush Shah
19e5f737ff
Usage fix - removed service_name from bulk_sink (#1956) 
* Query Usage fixed

* Usage updated

* Reformatted change

* Usage files reformatted
 2022-01-04 17:12:10 +05:30
Matt
ab4c9ede25
Add ingestion for access_control_policies to support RBAC (#1969) 2021-12-30 08:25:09 -08:00
Pere Miquel Brull
1e334af89c
Fix linting (#1958) 2021-12-29 17:33:40 +01:00
Sriharsha Chintalapani
30eeec498f
Fix #1952: Airflow Openmetadata lineage allow config to be read from … (#1953) 
* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable
 2021-12-29 11:31:18 +05:30
Matt
a2b1d96636
Address code smells within ingestion source (#1951) 2021-12-28 17:40:55 -08:00
Ayush Shah
431f47fb3c
Kafka and Oracle issues fixed (#1917) 
* Kafka and Oracle issues fixed

* Kafka failure status updated
 2021-12-27 08:04:56 -08:00
Matt
9ff0fdc533
Refactor Glue ingestion to use common config and client (#1925) 
- Use AWSClientConfigModel and AWSClient for glue.py
- Use helpers.get_storage_service_or_create for s3.py
 2021-12-26 20:27:08 -08:00
Matt
f7653f4122
Add support for GCS Lifecycle Policy Ingestion (#1923) 
- Add example gcs.json
- Add support for MoveAction and DeleteAction
 2021-12-26 20:26:33 -08:00