yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-08-16 13:07:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
10,673 Commits 726 Branches 153 Tags
Commit Graph

2916 Commits

Author SHA1 Message Date
Sriharsha Chintalapani
4cd5c2bea2
Issue #2457: remove logging utils (#2458) 2022-01-26 10:26:17 -08:00
Ayush Shah
02f3678f2a
Fixed Mysql profiler + removed additional dependency for profiler (#2447) 
* Fixed Mysql profiler + removed additional dependency for profiler

* Update sql_source.py

* Update sql_source.py

* Update setup.py
 2022-01-26 16:45:28 +05:30
Akash Jain
20309cde64
fix:cherry pick from 0.8.0 and miscellaneous fixes (#2445) 
* updating ingestion-core version to 0.8.0

* updating ingestion version to 0.8.0

* fixing file copy path

* update docker images to point to 0.8.0

* set up new release version

* reset ingestion version to 0.9.0.dev0

* reset ingestion-core version to 0.9.0
 2022-01-26 14:00:27 +05:30
Mayur Singal
cfca795edf
ISSUE-2422: Handled None datatype (#2423) 
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-01-25 08:42:37 -08:00
Sriharsha Chintalapani
0e736012a9
Fix #1843: Add Delta Lake Connector (#1844) 
* Fix #1843: Add Delta Lake Connector

* Fix #1843: Add Delta Lake Connector
 2022-01-25 14:35:44 +05:30
Ayush Shah
917ccd7147
Bigquery and Helpers fix - Ingestion (#2408) 
* Bigquery Ingestion fix - error handling and keyerror

* Update helpers.py and column_type_parser

* Update helpers.py
 2022-01-25 14:30:09 +05:30
Pere Miquel Brull
ce5df0eb88
Use path str properly (#2405) 2022-01-25 08:55:44 +01:00
Ayush Shah
b4d63bd951
Fix: Typo in setting up pyarrow dependency (#2398) 2022-01-25 11:58:10 +05:30
Ayush Shah
b689c2d632
Multiple fixes - Looker, Ingestion fix, pyarrow dependency (#2385) 
* Added pyarrow dependency

* Fix #2378

* Fix Looker

* Fix Looker, secretstr to str
 2022-01-24 12:29:54 -08:00
Pere Miquel Brull
05c8d1ebb6
Fix #2153 - Lineage date parsing (#2387) 
* Check for start_date

* Fix date keys and remove repr

* Add date management tests
 2022-01-24 12:29:16 -08:00
Pere Miquel Brull
6a8fc9e0fb
import on call (#2382) 2022-01-24 12:29:03 -08:00
Pere Miquel Brull
b39b698eb7
Fix #2345 - Make bump & unify (#2359) 
* Add installing instructions

* Unify makefile and automate bump version setup

* Update submake call

* Mark targets as phony

* Add help to recipes

* Update core publish

* Do not trigger install from echo

* Run make from root

* Prepare py_format from isort and black

* Run py_format

* Check black and isort formatting

* Update comment msg
 2022-01-23 09:47:06 -08:00
Ayush Shah
6f5a4eafe7
Bigquery support for env without passing any config (#2160) 
* Bigquery support for env without passing any config

* Bigquery File formatted
 2022-01-22 22:25:48 -08:00
Sriharsha Chintalapani
e5fdf77d8e
Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service (#2357) 
* Fix #2340: Ingestion pipeline deployment refactor - update ingestion

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service
 2022-01-22 21:12:25 -08:00
Pere Miquel Brull
1eb3f1ad41
[issue-2322] - Backup CLI (#2349) 
* Prepare cli module

* Check call

* Prepare backup

* Run pytest on schema changes

* Remove traces

* Add backup plugin

* run generate during CI

* Install after generate

* Address logger hotspot
 2022-01-22 12:08:14 -08:00
ulixius9
8fca53ec52
ISSUE-2048: Bigquery data profiler uses schema name instead of project id (#2335) 
* ISSUE-2048: Bigquery data profiler uses schema name instead of project id

* Update bigquery.py

* Update bigquery.py

* Update profiler_runner.py

* Update profiler.py

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-22 23:33:15 +05:30
Sriharsha Chintalapani
f390bac375
Fix 2270: Column Type Parser (#2271) 
* Fix 2270: Column Type Parser

* Fix 2270: Column Type Parser

* Added checks to allow arrayDataType and dataTypeDisplay

* Modified - arrayDataType and dataTypeDisplay

* Update sql_source.py

* Update sql_source.py

* file formatted

* Modified according to column_type_parser.py

* modified / refactored / deleted the files

* Added Tests, modified sql_source

* file formatted

* Added missing datatypes

* Added Tests

* Added Tests

* Added Tests - refactored expected output into a json file

* file formatted

* Sample Data Updated

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-22 23:32:58 +05:30
Pere Miquel Brull
6693b4ae40
[issue-2262] - Simplify sample data DAG (#2337) 
* Simplify sample data DAG

* Remove mkdir

* Generate sources before running compose

* Generate sources to install models to ingestion image

* Add python-on-whales for docker --start

* Remove python-on-whales from base

* Install venv

* Setup python
 2022-01-22 16:37:56 +05:30
Sriharsha Chintalapani
10e71a7e09
Fix #2340: Ingestion pipeline deployment refactor (#2341) 
* Fix #2340: Ingestion pipeline deployment refactor
 2022-01-21 22:06:14 -08:00
codingwithabhi
ddf4ae261b
Bigquery credential configuration updated (#2041) 
* bigquery-credentials-in-bigquery-json

* ui-changes-reverted

* credentials-moved-to-options

* Update bigquery.json

* Update bigquery.py

* delete-temp-code-added

* delete-temp-code-added

* removed-init-from-config

* Update datatypes_test.py

* Update column_helpers.py

* code-formatted

Co-authored-by: Abhishek <abhishek@Abhisheks-MacBook-Air.local>
Co-authored-by: = <=>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-21 17:23:02 +05:30
Matt
2a80caa8fe
Add logic to initialize relationships from seed data during application startup (#2307) 
* Add logic to initialize relationships from seed data during application startup

* Remove ingestion related code for access control policies

* Move PolicyEvaluator init to PolicyResource
 2022-01-20 22:18:03 -08:00
Sriharsha Chintalapani
18f5c288fe Fix #2320: Fix DB migration scripts to upgrade to 0.8 from 0.7 2022-01-20 15:08:03 -08:00
Sriharsha Chintalapani
20e6c4bc86 fix upgrade 2022-01-20 15:08:03 -08:00
ulixius9
d594499fa0
ISSUE-1838: Added table type in glue connector (#2309) 
* ISSUE-1838: Added table type in glue connector

* Initialized table type with regular

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-01-20 07:57:44 -08:00
ulixius9
d6522e2ab4
ISSUE-2272: Ingestion: Roles and Teams apis called only when ingesting users (#2284) 
* ISSUE-2272: Ingestion: Roles and Teams apis called only when ingesting users

* reformatted metadata_rest.py

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-01-20 12:24:35 +05:30
Sriharsha Chintalapani
0a3ba3ddfd
Fix 2276: fix elastic search indexing (#2278) 2022-01-18 23:30:36 -08:00
Sriharsha Chintalapani
0d3ded0742
Data Profiler Integration (#2235) 
* Fix 2234: Data profiler integration
 2022-01-18 20:25:43 -08:00
Pere Miquel Brull
a9c840778a
[issue-2116] - Add target property to MlModel (#2134) 
* Add target property to MlModel

* Use pull_request_target for forks

* Simplify str

* Update ingestion-core setup

* Bump ingestion-core version

* Update install
 2022-01-18 16:27:35 +01:00
Sriharsha Chintalapani
b4983764e5
Fix 2232: Presto - connection formatting issues when username/password configured (#2233) 
* Fix 2232: Presto - connection formatting issues when username/password configured

* Fix 2232: Presto - connection formatting issues when username/password configured
 2022-01-17 11:03:17 +05:30
Matt
276128e8b0
Add MetadataOperations RBAC support for PATCH APIs (#2128) 
* Add RBAC for PATCH APIs

* Expand scope to all resources except a few (Policy, User, Role, Team, resources that doesn't support PATCH)

* Fix code smells
 2022-01-16 09:24:26 -08:00
Pere Miquel Brull
ebda556adc
[issue-2223] - Lineage Callback for any operator (#2225) 
* Fix logging

* Reorganise lineage

* Add callback

* Add metadata config

* Fix type smell
 2022-01-15 18:57:48 +01:00
Pere Miquel Brull
2728041d0f
[issue-2213] - Airflow lineage refactor (#2218) 
* Fix ometa logging

* Refactor airflow lineage

* Refactor airflow lineage

* Add airflow lineage integration tests

* Fix circular dependency
 2022-01-14 13:00:33 -08:00
Akash Jain
105d8c1b67
Feat: docker cmd improvements (#1895) 
* revamped docker commands

* update metadata docker commands in comments

* Added Docker Compose Check

* code refactored to docker.py

* fix py-checkstyle lints

* Adding clean functionality; refactor code

* pylint checkstyle fixes

* version string revised
 2022-01-13 20:39:46 -08:00
codingwithabhi
5714c4122b
Move sql queries (#2183) 
* sql-queries-moved-separate-file

* sql-queries-moved-separate-file

* redshift-and-neo4j-queries-added

* redshift-and-neo4j-queries-added

* neo4j-query-imported

* code-formatted

Co-authored-by: = <=>
 2022-01-13 14:16:54 +05:30
Sriharsha Chintalapani
c4d7c76198
Event publishers to ElasticSearch and Slack (#2118) 
* ElasticSearch & Slack Event publishers

* Fix #2165: Support for deleted entities in ElasticSearch
 2022-01-12 16:35:27 -08:00
Matt
f323afe0cd
Support role ingestion from sample data (#2167) 2022-01-11 16:51:18 -08:00
Ayush Shah
f379b35279
Reflection Cache Implementation (#2016) 
* Reflection Cache for Bigquery and Redshift

* Overrided few sqlalchemy packages

* Added Geography Support

* Reformatted files

* DBT models error handling implemented

* Geography type added as a custom sqlalchemy datatype

* GEOGRAPHY and VARIANT added as custom sql types

* Implemented file formatting using black

* Implemented file formatting using black
 2022-01-11 14:58:03 +05:30
Sriharsha Chintalapani
cf6f438531
Fix #1994: Add support for marking dataset entities as deleted (#2141) 
* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted
 2022-01-11 11:42:53 +05:30
Pere Miquel Brull
bc05e4945d
update name (#2138) 2022-01-10 08:56:12 -08:00
Pere Miquel Brull
d3b6c7cf27
[issue-1973] - Python API from Sklearn to MlModel (#2119) 
* Move staticmethods to utils

* Use functions from utils

* Convert sklearn to MlModel

* merge main
 2022-01-10 09:36:08 +01:00
Pere Miquel Brull
178315d68a
[issue-2112] - Add helper to get EntityReference in OMeta API (#2114) 
* Use entity list from mixin

* Add entity reference helper

* Add tests for retrieving the entity reference

* Add missing space

* Fix shadowing

* Use get entity ref

* Use get entity ref
 2022-01-10 09:07:39 +01:00
Pere Miquel Brull
74e9038b92
[issue-997] - More pylint fixes (#2107) 
* Update naming and disable false positives

* Rename for naming convention

* Fix linting issues

* Moved pylint to pylintrc

* Moved pylint to pylintrc

* Rename CI

* Check sonar properties

* Fix pull_request_target checkout
 2022-01-07 19:50:53 +01:00
Teddy
dbcc5e9dd2
ISSUE-1048: Add Version Mixin (#2018) 
* Added entity version mixins logics to ometa API

* remove logging in line 385 used for testing

* Fixed black error + ran isort

* remove extra underscore in

* Added integration tests for OpenMetadata versions methods

* Fixed linting errors in versionMixin.py
 2022-01-07 10:37:56 +01:00
Pere Miquel Brull
48ebcffbd0
[issue-1976] - Ingestion SonarCloud (#2085) 
* Update path

* Prepare sonar properties

* Prepare coverage recipes

* Add coverage

* Simplify pytest

* Organise integration tests

* Update path

* Use setup instead of reqs

* Update recipes

* Fix PR event to target

* Update event_name

* Prepare sonar

* Run tests & sonar

* Use sonarcloud host

* Fix compose

* Use ingestion token
 2022-01-07 10:28:38 +01:00
Alberto Miorin
c8de136cca
First part of #2091: Python code in branch main is always formatted correctly (#2093) 2022-01-07 06:17:48 +01:00
Alberto Miorin
34a9d31240
Fix #2052: Revert the reverted pre-commit (#2053) 2022-01-07 03:53:21 +01:00
Pere Miquel Brull
eb01c55676
Update naming (#2080) 2022-01-06 14:42:50 +01:00
Ayush Shah
860f0ae8db
Iterable from Collections not working (#2019) 2022-01-04 06:48:18 -08:00
Ayush Shah
19e5f737ff
Usage fix - removed service_name from bulk_sink (#1956) 
* Query Usage fixed

* Usage updated

* Reformatted change

* Usage files reformatted
 2022-01-04 17:12:10 +05:30
Akash Jain
348e23ca6f
fix/ci-workflows (#1997) 
* add pytest and pytest-cov

* add timeout; update make command usages
 2022-01-04 12:49:22 +05:30