yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-10-31 18:48:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,739 Commits 873 Branches 177 Tags
Commit Graph

378 Commits

Author SHA1 Message Date
Pere Miquel Brull
9887e0773e
Fix #2516 - User Profile & Amundsen Source (#2546) 2022-02-01 13:55:26 -08:00
Ayush Shah
763f125861
Bigquery Os unlink issue resolved (#2544) 
* Bigquery Os unlink issue resolved

* Bigquery code modified
 2022-02-01 23:33:08 +05:30
Robert Pack
67a1c23482
Fix #2348: Harmonize request titles in json schemas 2022-01-31 16:29:56 -08:00
Ayush Shah
73d848ac3b
Docker Typeerror fixed (#2504) 2022-01-31 12:30:12 +05:30
Pere Miquel Brull
6ea6f0c43d
Fix #2383 - Multiple validation workflows (#2500) 
* Clean duplicated validation

* Move black dep to dev

* Fix black version for ingestion core

* Validate core generated in the same workflow
 2022-01-30 12:26:41 -08:00
Pere Miquel Brull
d10639958c
Fix black version 21.12b0 (#2499) 2022-01-30 06:54:11 -08:00
Sriharsha Chintalapani
f6ac393007
Issue #2492: Improve messaging on docker command (#2493) 
* Issue #2492: Improve messaging on docker command

* Issue #2492: Improve messaging on docker command
 2022-01-29 09:47:21 +05:30
Vivek Ratnavel Subramanian
52ec2e3ad1
Fix #2478: Update the auth header to Authorization Bearer for JWT token (#2483) 
* Fix #2478: Update the auth header to Authorization Bearer for JWT token
 2022-01-27 23:58:54 -08:00
Teddy
4f3e330dd8
Issue 997-01 (#2476) 
* Fix linting

* Fixed pyformat errors

* Address comments from PR review

* Added back phony in makefile

* Added back comment
 2022-01-27 18:45:45 -08:00
Ayush Shah
3a6217a7e9
Fixed data profiling and raw data type issue (#2474) 
* Fixed data profiling and raw data type issue

* Update metadata_rest.py

* Update sql_source.py
 2022-01-27 18:59:35 +05:30
Mayur Singal
a8bdf844dd
make black (#2475) 
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-01-27 18:58:30 +05:30
Sriharsha Chintalapani
c36ae72ccd
Issue #2462: Fix profiler group_by query (#2463) 2022-01-27 14:26:32 +05:30
Sriharsha Chintalapani
4cd5c2bea2
Issue #2457: remove logging utils (#2458) 2022-01-26 10:26:17 -08:00
Ayush Shah
02f3678f2a
Fixed Mysql profiler + removed additional dependency for profiler (#2447) 
* Fixed Mysql profiler + removed additional dependency for profiler

* Update sql_source.py

* Update sql_source.py

* Update setup.py
 2022-01-26 16:45:28 +05:30
Akash Jain
20309cde64
fix:cherry pick from 0.8.0 and miscellaneous fixes (#2445) 
* updating ingestion-core version to 0.8.0

* updating ingestion version to 0.8.0

* fixing file copy path

* update docker images to point to 0.8.0

* set up new release version

* reset ingestion version to 0.9.0.dev0

* reset ingestion-core version to 0.9.0
 2022-01-26 14:00:27 +05:30
Mayur Singal
cfca795edf
ISSUE-2422: Handled None datatype (#2423) 
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-01-25 08:42:37 -08:00
Sriharsha Chintalapani
0e736012a9
Fix #1843: Add Delta Lake Connector (#1844) 
* Fix #1843: Add Delta Lake Connector

* Fix #1843: Add Delta Lake Connector
 2022-01-25 14:35:44 +05:30
Ayush Shah
917ccd7147
Bigquery and Helpers fix - Ingestion (#2408) 
* Bigquery Ingestion fix - error handling and keyerror

* Update helpers.py and column_type_parser

* Update helpers.py
 2022-01-25 14:30:09 +05:30
Pere Miquel Brull
ce5df0eb88
Use path str properly (#2405) 2022-01-25 08:55:44 +01:00
Ayush Shah
b4d63bd951
Fix: Typo in setting up pyarrow dependency (#2398) 2022-01-25 11:58:10 +05:30
Ayush Shah
b689c2d632
Multiple fixes - Looker, Ingestion fix, pyarrow dependency (#2385) 
* Added pyarrow dependency

* Fix #2378

* Fix Looker

* Fix Looker, secretstr to str
 2022-01-24 12:29:54 -08:00
Pere Miquel Brull
05c8d1ebb6
Fix #2153 - Lineage date parsing (#2387) 
* Check for start_date

* Fix date keys and remove repr

* Add date management tests
 2022-01-24 12:29:16 -08:00
Pere Miquel Brull
6a8fc9e0fb
import on call (#2382) 2022-01-24 12:29:03 -08:00
Pere Miquel Brull
b39b698eb7
Fix #2345 - Make bump & unify (#2359) 
* Add installing instructions

* Unify makefile and automate bump version setup

* Update submake call

* Mark targets as phony

* Add help to recipes

* Update core publish

* Do not trigger install from echo

* Run make from root

* Prepare py_format from isort and black

* Run py_format

* Check black and isort formatting

* Update comment msg
 2022-01-23 09:47:06 -08:00
Ayush Shah
6f5a4eafe7
Bigquery support for env without passing any config (#2160) 
* Bigquery support for env without passing any config

* Bigquery File formatted
 2022-01-22 22:25:48 -08:00
Sriharsha Chintalapani
e5fdf77d8e
Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service (#2357) 
* Fix #2340: Ingestion pipeline deployment refactor - update ingestion

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service
 2022-01-22 21:12:25 -08:00
Pere Miquel Brull
1eb3f1ad41
[issue-2322] - Backup CLI (#2349) 
* Prepare cli module

* Check call

* Prepare backup

* Run pytest on schema changes

* Remove traces

* Add backup plugin

* run generate during CI

* Install after generate

* Address logger hotspot
 2022-01-22 12:08:14 -08:00
ulixius9
8fca53ec52
ISSUE-2048: Bigquery data profiler uses schema name instead of project id (#2335) 
* ISSUE-2048: Bigquery data profiler uses schema name instead of project id

* Update bigquery.py

* Update bigquery.py

* Update profiler_runner.py

* Update profiler.py

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-22 23:33:15 +05:30
Sriharsha Chintalapani
f390bac375
Fix 2270: Column Type Parser (#2271) 
* Fix 2270: Column Type Parser

* Fix 2270: Column Type Parser

* Added checks to allow arrayDataType and dataTypeDisplay

* Modified - arrayDataType and dataTypeDisplay

* Update sql_source.py

* Update sql_source.py

* file formatted

* Modified according to column_type_parser.py

* modified / refactored / deleted the files

* Added Tests, modified sql_source

* file formatted

* Added missing datatypes

* Added Tests

* Added Tests

* Added Tests - refactored expected output into a json file

* file formatted

* Sample Data Updated

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-22 23:32:58 +05:30
Pere Miquel Brull
6693b4ae40
[issue-2262] - Simplify sample data DAG (#2337) 
* Simplify sample data DAG

* Remove mkdir

* Generate sources before running compose

* Generate sources to install models to ingestion image

* Add python-on-whales for docker --start

* Remove python-on-whales from base

* Install venv

* Setup python
 2022-01-22 16:37:56 +05:30
Sriharsha Chintalapani
10e71a7e09
Fix #2340: Ingestion pipeline deployment refactor (#2341) 
* Fix #2340: Ingestion pipeline deployment refactor
 2022-01-21 22:06:14 -08:00
codingwithabhi
ddf4ae261b
Bigquery credential configuration updated (#2041) 
* bigquery-credentials-in-bigquery-json

* ui-changes-reverted

* credentials-moved-to-options

* Update bigquery.json

* Update bigquery.py

* delete-temp-code-added

* delete-temp-code-added

* removed-init-from-config

* Update datatypes_test.py

* Update column_helpers.py

* code-formatted

Co-authored-by: Abhishek <abhishek@Abhisheks-MacBook-Air.local>
Co-authored-by: = <=>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-21 17:23:02 +05:30
Matt
2a80caa8fe
Add logic to initialize relationships from seed data during application startup (#2307) 
* Add logic to initialize relationships from seed data during application startup

* Remove ingestion related code for access control policies

* Move PolicyEvaluator init to PolicyResource
 2022-01-20 22:18:03 -08:00
Sriharsha Chintalapani
18f5c288fe Fix #2320: Fix DB migration scripts to upgrade to 0.8 from 0.7 2022-01-20 15:08:03 -08:00
Sriharsha Chintalapani
20e6c4bc86 fix upgrade 2022-01-20 15:08:03 -08:00
ulixius9
d594499fa0
ISSUE-1838: Added table type in glue connector (#2309) 
* ISSUE-1838: Added table type in glue connector

* Initialized table type with regular

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-01-20 07:57:44 -08:00
ulixius9
d6522e2ab4
ISSUE-2272: Ingestion: Roles and Teams apis called only when ingesting users (#2284) 
* ISSUE-2272: Ingestion: Roles and Teams apis called only when ingesting users

* reformatted metadata_rest.py

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-01-20 12:24:35 +05:30
Sriharsha Chintalapani
0a3ba3ddfd
Fix 2276: fix elastic search indexing (#2278) 2022-01-18 23:30:36 -08:00
Sriharsha Chintalapani
0d3ded0742
Data Profiler Integration (#2235) 
* Fix 2234: Data profiler integration
 2022-01-18 20:25:43 -08:00
Pere Miquel Brull
a9c840778a
[issue-2116] - Add target property to MlModel (#2134) 
* Add target property to MlModel

* Use pull_request_target for forks

* Simplify str

* Update ingestion-core setup

* Bump ingestion-core version

* Update install
 2022-01-18 16:27:35 +01:00
Sriharsha Chintalapani
b4983764e5
Fix 2232: Presto - connection formatting issues when username/password configured (#2233) 
* Fix 2232: Presto - connection formatting issues when username/password configured

* Fix 2232: Presto - connection formatting issues when username/password configured
 2022-01-17 11:03:17 +05:30
Matt
276128e8b0
Add MetadataOperations RBAC support for PATCH APIs (#2128) 
* Add RBAC for PATCH APIs

* Expand scope to all resources except a few (Policy, User, Role, Team, resources that doesn't support PATCH)

* Fix code smells
 2022-01-16 09:24:26 -08:00
Pere Miquel Brull
ebda556adc
[issue-2223] - Lineage Callback for any operator (#2225) 
* Fix logging

* Reorganise lineage

* Add callback

* Add metadata config

* Fix type smell
 2022-01-15 18:57:48 +01:00
Pere Miquel Brull
2728041d0f
[issue-2213] - Airflow lineage refactor (#2218) 
* Fix ometa logging

* Refactor airflow lineage

* Refactor airflow lineage

* Add airflow lineage integration tests

* Fix circular dependency
 2022-01-14 13:00:33 -08:00
Akash Jain
105d8c1b67
Feat: docker cmd improvements (#1895) 
* revamped docker commands

* update metadata docker commands in comments

* Added Docker Compose Check

* code refactored to docker.py

* fix py-checkstyle lints

* Adding clean functionality; refactor code

* pylint checkstyle fixes

* version string revised
 2022-01-13 20:39:46 -08:00
codingwithabhi
5714c4122b
Move sql queries (#2183) 
* sql-queries-moved-separate-file

* sql-queries-moved-separate-file

* redshift-and-neo4j-queries-added

* redshift-and-neo4j-queries-added

* neo4j-query-imported

* code-formatted

Co-authored-by: = <=>
 2022-01-13 14:16:54 +05:30
Sriharsha Chintalapani
c4d7c76198
Event publishers to ElasticSearch and Slack (#2118) 
* ElasticSearch & Slack Event publishers

* Fix #2165: Support for deleted entities in ElasticSearch
 2022-01-12 16:35:27 -08:00
Matt
f323afe0cd
Support role ingestion from sample data (#2167) 2022-01-11 16:51:18 -08:00
Ayush Shah
f379b35279
Reflection Cache Implementation (#2016) 
* Reflection Cache for Bigquery and Redshift

* Overrided few sqlalchemy packages

* Added Geography Support

* Reformatted files

* DBT models error handling implemented

* Geography type added as a custom sqlalchemy datatype

* GEOGRAPHY and VARIANT added as custom sql types

* Implemented file formatting using black

* Implemented file formatting using black
 2022-01-11 14:58:03 +05:30
Sriharsha Chintalapani
cf6f438531
Fix #1994: Add support for marking dataset entities as deleted (#2141) 
* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted
 2022-01-11 11:42:53 +05:30