yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-11-20 21:06:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
14,040 Commits 910 Branches 181 Tags
Commit Graph

2979 Commits

Author SHA1 Message Date
Pere Miquel Brull
1eb3f1ad41
[issue-2322] - Backup CLI (#2349) 
* Prepare cli module

* Check call

* Prepare backup

* Run pytest on schema changes

* Remove traces

* Add backup plugin

* run generate during CI

* Install after generate

* Address logger hotspot
 2022-01-22 12:08:14 -08:00
ulixius9
8fca53ec52
ISSUE-2048: Bigquery data profiler uses schema name instead of project id (#2335) 
* ISSUE-2048: Bigquery data profiler uses schema name instead of project id

* Update bigquery.py

* Update bigquery.py

* Update profiler_runner.py

* Update profiler.py

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-22 23:33:15 +05:30
Sriharsha Chintalapani
f390bac375
Fix 2270: Column Type Parser (#2271) 
* Fix 2270: Column Type Parser

* Fix 2270: Column Type Parser

* Added checks to allow arrayDataType and dataTypeDisplay

* Modified - arrayDataType and dataTypeDisplay

* Update sql_source.py

* Update sql_source.py

* file formatted

* Modified according to column_type_parser.py

* modified / refactored / deleted the files

* Added Tests, modified sql_source

* file formatted

* Added missing datatypes

* Added Tests

* Added Tests

* Added Tests - refactored expected output into a json file

* file formatted

* Sample Data Updated

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-22 23:32:58 +05:30
Sriharsha Chintalapani
10e71a7e09
Fix #2340: Ingestion pipeline deployment refactor (#2341) 
* Fix #2340: Ingestion pipeline deployment refactor
 2022-01-21 22:06:14 -08:00
codingwithabhi
ddf4ae261b
Bigquery credential configuration updated (#2041) 
* bigquery-credentials-in-bigquery-json

* ui-changes-reverted

* credentials-moved-to-options

* Update bigquery.json

* Update bigquery.py

* delete-temp-code-added

* delete-temp-code-added

* removed-init-from-config

* Update datatypes_test.py

* Update column_helpers.py

* code-formatted

Co-authored-by: Abhishek <abhishek@Abhisheks-MacBook-Air.local>
Co-authored-by: = <=>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-01-21 17:23:02 +05:30
Matt
2a80caa8fe
Add logic to initialize relationships from seed data during application startup (#2307) 
* Add logic to initialize relationships from seed data during application startup

* Remove ingestion related code for access control policies

* Move PolicyEvaluator init to PolicyResource
 2022-01-20 22:18:03 -08:00
Sriharsha Chintalapani
18f5c288fe Fix #2320: Fix DB migration scripts to upgrade to 0.8 from 0.7 2022-01-20 15:08:03 -08:00
Sriharsha Chintalapani
20e6c4bc86 fix upgrade 2022-01-20 15:08:03 -08:00
ulixius9
d594499fa0
ISSUE-1838: Added table type in glue connector (#2309) 
* ISSUE-1838: Added table type in glue connector

* Initialized table type with regular

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-01-20 07:57:44 -08:00
ulixius9
d6522e2ab4
ISSUE-2272: Ingestion: Roles and Teams apis called only when ingesting users (#2284) 
* ISSUE-2272: Ingestion: Roles and Teams apis called only when ingesting users

* reformatted metadata_rest.py

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
 2022-01-20 12:24:35 +05:30
Sriharsha Chintalapani
0a3ba3ddfd
Fix 2276: fix elastic search indexing (#2278) 2022-01-18 23:30:36 -08:00
Sriharsha Chintalapani
0d3ded0742
Data Profiler Integration (#2235) 
* Fix 2234: Data profiler integration
 2022-01-18 20:25:43 -08:00
Sriharsha Chintalapani
b4983764e5
Fix 2232: Presto - connection formatting issues when username/password configured (#2233) 
* Fix 2232: Presto - connection formatting issues when username/password configured

* Fix 2232: Presto - connection formatting issues when username/password configured
 2022-01-17 11:03:17 +05:30
Pere Miquel Brull
ebda556adc
[issue-2223] - Lineage Callback for any operator (#2225) 
* Fix logging

* Reorganise lineage

* Add callback

* Add metadata config

* Fix type smell
 2022-01-15 18:57:48 +01:00
Pere Miquel Brull
2728041d0f
[issue-2213] - Airflow lineage refactor (#2218) 
* Fix ometa logging

* Refactor airflow lineage

* Refactor airflow lineage

* Add airflow lineage integration tests

* Fix circular dependency
 2022-01-14 13:00:33 -08:00
Akash Jain
105d8c1b67
Feat: docker cmd improvements (#1895) 
* revamped docker commands

* update metadata docker commands in comments

* Added Docker Compose Check

* code refactored to docker.py

* fix py-checkstyle lints

* Adding clean functionality; refactor code

* pylint checkstyle fixes

* version string revised
 2022-01-13 20:39:46 -08:00
codingwithabhi
5714c4122b
Move sql queries (#2183) 
* sql-queries-moved-separate-file

* sql-queries-moved-separate-file

* redshift-and-neo4j-queries-added

* redshift-and-neo4j-queries-added

* neo4j-query-imported

* code-formatted

Co-authored-by: = <=>
 2022-01-13 14:16:54 +05:30
Sriharsha Chintalapani
c4d7c76198
Event publishers to ElasticSearch and Slack (#2118) 
* ElasticSearch & Slack Event publishers

* Fix #2165: Support for deleted entities in ElasticSearch
 2022-01-12 16:35:27 -08:00
Matt
f323afe0cd
Support role ingestion from sample data (#2167) 2022-01-11 16:51:18 -08:00
Ayush Shah
f379b35279
Reflection Cache Implementation (#2016) 
* Reflection Cache for Bigquery and Redshift

* Overrided few sqlalchemy packages

* Added Geography Support

* Reformatted files

* DBT models error handling implemented

* Geography type added as a custom sqlalchemy datatype

* GEOGRAPHY and VARIANT added as custom sql types

* Implemented file formatting using black

* Implemented file formatting using black
 2022-01-11 14:58:03 +05:30
Sriharsha Chintalapani
cf6f438531
Fix #1994: Add support for marking dataset entities as deleted (#2141) 
* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted
 2022-01-11 11:42:53 +05:30
Pere Miquel Brull
d3b6c7cf27
[issue-1973] - Python API from Sklearn to MlModel (#2119) 
* Move staticmethods to utils

* Use functions from utils

* Convert sklearn to MlModel

* merge main
 2022-01-10 09:36:08 +01:00
Pere Miquel Brull
178315d68a
[issue-2112] - Add helper to get EntityReference in OMeta API (#2114) 
* Use entity list from mixin

* Add entity reference helper

* Add tests for retrieving the entity reference

* Add missing space

* Fix shadowing

* Use get entity ref

* Use get entity ref
 2022-01-10 09:07:39 +01:00
Pere Miquel Brull
74e9038b92
[issue-997] - More pylint fixes (#2107) 
* Update naming and disable false positives

* Rename for naming convention

* Fix linting issues

* Moved pylint to pylintrc

* Moved pylint to pylintrc

* Rename CI

* Check sonar properties

* Fix pull_request_target checkout
 2022-01-07 19:50:53 +01:00
Teddy
dbcc5e9dd2
ISSUE-1048: Add Version Mixin (#2018) 
* Added entity version mixins logics to ometa API

* remove logging in line 385 used for testing

* Fixed black error + ran isort

* remove extra underscore in

* Added integration tests for OpenMetadata versions methods

* Fixed linting errors in versionMixin.py
 2022-01-07 10:37:56 +01:00
Alberto Miorin
c8de136cca
First part of #2091: Python code in branch main is always formatted correctly (#2093) 2022-01-07 06:17:48 +01:00
Ayush Shah
860f0ae8db
Iterable from Collections not working (#2019) 2022-01-04 06:48:18 -08:00
Ayush Shah
19e5f737ff
Usage fix - removed service_name from bulk_sink (#1956) 
* Query Usage fixed

* Usage updated

* Reformatted change

* Usage files reformatted
 2022-01-04 17:12:10 +05:30
Matt
ab4c9ede25
Add ingestion for access_control_policies to support RBAC (#1969) 2021-12-30 08:25:09 -08:00
Pere Miquel Brull
1e334af89c
Fix linting (#1958) 2021-12-29 17:33:40 +01:00
Sriharsha Chintalapani
30eeec498f
Fix #1952: Airflow Openmetadata lineage allow config to be read from … (#1953) 
* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable
 2021-12-29 11:31:18 +05:30
Matt
a2b1d96636
Address code smells within ingestion source (#1951) 2021-12-28 17:40:55 -08:00
Ayush Shah
431f47fb3c
Kafka and Oracle issues fixed (#1917) 
* Kafka and Oracle issues fixed

* Kafka failure status updated
 2021-12-27 08:04:56 -08:00
Matt
9ff0fdc533
Refactor Glue ingestion to use common config and client (#1925) 
- Use AWSClientConfigModel and AWSClient for glue.py
- Use helpers.get_storage_service_or_create for s3.py
 2021-12-26 20:27:08 -08:00
Matt
f7653f4122
Add support for GCS Lifecycle Policy Ingestion (#1923) 
- Add example gcs.json
- Add support for MoveAction and DeleteAction
 2021-12-26 20:26:33 -08:00
Matt
4588121520
Add support for DeleteAction in S3 Lifecycle Policy (#1916) 
- Support Delete Action for S3
- Add Example s3.json
- Create AWSClient util
- Use AWSClient util in S3 ingestion source
- Remove ambiguity in policy filters by removing array with different types
 2021-12-25 16:29:10 -08:00
Matt
f1a8a7886e
Link Lifecycle Policy to specific Location (#1720) 
- Amend ingestion to support Location reference from Policy
- Amend PolicyResource and PolicyRepository to accommodate Location reference
 2021-12-23 11:06:15 -08:00
Alberto Miorin
a9e2df8993
Fix #1821: PyHive registers the trino:// schema too (#1835) 2021-12-22 20:46:01 +01:00
Pere Miquel Brull
d303be847e
Prepare MlModel lineage (#1879) 2021-12-21 17:44:29 -08:00
Sriharsha Chintalapani
485fdea7b1
Fix #1854: Support DBT 1.0 format (#1855) 
* Fix #1854: Support DBT 1.0 format
 2021-12-20 12:25:26 -08:00
Sriharsha Chintalapani
3c254440a5
Fix #1841: Presto Connector issues (#1842) 2021-12-18 23:27:51 -08:00
Pere Miquel Brull
ab620e95af
[issue-1750] - Expandvars transforming values (#1830) 
* Prepare tests for basic config cases

* Add test cases

* Use os expandvars

* Add missing commas

* Add missing commas

* Remove dataclasses backport
 2021-12-18 15:35:12 -08:00
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup (#1730) 
* Fix typo

* Clean setup

* Update ingestion local image to be barebone on connector dependencies

* Prepare ingestion connectors base image

* Add system dependencies

* Prepare docker CLI

* Add docker provider

* Prepare entrypoint for the image

* Remove DBT pipeline as per Issue 1658

* Add TODO for ingestion build

* Bind docker socket

* Update comment

* Update README

* Use DockerOperator in sample data

* Build images with latest tag

* Prepare symlink to pass the volume to the DockerOperator

* Update README

* Prepare Base image for CI

* COPY multiple files into dir

* COPY multiple files into dir

* Remove DBT source as is now part of table ingestion

* Build docker base in run_local_docker
 2021-12-18 16:41:38 +01:00
Carsten Steckel
1bd1f99a47
fix for #1796: hard-coded /tmp folder removed (#1804) 
* fix for #1796

* fix for #1796

* missed some merge trouble artifacts

* reformat to pass black py-code-style-check
 2021-12-17 18:59:17 +05:30
Ayush Shah
be1c4d4833
Stopping Docker containers and network with container name and not project name (#1819) 2021-12-17 18:58:54 +05:30
Sriharsha Chintalapani
df6232ef52
Fix #1808: check for None in description (#1809) 2021-12-16 21:15:42 +01:00
Ayush Shah
d71c75aaa1
Update snowflake.py (#1803) 
* Update snowflake.py

* snowflake needs database and warehouse
 2021-12-16 10:46:49 -08:00
Ayush Shah
4a0efe041f
Added connect_args in sql_source (#1791) 2021-12-16 12:33:20 +05:30
Ayush Shah
06712fa425
Metabase and Snowflake Fixed (#1788) 
* Metabase and Snowflake Fixed

* Removed DBT
 2021-12-16 11:57:43 +05:30
Ayush Shah
07924f80c3
Added Metabase Connector (#1726) 
* Added Metabase Connector

* Metabase Added support for lineage

* Modified Metabase as per comments

* Metabase filter patterns added

* Update databaseService.json
 2021-12-15 09:53:55 -08:00