2535 Commits

Author SHA1 Message Date
Ayush Shah
b689c2d632
Multiple fixes - Looker, Ingestion fix, pyarrow dependency (#2385)
* Added pyarrow dependency

* Fix #2378

* Fix Looker

* Fix Looker, secretstr to str
2022-01-24 12:29:54 -08:00
Pere Miquel Brull
05c8d1ebb6
Fix #2153 - Lineage date parsing (#2387)
* Check for start_date

* Fix date keys and remove repr

* Add date management tests
2022-01-24 12:29:16 -08:00
Pere Miquel Brull
6a8fc9e0fb
import on call (#2382) 2022-01-24 12:29:03 -08:00
Pere Miquel Brull
b39b698eb7
Fix #2345 - Make bump & unify (#2359)
* Add installing instructions

* Unify makefile and automate bump version setup

* Update submake call

* Mark targets as phony

* Add help to recipes

* Update core publish

* Do not trigger install from echo

* Run make from root

* Prepare py_format from isort and black

* Run py_format

* Check black and isort formatting

* Update comment msg
2022-01-23 09:47:06 -08:00
Ayush Shah
6f5a4eafe7
Bigquery support for env without passing any config (#2160)
* Bigquery support for env without passing any config

* Bigquery File formatted
2022-01-22 22:25:48 -08:00
Sriharsha Chintalapani
e5fdf77d8e
Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service (#2357)
* Fix #2340: Ingestion pipeline deployment refactor - update ingestion

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service
2022-01-22 21:12:25 -08:00
Pere Miquel Brull
1eb3f1ad41
[issue-2322] - Backup CLI (#2349)
* Prepare cli module

* Check call

* Prepare backup

* Run pytest on schema changes

* Remove traces

* Add backup plugin

* run generate during CI

* Install after generate

* Address logger hotspot
2022-01-22 12:08:14 -08:00
ulixius9
8fca53ec52
ISSUE-2048: Bigquery data profiler uses schema name instead of project id (#2335)
* ISSUE-2048: Bigquery data profiler uses schema name instead of project id

* Update bigquery.py

* Update bigquery.py

* Update profiler_runner.py

* Update profiler.py

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-22 23:33:15 +05:30
Sriharsha Chintalapani
f390bac375
Fix 2270: Column Type Parser (#2271)
* Fix 2270: Column Type Parser

* Fix 2270: Column Type Parser

* Added checks to allow arrayDataType and dataTypeDisplay

* Modified - arrayDataType and dataTypeDisplay

* Update sql_source.py

* Update sql_source.py

* file formatted

* Modified according to column_type_parser.py

* modified / refactored / deleted the files

* Added Tests, modified sql_source

* file formatted

* Added missing datatypes

* Added Tests

* Added Tests

* Added Tests - refactored expected output into a json file

* file formatted

* Sample Data Updated

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-22 23:32:58 +05:30
Sriharsha Chintalapani
10e71a7e09
Fix #2340: Ingestion pipeline deployment refactor (#2341)
* Fix #2340: Ingestion pipeline deployment refactor
2022-01-21 22:06:14 -08:00
codingwithabhi
ddf4ae261b
Bigquery credential configuration updated (#2041)
* bigquery-credentials-in-bigquery-json

* ui-changes-reverted

* credentials-moved-to-options

* Update bigquery.json

* Update bigquery.py

* delete-temp-code-added

* delete-temp-code-added

* removed-init-from-config

* Update datatypes_test.py

* Update column_helpers.py

* code-formatted

Co-authored-by: Abhishek <abhishek@Abhisheks-MacBook-Air.local>
Co-authored-by: = <=>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-21 17:23:02 +05:30
Matt
2a80caa8fe
Add logic to initialize relationships from seed data during application startup (#2307)
* Add logic to initialize relationships from seed data during application startup

* Remove ingestion related code for access control policies

* Move PolicyEvaluator init to PolicyResource
2022-01-20 22:18:03 -08:00
Sriharsha Chintalapani
18f5c288fe Fix #2320: Fix DB migration scripts to upgrade to 0.8 from 0.7 2022-01-20 15:08:03 -08:00
Sriharsha Chintalapani
20e6c4bc86 fix upgrade 2022-01-20 15:08:03 -08:00
ulixius9
d594499fa0
ISSUE-1838: Added table type in glue connector (#2309)
* ISSUE-1838: Added table type in glue connector

* Initialized table type with regular

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-01-20 07:57:44 -08:00
ulixius9
d6522e2ab4
ISSUE-2272: Ingestion: Roles and Teams apis called only when ingesting users (#2284)
* ISSUE-2272: Ingestion: Roles and Teams apis called only when ingesting users

* reformatted metadata_rest.py

Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-01-20 12:24:35 +05:30
Sriharsha Chintalapani
0a3ba3ddfd
Fix 2276: fix elastic search indexing (#2278) 2022-01-18 23:30:36 -08:00
Sriharsha Chintalapani
0d3ded0742
Data Profiler Integration (#2235)
* Fix 2234: Data profiler integration
2022-01-18 20:25:43 -08:00
Sriharsha Chintalapani
b4983764e5
Fix 2232: Presto - connection formatting issues when username/password configured (#2233)
* Fix 2232: Presto - connection formatting issues when username/password configured

* Fix 2232: Presto - connection formatting issues when username/password configured
2022-01-17 11:03:17 +05:30
Pere Miquel Brull
ebda556adc
[issue-2223] - Lineage Callback for any operator (#2225)
* Fix logging

* Reorganise lineage

* Add callback

* Add metadata config

* Fix type smell
2022-01-15 18:57:48 +01:00
Pere Miquel Brull
2728041d0f
[issue-2213] - Airflow lineage refactor (#2218)
* Fix ometa logging

* Refactor airflow lineage

* Refactor airflow lineage

* Add airflow lineage integration tests

* Fix circular dependency
2022-01-14 13:00:33 -08:00
Akash Jain
105d8c1b67
Feat: docker cmd improvements (#1895)
* revamped docker commands

* update metadata docker commands in comments

* Added Docker Compose Check

* code refactored to docker.py

* fix py-checkstyle lints

* Adding clean functionality; refactor code

* pylint checkstyle fixes

* version string revised
2022-01-13 20:39:46 -08:00
codingwithabhi
5714c4122b
Move sql queries (#2183)
* sql-queries-moved-separate-file

* sql-queries-moved-separate-file

* redshift-and-neo4j-queries-added

* redshift-and-neo4j-queries-added

* neo4j-query-imported

* code-formatted

Co-authored-by: = <=>
2022-01-13 14:16:54 +05:30
Sriharsha Chintalapani
c4d7c76198
Event publishers to ElasticSearch and Slack (#2118)
* ElasticSearch & Slack Event publishers

* Fix #2165: Support for deleted entities in ElasticSearch
2022-01-12 16:35:27 -08:00
Matt
f323afe0cd
Support role ingestion from sample data (#2167) 2022-01-11 16:51:18 -08:00
Ayush Shah
f379b35279
Reflection Cache Implementation (#2016)
* Reflection Cache for Bigquery and Redshift

* Overrided few sqlalchemy packages

* Added Geography Support

* Reformatted files

* DBT models error handling implemented

* Geography type added as a custom sqlalchemy datatype

* GEOGRAPHY and VARIANT added as custom sql types

* Implemented file formatting using black

* Implemented file formatting using black
2022-01-11 14:58:03 +05:30
Sriharsha Chintalapani
cf6f438531
Fix #1994: Add support for marking dataset entities as deleted (#2141)
* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted
2022-01-11 11:42:53 +05:30
Pere Miquel Brull
d3b6c7cf27
[issue-1973] - Python API from Sklearn to MlModel (#2119)
* Move staticmethods to utils

* Use functions from utils

* Convert sklearn to MlModel

* merge main
2022-01-10 09:36:08 +01:00
Pere Miquel Brull
178315d68a
[issue-2112] - Add helper to get EntityReference in OMeta API (#2114)
* Use entity list from mixin

* Add entity reference helper

* Add tests for retrieving the entity reference

* Add missing space

* Fix shadowing

* Use get entity ref

* Use get entity ref
2022-01-10 09:07:39 +01:00
Pere Miquel Brull
74e9038b92
[issue-997] - More pylint fixes (#2107)
* Update naming and disable false positives

* Rename for naming convention

* Fix linting issues

* Moved pylint to pylintrc

* Moved pylint to pylintrc

* Rename CI

* Check sonar properties

* Fix pull_request_target checkout
2022-01-07 19:50:53 +01:00
Teddy
dbcc5e9dd2
ISSUE-1048: Add Version Mixin (#2018)
* Added entity version mixins logics to ometa API

* remove logging in line 385 used for testing

* Fixed black error + ran isort

* remove extra underscore in

* Added integration tests for OpenMetadata versions methods

* Fixed linting errors in versionMixin.py
2022-01-07 10:37:56 +01:00
Alberto Miorin
c8de136cca
First part of #2091: Python code in branch main is always formatted correctly (#2093) 2022-01-07 06:17:48 +01:00
Ayush Shah
860f0ae8db
Iterable from Collections not working (#2019) 2022-01-04 06:48:18 -08:00
Ayush Shah
19e5f737ff
Usage fix - removed service_name from bulk_sink (#1956)
* Query Usage fixed

* Usage updated

* Reformatted change

* Usage files reformatted
2022-01-04 17:12:10 +05:30
Matt
ab4c9ede25
Add ingestion for access_control_policies to support RBAC (#1969) 2021-12-30 08:25:09 -08:00
Pere Miquel Brull
1e334af89c
Fix linting (#1958) 2021-12-29 17:33:40 +01:00
Sriharsha Chintalapani
30eeec498f
Fix #1952: Airflow Openmetadata lineage allow config to be read from … (#1953)
* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable
2021-12-29 11:31:18 +05:30
Matt
a2b1d96636
Address code smells within ingestion source (#1951) 2021-12-28 17:40:55 -08:00
Ayush Shah
431f47fb3c
Kafka and Oracle issues fixed (#1917)
* Kafka and Oracle issues fixed

* Kafka failure status updated
2021-12-27 08:04:56 -08:00
Matt
9ff0fdc533
Refactor Glue ingestion to use common config and client (#1925)
- Use AWSClientConfigModel and AWSClient for glue.py
- Use helpers.get_storage_service_or_create for s3.py
2021-12-26 20:27:08 -08:00
Matt
f7653f4122
Add support for GCS Lifecycle Policy Ingestion (#1923)
- Add example gcs.json
- Add support for MoveAction and DeleteAction
2021-12-26 20:26:33 -08:00
Matt
4588121520
Add support for DeleteAction in S3 Lifecycle Policy (#1916)
- Support Delete Action for S3
- Add Example s3.json
- Create AWSClient util
- Use AWSClient util in S3 ingestion source
- Remove ambiguity in policy filters by removing array with different types
2021-12-25 16:29:10 -08:00
Matt
f1a8a7886e
Link Lifecycle Policy to specific Location (#1720)
- Amend ingestion to support Location reference from Policy
- Amend PolicyResource and PolicyRepository to accommodate Location reference
2021-12-23 11:06:15 -08:00
Alberto Miorin
a9e2df8993
Fix #1821: PyHive registers the trino:// schema too (#1835) 2021-12-22 20:46:01 +01:00
Pere Miquel Brull
d303be847e
Prepare MlModel lineage (#1879) 2021-12-21 17:44:29 -08:00
Sriharsha Chintalapani
485fdea7b1
Fix #1854: Support DBT 1.0 format (#1855)
* Fix #1854: Support DBT 1.0 format
2021-12-20 12:25:26 -08:00
Sriharsha Chintalapani
3c254440a5
Fix #1841: Presto Connector issues (#1842) 2021-12-18 23:27:51 -08:00
Pere Miquel Brull
ab620e95af
[issue-1750] - Expandvars transforming values (#1830)
* Prepare tests for basic config cases

* Add test cases

* Use os expandvars

* Add missing commas

* Add missing commas

* Remove dataclasses backport
2021-12-18 15:35:12 -08:00
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup (#1730)
* Fix typo

* Clean setup

* Update ingestion local image to be barebone on connector dependencies

* Prepare ingestion connectors base image

* Add system dependencies

* Prepare docker CLI

* Add docker provider

* Prepare entrypoint for the image

* Remove DBT pipeline as per Issue 1658

* Add TODO for ingestion build

* Bind docker socket

* Update comment

* Update README

* Use DockerOperator in sample data

* Build images with latest tag

* Prepare symlink to pass the volume to the DockerOperator

* Update README

* Prepare Base image for CI

* COPY multiple files into dir

* COPY multiple files into dir

* Remove DBT source as is now part of table ingestion

* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
Carsten Steckel
1bd1f99a47
fix for #1796: hard-coded /tmp folder removed (#1804)
* fix for #1796

* fix for #1796

* missed some merge trouble artifacts

* reformat to pass black py-code-style-check
2021-12-17 18:59:17 +05:30