2826 Commits

Author SHA1 Message Date
Pere Miquel Brull
178315d68a
[issue-2112] - Add helper to get EntityReference in OMeta API (#2114)
* Use entity list from mixin

* Add entity reference helper

* Add tests for retrieving the entity reference

* Add missing space

* Fix shadowing

* Use get entity ref

* Use get entity ref
2022-01-10 09:07:39 +01:00
Pere Miquel Brull
74e9038b92
[issue-997] - More pylint fixes (#2107)
* Update naming and disable false positives

* Rename for naming convention

* Fix linting issues

* Moved pylint to pylintrc

* Moved pylint to pylintrc

* Rename CI

* Check sonar properties

* Fix pull_request_target checkout
2022-01-07 19:50:53 +01:00
Teddy
dbcc5e9dd2
ISSUE-1048: Add Version Mixin (#2018)
* Added entity version mixins logics to ometa API

* remove logging in line 385 used for testing

* Fixed black error + ran isort

* remove extra underscore in

* Added integration tests for OpenMetadata versions methods

* Fixed linting errors in versionMixin.py
2022-01-07 10:37:56 +01:00
Pere Miquel Brull
48ebcffbd0
[issue-1976] - Ingestion SonarCloud (#2085)
* Update path

* Prepare sonar properties

* Prepare coverage recipes

* Add coverage

* Simplify pytest

* Organise integration tests

* Update path

* Use setup instead of reqs

* Update recipes

* Fix PR event to target

* Update event_name

* Prepare sonar

* Run tests & sonar

* Use sonarcloud host

* Fix compose

* Use ingestion token
2022-01-07 10:28:38 +01:00
Alberto Miorin
c8de136cca
First part of #2091: Python code in branch main is always formatted correctly (#2093) 2022-01-07 06:17:48 +01:00
Alberto Miorin
34a9d31240
Fix #2052: Revert the reverted pre-commit (#2053) 2022-01-07 03:53:21 +01:00
Pere Miquel Brull
eb01c55676
Update naming (#2080) 2022-01-06 14:42:50 +01:00
Ayush Shah
860f0ae8db
Iterable from Collections not working (#2019) 2022-01-04 06:48:18 -08:00
Ayush Shah
19e5f737ff
Usage fix - removed service_name from bulk_sink (#1956)
* Query Usage fixed

* Usage updated

* Reformatted change

* Usage files reformatted
2022-01-04 17:12:10 +05:30
Akash Jain
348e23ca6f
fix/ci-workflows (#1997)
* add pytest and pytest-cov

* add timeout; update make command usages
2022-01-04 12:49:22 +05:30
Ayush Shah
6175053dd4
Sample Topics fixed (#2005) 2022-01-03 11:00:04 -08:00
Matt
ab4c9ede25
Add ingestion for access_control_policies to support RBAC (#1969) 2021-12-30 08:25:09 -08:00
Ayush Shah
5d6f385a75
Added Filter Params for Table and Schema (#1954)
* Added Filter Params for table and Schema

* Bigquery Doc changes

* Doc Changes for databases

* Filter Pattern Changes

* Table Filter Pattern Example Changes

* Filter Pattern Example Changes
2021-12-29 09:13:09 -08:00
Pere Miquel Brull
1e334af89c
Fix linting (#1958) 2021-12-29 17:33:40 +01:00
Sriharsha Chintalapani
30eeec498f
Fix #1952: Airflow Openmetadata lineage allow config to be read from … (#1953)
* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable

* Fix #1952: Airflow Openmetadata lineage allow config to be read from env variable
2021-12-29 11:31:18 +05:30
Matt
a2b1d96636
Address code smells within ingestion source (#1951) 2021-12-28 17:40:55 -08:00
Ayush Shah
431f47fb3c
Kafka and Oracle issues fixed (#1917)
* Kafka and Oracle issues fixed

* Kafka failure status updated
2021-12-27 08:04:56 -08:00
Matt
9ff0fdc533
Refactor Glue ingestion to use common config and client (#1925)
- Use AWSClientConfigModel and AWSClient for glue.py
- Use helpers.get_storage_service_or_create for s3.py
2021-12-26 20:27:08 -08:00
Matt
f7653f4122
Add support for GCS Lifecycle Policy Ingestion (#1923)
- Add example gcs.json
- Add support for MoveAction and DeleteAction
2021-12-26 20:26:33 -08:00
Matt
4588121520
Add support for DeleteAction in S3 Lifecycle Policy (#1916)
- Support Delete Action for S3
- Add Example s3.json
- Create AWSClient util
- Use AWSClient util in S3 ingestion source
- Remove ambiguity in policy filters by removing array with different types
2021-12-25 16:29:10 -08:00
Sriharsha Chintalapani
bb847bc004
Revert "Use defaults of google java format action (#1845)" (#1912)
This reverts commit 782209ead58f4cd82882872e9f8f40779d8b6e54.
2021-12-23 19:22:47 -08:00
Matt
f1a8a7886e
Link Lifecycle Policy to specific Location (#1720)
- Amend ingestion to support Location reference from Policy
- Amend PolicyResource and PolicyRepository to accommodate Location reference
2021-12-23 11:06:15 -08:00
Alberto Miorin
a9e2df8993
Fix #1821: PyHive registers the trino:// schema too (#1835) 2021-12-22 20:46:01 +01:00
Pere Miquel Brull
1f01081698
Clean DBT Model (#1888) 2021-12-22 17:17:05 +01:00
Pere Miquel Brull
d303be847e
Prepare MlModel lineage (#1879) 2021-12-21 17:44:29 -08:00
Alberto Miorin
782209ead5
Use defaults of google java format action (#1845)
* Use defaults of google java format action

* Fix the first Java file

* Add lint-staged

* Remove husky
2021-12-21 17:51:18 +01:00
Alberto Miorin
c081eabb7e
Fix #1868: Add openmetadata-ingestion-core as dependency (#1869) 2021-12-21 17:41:20 +01:00
Sriharsha Chintalapani
7111267669
Update version to 0.8 and ingestion to 0.6 (#1859) 2021-12-20 22:02:59 -08:00
Alberto Miorin
ec1e3a2d61
Fix #1344: Remove requirements.txt (#1588) 2021-12-20 21:36:23 -08:00
Sriharsha Chintalapani
485fdea7b1
Fix #1854: Support DBT 1.0 format (#1855)
* Fix #1854: Support DBT 1.0 format
2021-12-20 12:25:26 -08:00
Sriharsha Chintalapani
3c254440a5
Fix #1841: Presto Connector issues (#1842) 2021-12-18 23:27:51 -08:00
Pere Miquel Brull
ab620e95af
[issue-1750] - Expandvars transforming values (#1830)
* Prepare tests for basic config cases

* Add test cases

* Use os expandvars

* Add missing commas

* Add missing commas

* Remove dataclasses backport
2021-12-18 15:35:12 -08:00
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup (#1730)
* Fix typo

* Clean setup

* Update ingestion local image to be barebone on connector dependencies

* Prepare ingestion connectors base image

* Add system dependencies

* Prepare docker CLI

* Add docker provider

* Prepare entrypoint for the image

* Remove DBT pipeline as per Issue 1658

* Add TODO for ingestion build

* Bind docker socket

* Update comment

* Update README

* Use DockerOperator in sample data

* Build images with latest tag

* Prepare symlink to pass the volume to the DockerOperator

* Update README

* Prepare Base image for CI

* COPY multiple files into dir

* COPY multiple files into dir

* Remove DBT source as is now part of table ingestion

* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
Carsten Steckel
1bd1f99a47
fix for #1796: hard-coded /tmp folder removed (#1804)
* fix for #1796

* fix for #1796

* missed some merge trouble artifacts

* reformat to pass black py-code-style-check
2021-12-17 18:59:17 +05:30
Ayush Shah
be1c4d4833
Stopping Docker containers and network with container name and not project name (#1819) 2021-12-17 18:58:54 +05:30
Sriharsha Chintalapani
df6232ef52
Fix #1808: check for None in description (#1809) 2021-12-16 21:15:42 +01:00
Ayush Shah
d71c75aaa1
Update snowflake.py (#1803)
* Update snowflake.py

* snowflake needs database and warehouse
2021-12-16 10:46:49 -08:00
Ayush Shah
4a0efe041f
Added connect_args in sql_source (#1791) 2021-12-16 12:33:20 +05:30
Ayush Shah
06712fa425
Metabase and Snowflake Fixed (#1788)
* Metabase and Snowflake Fixed

* Removed DBT
2021-12-16 11:57:43 +05:30
Ayush Shah
07924f80c3
Added Metabase Connector (#1726)
* Added Metabase Connector

* Metabase Added support for lineage

* Modified Metabase as per comments

* Metabase filter patterns added

* Update databaseService.json
2021-12-15 09:53:55 -08:00
Ayush Shah
90618342cb
Team Description added (#1776) 2021-12-15 09:53:41 -08:00
Sriharsha Chintalapani
92a8ecd0ed
Issue-1762: User profile updates should be only allowed by admin or bots (#1765)
* Issue-1762: User profile updates should be only allowed by admin or bots
2021-12-15 01:10:44 -08:00
Alberto Miorin
5c56d479d8
Fix #1749: Location is missing some fields (#1751) 2021-12-14 11:55:13 -08:00
Sriharsha Chintalapani
5a8c70bbe9
Fix #1747: Ingestion: parse connection config method (#1748)
* Fix #1747:  Ingestion: parse connection config method

* Fix #1747:  Ingestion: parse connection config method

* Fix #1747:  Ingestion: parse connection config method
2021-12-15 00:58:57 +05:30
Sriharsha Chintalapani
c6ef94cac7
Fix #1733: Ingestion: ElasticSearch index update should use changeLog… (#1736)
* Fix #1733: Ingestion: ElasticSearch index update should use changeLog timestamp
2021-12-14 09:49:25 -08:00
Sriharsha Chintalapani
e3d1a95d2c
Fix #1737: Add separate filter patterns for database/schema and tables (#1739)
* Fix #1737: Add separate filter patterns for database/schema and tables
2021-12-14 08:50:05 -08:00
Alberto Miorin
93276079ec
Fix #1729: Connector Trino works with username. (#1732)
Implementation details
I have decided to rename schema_name to database and make it mandatory. Without database there is an error while scanning all available tables. The connector doesn't support multiple databases at the moment. It has to be tested with passwords. Trino requires SSL if you use passwords. It has to be tested with impersonation. I have removed quote_plus because I don't think it's needed.

- [x] Support username
- [ ] There is an integration test
- [ ] Support impersonation
- [ ] Support passwords
- [ ] Support tokens
- [ ] Support multiple databases
2021-12-13 22:05:28 +01:00
Ayush Shah
14982501c6
Snowflake Version Upgrade and Connection string change (#1707) 2021-12-12 08:26:15 -08:00
Sriharsha Chintalapani
a9198c72f2
Fix #1690: Ingestion: Apply DBT models to views (#1691)
* Fix #1690: Ingestion: Apply DBT models to views

* Fix #1690: Ingestion: Apply DBT models to views
2021-12-12 20:53:08 +05:30
Pere Miquel Brull
a7c8056bf5
[issue-1079] - MlFlow Connector (#1677)
* Prepare infra

* Store experiment information in MySQL & MinIO

* Use CreateMlModelEntityRequest instead of MlModel for PUT operations

* Update MlFlow infra

* Prepare MlFlow source

* Prepare Mlflow workflow

* Simplify test and prepare README

* Revert compose

* Fix compose

* Prepare warnings and fix features

* Use non-default port for integration test

* Use mlflow-skinny for the client application
2021-12-11 18:40:19 -08:00