432 Commits

Author SHA1 Message Date
Sriharsha Chintalapani
0d3ded0742
Data Profiler Integration (#2235)
* Fix 2234: Data profiler integration
2022-01-18 20:25:43 -08:00
Pere Miquel Brull
a9c840778a
[issue-2116] - Add target property to MlModel (#2134)
* Add target property to MlModel

* Use pull_request_target for forks

* Simplify str

* Update ingestion-core setup

* Bump ingestion-core version

* Update install
2022-01-18 16:27:35 +01:00
Pere Miquel Brull
2728041d0f
[issue-2213] - Airflow lineage refactor (#2218)
* Fix ometa logging

* Refactor airflow lineage

* Refactor airflow lineage

* Add airflow lineage integration tests

* Fix circular dependency
2022-01-14 13:00:33 -08:00
Pere Miquel Brull
178315d68a
[issue-2112] - Add helper to get EntityReference in OMeta API (#2114)
* Use entity list from mixin

* Add entity reference helper

* Add tests for retrieving the entity reference

* Add missing space

* Fix shadowing

* Use get entity ref

* Use get entity ref
2022-01-10 09:07:39 +01:00
Teddy
dbcc5e9dd2
ISSUE-1048: Add Version Mixin (#2018)
* Added entity version mixins logics to ometa API

* remove logging in line 385 used for testing

* Fixed black error + ran isort

* remove extra underscore in

* Added integration tests for OpenMetadata versions methods

* Fixed linting errors in versionMixin.py
2022-01-07 10:37:56 +01:00
Pere Miquel Brull
48ebcffbd0
[issue-1976] - Ingestion SonarCloud (#2085)
* Update path

* Prepare sonar properties

* Prepare coverage recipes

* Add coverage

* Simplify pytest

* Organise integration tests

* Update path

* Use setup instead of reqs

* Update recipes

* Fix PR event to target

* Update event_name

* Prepare sonar

* Run tests & sonar

* Use sonarcloud host

* Fix compose

* Use ingestion token
2022-01-07 10:28:38 +01:00
Pere Miquel Brull
eb01c55676
Update naming (#2080) 2022-01-06 14:42:50 +01:00
Pere Miquel Brull
d303be847e
Prepare MlModel lineage (#1879) 2021-12-21 17:44:29 -08:00
Pere Miquel Brull
a7c8056bf5
[issue-1079] - MlFlow Connector (#1677)
* Prepare infra

* Store experiment information in MySQL & MinIO

* Use CreateMlModelEntityRequest instead of MlModel for PUT operations

* Update MlFlow infra

* Prepare MlFlow source

* Prepare Mlflow workflow

* Simplify test and prepare README

* Revert compose

* Fix compose

* Prepare warnings and fix features

* Use non-default port for integration test

* Use mlflow-skinny for the client application
2021-12-11 18:40:19 -08:00
Ayush Shah
67433f559c
Trino JDBC updated (#1614) 2021-12-08 13:14:38 +05:30
Pere Miquel Brull
185ef5e98f
Get id as str or uuid (#1562) 2021-12-06 08:40:53 +01:00
parthp2107
a9e70e6a89
License header update (#1498)
* updated liscense header

* updated license header

* updated license header

* updated license header

* addressing stylecheck findings
2021-12-01 12:46:28 +05:30
omriAl
96e473b5cb
#1051: migrated all usages of models.MetadataUser & models.User classes to g… (#1473)
* migrated all usages of models.MetadataUser & models.User classes to generated.schema.teams.user.User classes instead.

* formatted file with black

Co-authored-by: omri.alon <omri.alon@myheritage.com>
2021-11-30 07:57:07 -08:00
Ayush Shah
267a3d7aa2 Removed Platform from docker-compose.yml files, emulation fix 2021-11-26 18:10:27 +01:00
Pere Miquel Brull
ea79b47c45
[Issue-1268] - MlModel generated sources (#1288)
* Update ML model naming for generated python src

* Update MlModel id and title

* Update MlModel descriptions

* Generate pydantic models

* Update OpenMetadata with new MlModel naming

* Update MlModel naming

* Update MlModel naming

* Update MlModel namings for backend sources
2021-11-20 08:26:45 -08:00
Pere Miquel Brull
ffd7818978
[Issue-1099] - Update MLModel Ingestion (#1179)
* Refactor MLModel

* Prepare MLModel properties example

* Update API tags to MLModels

* Update API descriptions for MLModel

* Rename tags to mlModels
2021-11-15 07:58:15 -08:00
vijaypm
d5758a0fbb
Issue 1023 (#1088)
* fixed bug that was setting to lowercase property names that were already in camelCase

* added datetime prefix to allow integration tests to be run repeatedly

* fixed test to use the ldap entries from ldap service in docker to test the service

* updated instructions for ldap test

* added comments to clarify the fix done in snake_to_camel method

Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>
2021-11-06 16:42:23 -07:00
Pere Miquel Brull
bc45c33f4d
[Issue-998] - Deprecate OpenMetadataAPIClient (#1052)
* Refactor OpenMetadata and remove unused task functionality

* Refactor OpenMetadata in helpers

* Refactor OpenMetadata in sample_data helpers

* Remove unused client

* Remove unused client

* Remove unused client

* Align fields signature

* Refactor source OpenMetadata

* Refactor metadata_rest sink OpenMetadata

* Add list tags

* Refactor PII OpenMetadata

* Refactor LDAP REST users OpenMetadata

* isort imports

* Fix metadata to ES pipeline

* Refactor sample_entity OpenMetadata

* Remove helpers test - covered by OMeta testing

* Refactor workflow_test OpenMetadata

* Refactor mysql integration test for OpenMetadata

* Remove tasks as their refactor to pipelines

* Refactor Hive integration tests for OpenMetadata

* Remove OpenMetadataAPIClient

* Remove OpenMetadataAPIClient
2021-11-03 13:02:34 -07:00
Pere Miquel Brull
87fd93a3d0
Add GET with fields param (#1033) 2021-11-02 13:04:55 -07:00
Pere Miquel Brull
11f9ddae3c
Issue 954 (#1020)
* Prepare mixins

* Remove tasks

* Prepare table mixin

* Clean tasks from sample_data

* Add table mixin tests

* Prepare compute percentile

* Refactor OpenMetadata
2021-11-01 08:57:53 -07:00
vijaypm
5d796172ce
ISSUE-960: (#988)
* ISSUE-960: fixed test cases and updated documented with list of pip install

* using requests package for testing http port responsiveness and socket for non-http ports

* use healthcheck api instead of just service endpoint

Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>
2021-11-01 08:32:34 -07:00
Pere Miquel Brull
423e81ee68
Generate python models (#990)
* Mark internal functions

* Regenerate python models

* Prepare health check
2021-10-31 08:23:01 -07:00
Pere Miquel Brull
42db57e3b8
Issue 953 (#972)
* Rename docstring

* Fix typo

* Prepare linage API

* Centralise GET API

* Fix delete test by FQDN

* Add hints

* Add lineage tests

* Migrate lineage ops to OpenMetadata API
2021-10-28 11:31:38 -07:00
Pere Miquel Brull
710675d51a
OpenMetadata High Level API (#942)
* update arg name for fqdn

* fix suite setup and teardown

* Update list logic

* Add Chart tests

* Add Dashboard tests

* Add pipeline tests

* Add table tests

* Simplify create API

* Formatting tests

* Update create API

* Add Tasks and Topics tests

* Rename OpenMetadata

* Add client close

* Use Optional for GET id and name

* Fix #931 fetching dashboards using client

* Add tests

* Fix sample users ingestion

* Remove lineage test

* Rename services for independence
2021-10-26 14:18:43 -07:00
vijaypm
9edd824c3a
Issue 910 (#914)
* Issue 898 (#905)

* ISSUE-898: additional information in the prerequisities for building and running code

* ISSUE-898: removed unreachable old doc

* ISSUE-898: added new docker compose to expose MySQL and ES ports to host machines

* ISSUE-898: changed jdbc connect url to allow Public Key Retrieval

* ISSUE-898: fixed log name to openmetadata.log

Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>

* Fixes #906 Remove unused methods lingering from #899

* Update pull_request_template.md

* Update pull_request_template.md

* ISSUE-861: add elasticsearch username & password (#894)

* ISSUE-861: add elasticsearch username & password

* ISSUE-861:  python elasticsearch sink add username & password

* ISSUE-861: bugfix

* format code

* format code

* updated instructions to run integration tests

* fixed api call to metadata server; changed test to cover both database as well as table operations everytime

Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>
Co-authored-by: sureshms <suresh@getcollate.io>
Co-authored-by: Suresh Srinivas <srini30005@gmail.com>
Co-authored-by: rong fengliang <1141591465@qq.com>
2021-10-23 19:58:26 -07:00
Pere Miquel Brull
f91bcc03f6
[Issue-877] - High Level API (#890)
* First approach

* Test model

* Validate endpoints

* Update schema codegen

* Prepare reqs dev

* Rename titles and codegen

* Update README

* Keep lineage title as addLineage

* Update pydantic classes

* Update API wrapper & tests

* Handle services methods

* Prepare tests
2021-10-21 14:51:38 -07:00
Sriharsha Chintalapani
e3cfb4dc65
For hive's complex data types parse raw type (#560)
* For hive's complex data types parse raw type

* Complex Data type logic modification

* Complex Data Type parsing implemented

* Raw Data type helper modification

* handling unnamed/anonymous struct

* Complex Nested structure implementation

* print statements removed and reverted to raw_data_type

* Complex Structure Array & MAP logic implemented

* Raw Data Type Logic revamped

* Redshift Integration

* MAP and UnionType support added

* Redshift Pypi package updated

* dataLength validationError fix

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-10-04 23:36:35 +05:30
Ayush Shah
72a355bd5d Ingestion Setup.py modification 2021-08-13 01:40:56 +05:30
Ayush Shah
754813d220 connection string abstract method 2021-08-11 23:49:00 +05:30
Ayush Shah
d9551ed337
Fix #15 (#16)
* Fix #15

* Pylint Relative Imports handled

* Final Newline lint error resolved

* Pylint Unused Imports handled

* Pylint command modified

* AllowDenyPattern error resolved

* EntityReference changes for common to entityReference

* vscode settings removed

* Resolved reoccuring function

* pylintrc file implemented

* Fix pylintrc file

* Pylint Errors resolved + import errors resolved and includepattern included

Co-authored-by: Suresh Srinivas <srini3005@gmail.com>
2021-08-07 11:00:35 +05:30
parthp2107
b5baa9f2db applied license to the backend files 2021-08-02 15:08:30 +05:30
sureshms
8ce7c879e8 OpenMetadata snapshot release 0.3 2021-08-01 14:27:44 -07:00