374 Commits

Author SHA1 Message Date
omriAl
96e473b5cb
#1051: migrated all usages of models.MetadataUser & models.User classes to g… (#1473)
* migrated all usages of models.MetadataUser & models.User classes to generated.schema.teams.user.User classes instead.

* formatted file with black

Co-authored-by: omri.alon <omri.alon@myheritage.com>
2021-11-30 07:57:07 -08:00
Ayush Shah
267a3d7aa2 Removed Platform from docker-compose.yml files, emulation fix 2021-11-26 18:10:27 +01:00
Pere Miquel Brull
ea79b47c45
[Issue-1268] - MlModel generated sources (#1288)
* Update ML model naming for generated python src

* Update MlModel id and title

* Update MlModel descriptions

* Generate pydantic models

* Update OpenMetadata with new MlModel naming

* Update MlModel naming

* Update MlModel naming

* Update MlModel namings for backend sources
2021-11-20 08:26:45 -08:00
pmbrull
c53e5d0577 Remove deprecated import 2021-11-18 20:22:05 +01:00
Pere Miquel Brull
ffd7818978
[Issue-1099] - Update MLModel Ingestion (#1179)
* Refactor MLModel

* Prepare MLModel properties example

* Update API tags to MLModels

* Update API descriptions for MLModel

* Rename tags to mlModels
2021-11-15 07:58:15 -08:00
vijaypm
d14755b4e1
Issue - 1091 (#1096)
* fixed issue with not being able to load config file; cleaned up unused code

* fixed test case to test configuration error

Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>
2021-11-07 11:19:06 -08:00
vijaypm
d5758a0fbb
Issue 1023 (#1088)
* fixed bug that was setting to lowercase property names that were already in camelCase

* added datetime prefix to allow integration tests to be run repeatedly

* fixed test to use the ldap entries from ldap service in docker to test the service

* updated instructions for ldap test

* added comments to clarify the fix done in snake_to_camel method

Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>
2021-11-06 16:42:23 -07:00
Pere Miquel Brull
bc45c33f4d
[Issue-998] - Deprecate OpenMetadataAPIClient (#1052)
* Refactor OpenMetadata and remove unused task functionality

* Refactor OpenMetadata in helpers

* Refactor OpenMetadata in sample_data helpers

* Remove unused client

* Remove unused client

* Remove unused client

* Align fields signature

* Refactor source OpenMetadata

* Refactor metadata_rest sink OpenMetadata

* Add list tags

* Refactor PII OpenMetadata

* Refactor LDAP REST users OpenMetadata

* isort imports

* Fix metadata to ES pipeline

* Refactor sample_entity OpenMetadata

* Remove helpers test - covered by OMeta testing

* Refactor workflow_test OpenMetadata

* Refactor mysql integration test for OpenMetadata

* Remove tasks as their refactor to pipelines

* Refactor Hive integration tests for OpenMetadata

* Remove OpenMetadataAPIClient

* Remove OpenMetadataAPIClient
2021-11-03 13:02:34 -07:00
Pere Miquel Brull
87fd93a3d0
Add GET with fields param (#1033) 2021-11-02 13:04:55 -07:00
Pere Miquel Brull
11f9ddae3c
Issue 954 (#1020)
* Prepare mixins

* Remove tasks

* Prepare table mixin

* Clean tasks from sample_data

* Add table mixin tests

* Prepare compute percentile

* Refactor OpenMetadata
2021-11-01 08:57:53 -07:00
vijaypm
5d796172ce
ISSUE-960: (#988)
* ISSUE-960: fixed test cases and updated documented with list of pip install

* using requests package for testing http port responsiveness and socket for non-http ports

* use healthcheck api instead of just service endpoint

Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>
2021-11-01 08:32:34 -07:00
Pere Miquel Brull
423e81ee68
Generate python models (#990)
* Mark internal functions

* Regenerate python models

* Prepare health check
2021-10-31 08:23:01 -07:00
Pere Miquel Brull
42db57e3b8
Issue 953 (#972)
* Rename docstring

* Fix typo

* Prepare linage API

* Centralise GET API

* Fix delete test by FQDN

* Add hints

* Add lineage tests

* Migrate lineage ops to OpenMetadata API
2021-10-28 11:31:38 -07:00
Pere Miquel Brull
710675d51a
OpenMetadata High Level API (#942)
* update arg name for fqdn

* fix suite setup and teardown

* Update list logic

* Add Chart tests

* Add Dashboard tests

* Add pipeline tests

* Add table tests

* Simplify create API

* Formatting tests

* Update create API

* Add Tasks and Topics tests

* Rename OpenMetadata

* Add client close

* Use Optional for GET id and name

* Fix #931 fetching dashboards using client

* Add tests

* Fix sample users ingestion

* Remove lineage test

* Rename services for independence
2021-10-26 14:18:43 -07:00
vijaypm
9edd824c3a
Issue 910 (#914)
* Issue 898 (#905)

* ISSUE-898: additional information in the prerequisities for building and running code

* ISSUE-898: removed unreachable old doc

* ISSUE-898: added new docker compose to expose MySQL and ES ports to host machines

* ISSUE-898: changed jdbc connect url to allow Public Key Retrieval

* ISSUE-898: fixed log name to openmetadata.log

Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>

* Fixes #906 Remove unused methods lingering from #899

* Update pull_request_template.md

* Update pull_request_template.md

* ISSUE-861: add elasticsearch username & password (#894)

* ISSUE-861: add elasticsearch username & password

* ISSUE-861:  python elasticsearch sink add username & password

* ISSUE-861: bugfix

* format code

* format code

* updated instructions to run integration tests

* fixed api call to metadata server; changed test to cover both database as well as table operations everytime

Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>
Co-authored-by: sureshms <suresh@getcollate.io>
Co-authored-by: Suresh Srinivas <srini30005@gmail.com>
Co-authored-by: rong fengliang <1141591465@qq.com>
2021-10-23 19:58:26 -07:00
Pere Miquel Brull
f91bcc03f6
[Issue-877] - High Level API (#890)
* First approach

* Test model

* Validate endpoints

* Update schema codegen

* Prepare reqs dev

* Rename titles and codegen

* Update README

* Keep lineage title as addLineage

* Update pydantic classes

* Update API wrapper & tests

* Handle services methods

* Prepare tests
2021-10-21 14:51:38 -07:00
Ayush Shah
72e4b8831f
Removing Cron from all configs (#773)
* Removing Cron from all configs

* Removing Cron from Documentation
2021-10-14 15:21:47 -07:00
Sriharsha Chintalapani
e3cfb4dc65
For hive's complex data types parse raw type (#560)
* For hive's complex data types parse raw type

* Complex Data type logic modification

* Complex Data Type parsing implemented

* Raw Data type helper modification

* handling unnamed/anonymous struct

* Complex Nested structure implementation

* print statements removed and reverted to raw_data_type

* Complex Structure Array & MAP logic implemented

* Raw Data Type Logic revamped

* Redshift Integration

* MAP and UnionType support added

* Redshift Pypi package updated

* dataLength validationError fix

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-10-04 23:36:35 +05:30
parthp2107
191ecb7332
added unit tests to helper and workflow classes in ingestion (#391)
* added unit tests to helper and workflow classes in ingestion

* Addressing comments

* Adressing comments

* Adressing comments

* Failing Unit tests fixed

* Workflow test and config file modified

Co-authored-by: parthp2107 <parth.panchal@deuexsoultions.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-09-12 01:07:13 +05:30
Ayush Shah
72a355bd5d Ingestion Setup.py modification 2021-08-13 01:40:56 +05:30
Ayush Shah
754813d220 connection string abstract method 2021-08-11 23:49:00 +05:30
Ayush Shah
d9551ed337
Fix #15 (#16)
* Fix #15

* Pylint Relative Imports handled

* Final Newline lint error resolved

* Pylint Unused Imports handled

* Pylint command modified

* AllowDenyPattern error resolved

* EntityReference changes for common to entityReference

* vscode settings removed

* Resolved reoccuring function

* pylintrc file implemented

* Fix pylintrc file

* Pylint Errors resolved + import errors resolved and includepattern included

Co-authored-by: Suresh Srinivas <srini3005@gmail.com>
2021-08-07 11:00:35 +05:30
parthp2107
b5baa9f2db applied license to the backend files 2021-08-02 15:08:30 +05:30
sureshms
8ce7c879e8 OpenMetadata snapshot release 0.3 2021-08-01 14:27:44 -07:00