1489 Commits

Author SHA1 Message Date
Tom Vijlbrief
20887c92dd
Map PostGIS spatial types to GEOGRAPHY (#1009)
* Map PostGIS spatial types to GEOGRAPHY

* Use engine.raw_connection()
2021-11-03 08:19:33 -07:00
Akash Jain
06b2f70e3e
kubernetes docker image compatibility for #1026 (#1042) 2021-11-03 08:18:52 -07:00
Pere Miquel Brull
87fd93a3d0
Add GET with fields param (#1033) 2021-11-02 13:04:55 -07:00
Pere Miquel Brull
11f9ddae3c
Issue 954 (#1020)
* Prepare mixins

* Remove tasks

* Prepare table mixin

* Clean tasks from sample_data

* Add table mixin tests

* Prepare compute percentile

* Refactor OpenMetadata
2021-11-01 08:57:53 -07:00
vijaypm
5d796172ce
ISSUE-960: (#988)
* ISSUE-960: fixed test cases and updated documented with list of pip install

* using requests package for testing http port responsiveness and socket for non-http ports

* use healthcheck api instead of just service endpoint

Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>
2021-11-01 08:32:34 -07:00
Tom Vijlbrief
0c23b68d40
Handle . in column names (#989)
* Handle . in column names

* Handle . for profiling
2021-11-01 07:22:40 -07:00
Sriharsha Chintalapani
9b7d3e313f
Fix #1004: Update Pipeline Ingestion and Indexing to accommodate new API changes (#1005) 2021-11-01 07:22:01 -07:00
AviG
f105f35e58
ISSUE-995: fix redash dashboard slug (#996)
Co-authored-by: user <user@users-MacBook-Pro.local>
2021-10-31 11:57:41 -07:00
Pere Miquel Brull
423e81ee68
Generate python models (#990)
* Mark internal functions

* Regenerate python models

* Prepare health check
2021-10-31 08:23:01 -07:00
Tom Vijlbrief
e6f6d9c2bb
Map SQL columns with unknown type to VARCHAR (#980)
Change pydantic version requirement
2021-10-30 21:07:39 -07:00
Ayush Shah
93921814af
Docker fix - latest release changes (#983)
* Docker fix - Architecture, MySQL

* Docker Airflow API Dagrun support

* Docker latest changes modified
2021-10-30 09:05:30 -07:00
Pere Miquel Brull
42db57e3b8
Issue 953 (#972)
* Rename docstring

* Fix typo

* Prepare linage API

* Centralise GET API

* Fix delete test by FQDN

* Add hints

* Add lineage tests

* Migrate lineage ops to OpenMetadata API
2021-10-28 11:31:38 -07:00
Sriharsha Chintalapani
7b09571a22
Profiler: Module to calculate measurements for a table (#933)
* Profiler: Module to calculate measurements for a table

* Profiler: refactor sql expressions

* Profiler: fix supported types

* Update README.md

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-10-27 08:31:10 +05:30
Pere Miquel Brull
710675d51a
OpenMetadata High Level API (#942)
* update arg name for fqdn

* fix suite setup and teardown

* Update list logic

* Add Chart tests

* Add Dashboard tests

* Add pipeline tests

* Add table tests

* Simplify create API

* Formatting tests

* Update create API

* Add Tasks and Topics tests

* Rename OpenMetadata

* Add client close

* Use Optional for GET id and name

* Fix #931 fetching dashboards using client

* Add tests

* Fix sample users ingestion

* Remove lineage test

* Rename services for independence
2021-10-26 14:18:43 -07:00
Sriharsha Chintalapani
bb7899638f
Fix #941: Fix Topic FQDN (#943) 2021-10-26 11:15:38 -07:00
Ayush Shah
759574a8de
Ingestion Optimization - Sample Users, Dockerfiles, Removal of Pandas (#935)
* Sample Profile Data for Sample Tables (#815)

* Sample Profile Data for Sample Tables

* Disabling Profile as Default

* Added Sample Profile Data to 3 additional sample tables

* Sample Tables fixed (#850)

* Pydantic fix, Docker update (#860)

* Setup.py Modified with openmetadata-airflow package, docker update

* Setup.py Modified

* Update setup.py

* Removed Pandas from Sample Data

* Sample Users added under sample data

* Sample User Standalone pipelines and modules removed

* Docker release package updated

* Dockerfile updated, removed redundant files

* Setup.py removed from ingestion src directory

* User Resource failing check resolved

* Modifying Usage Columns Datatype
2021-10-26 09:14:24 -07:00
parthp2107
7b05ad7161
changed elasticsearch username and password from required to optional (#922) 2021-10-25 10:25:49 +05:30
vijaypm
9edd824c3a
Issue 910 (#914)
* Issue 898 (#905)

* ISSUE-898: additional information in the prerequisities for building and running code

* ISSUE-898: removed unreachable old doc

* ISSUE-898: added new docker compose to expose MySQL and ES ports to host machines

* ISSUE-898: changed jdbc connect url to allow Public Key Retrieval

* ISSUE-898: fixed log name to openmetadata.log

Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>

* Fixes #906 Remove unused methods lingering from #899

* Update pull_request_template.md

* Update pull_request_template.md

* ISSUE-861: add elasticsearch username & password (#894)

* ISSUE-861: add elasticsearch username & password

* ISSUE-861:  python elasticsearch sink add username & password

* ISSUE-861: bugfix

* format code

* format code

* updated instructions to run integration tests

* fixed api call to metadata server; changed test to cover both database as well as table operations everytime

Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>
Co-authored-by: sureshms <suresh@getcollate.io>
Co-authored-by: Suresh Srinivas <srini30005@gmail.com>
Co-authored-by: rong fengliang <1141591465@qq.com>
2021-10-23 19:58:26 -07:00
rong fengliang
8e06fa25b6
ISSUE-861: add elasticsearch username & password (#894)
* ISSUE-861: add elasticsearch username & password

* ISSUE-861:  python elasticsearch sink add username & password

* ISSUE-861: bugfix

* format code

* format code
2021-10-23 10:03:24 -07:00
Ayush Shah
7bf6de0b40
Sample Usage modified as per Model (#889)
* Sample Usage modified as per Model

* Resolved - Named Parameters
2021-10-22 08:11:57 -07:00
Pere Miquel Brull
f91bcc03f6
[Issue-877] - High Level API (#890)
* First approach

* Test model

* Validate endpoints

* Update schema codegen

* Prepare reqs dev

* Rename titles and codegen

* Update README

* Keep lineage title as addLineage

* Update pydantic classes

* Update API wrapper & tests

* Handle services methods

* Prepare tests
2021-10-21 14:51:38 -07:00
Ayush Shah
13e78a74f3
Usage Pipeline fix (#883)
* Snowflake Usage query modified

* Snowflake and Bigquery Usage pipelines added

* Update bigquery_usage.py
2021-10-21 15:20:06 +05:30
Sriharsha Chintalapani
29c87f77d0
Improve Redshift Usage query (#881)
* Improve Redshift Usage query

* Improve Redshift Usage query

* Improve Redshift Usage query
2021-10-20 21:50:02 -07:00
Sriharsha Chintalapani
39a7b3e8c6
Fix #845: Fix FQN when the service is passed with Entity ID and Type only (#880) 2021-10-20 17:03:01 -07:00
Sriharsha Chintalapani
ee1ec15ef9
Fix #853: Ingestion: handle profiler returning None (#857)
* Fix #853: Ingestion: handle profiler returning None

* Fix #853: Ingestion: handle profiler returning None
2021-10-19 10:08:35 +05:30
Sriharsha Chintalapani
1db18a50cf
Issue-823: data profiler should have an option of configuring a profiler date (#831)
* Issue-823: data profiler should have an option of configuring a profiler date

* Fix error logging, remove stack trace

* Issue-823: data profiler should have an option of configuring a profiler date

* Issue-823: Fix formatting
2021-10-18 15:00:19 -07:00
Ayush Shah
9cf3a4045f
Sample Profile Data for Sample Tables (#815)
* Sample Profile Data for Sample Tables

* Disabling Profile as Default

* Added Sample Profile Data to 3 additional sample tables
2021-10-17 17:24:39 -07:00
Sriharsha Chintalapani
f43bf000cd
Issue-787: sql_source mark entities failed to ingest in warning (#796) 2021-10-16 22:50:36 +05:30
Pere Miquel Brull
3bf5999224
[Issue-759] - Model Entity Sample (#797)
* Fix typo

* Prepare sample data for Model Entity

* Clean print

* Prepare sample model data ingestion

* Allow fstring in logging for style unification

* Update sink status signature

* Update sink status signature

* Return Model in REST

* Update workflow status info for readability
2021-10-16 09:59:32 -07:00
Ayush Shah
0eb3e7b964
Changing Pydantic from 1.8.2 to 1.7.4 resolves the conflict (#788) 2021-10-16 07:58:55 -07:00
Pere Miquel Brull
1804de6d73
[Issue-760] - Precommit & isort (#772)
* Update configs

* Update configs

* Update precommit black exclude

* Update precommit exclude isort

* Test precommit

* Revert test commit

* isort & black

* Test precommit

* Revert test precommit

* Update docs

* Apply recipe update

* Fix black_check recipe

* Grammar
2021-10-14 15:22:59 -07:00
Ayush Shah
72e4b8831f
Removing Cron from all configs (#773)
* Removing Cron from all configs

* Removing Cron from Documentation
2021-10-14 15:21:47 -07:00
Pere Miquel Brull
9a362dafdc
[Issue 760] - Python tooling (#761)
* Update python tooling with black

* black formatting

* Change ci config for python
2021-10-14 07:48:42 -07:00
Ayush Shah
85b6b72848
Airflow docker (#762)
* Airflow Docker implementation - Ingestion

* Dockerfiles modified
2021-10-14 07:46:24 -07:00
Sriharsha Chintalapani
5644addc9a
Fix #748: Airflow lineage add inlet , outlet parsing to construct lineage (#749) 2021-10-13 09:33:15 +05:30
Sriharsha Chintalapani
c28665bca7
Sample lineage (#735)
* Fix #727: Add sample lineage data and ingestion support
2021-10-11 20:12:40 -07:00
Matt
78e97c81ca
Add exponentional backoff for ometa REST Client (#711)
Co-authored-by: Mithun Mathew <matt@Mithuns-MacBook-Pro.local>
2021-10-11 11:20:36 -07:00
Ayush Shah
323513755f
register_custom_type import path changed (#722) 2021-10-09 11:06:25 -07:00
Ayush Shah
319f7a795c
Docker local (#723)
* added docker file for local build and ingestion

* Docker Modified for local setup

* Ingestion script optimized

* Added script to directly run Local Docker

Co-authored-by: Parth Panchal <parthpanchal@Parths-MacBook-Air.local>
2021-10-09 10:58:33 -07:00
Ayush Shah
1650a4ef4a
Added support for struct in bigquery, pyhive and bigquery pypi packag… (#717)
* Added support for struct in bigquery, pyhive and bigquery pypi packages modified

* Versions added, Naming changed, Newlines removed
2021-10-09 07:15:41 -07:00
Ayush Shah
7ef490ed2d
Sample Data set to generate proper data for Currency column (#713) 2021-10-09 07:15:09 -07:00
Ayush Shah
a19c5bceca
Trino fix #705 (#706)
* Trino fix #705

* Update trino.json to point to default database
2021-10-07 12:25:03 -07:00
James
d455409cc9
issue-696: Added trino support for Openmetadata (#697)
* issue-696: Added trino support for Openmetadata

* issue-696: fixed linting issues

* issue-696: not mentioning Trino for now as it will be part of 0.5 release

Co-authored-by: jbuoncri <jbuoncri@cisco.com>
2021-10-07 11:15:34 -07:00
Sriharsha Chintalapani
e3cfb4dc65
For hive's complex data types parse raw type (#560)
* For hive's complex data types parse raw type

* Complex Data type logic modification

* Complex Data Type parsing implemented

* Raw Data type helper modification

* handling unnamed/anonymous struct

* Complex Nested structure implementation

* print statements removed and reverted to raw_data_type

* Complex Structure Array & MAP logic implemented

* Raw Data Type Logic revamped

* Redshift Integration

* MAP and UnionType support added

* Redshift Pypi package updated

* dataLength validationError fix

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-10-04 23:36:35 +05:30
Sriharsha Chintalapani
1361bace11
sample entity generation (#646)
* 1M entities to test scalability

* 1M entities to test scalability

* 1M entities test

* column name modification

* Fix service add and tags

* Add sample entity generation

Co-authored-by: parthp2107 <parth@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-10-04 22:56:28 +05:30
Sriharsha Chintalapani
37796c655b
Fix #650: Add support for pipelines in sample data & search integration (#651) 2021-10-04 17:48:45 +05:30
Sriharsha Chintalapani
08cefee56a
Fix #636: Pipeline/TaskEntity capture more metadata like start, end time, type etc.. (#637) 2021-10-02 17:12:18 -07:00
Sriharsha Chintalapani
ebfbd39cc2
Fix #624: Ingestion: Fix REST API calls to get service and create service (#625)
* Fix metadata usage listing of tables

* Fix #624: Ingestion: Fix REST API calls to get service and create service
2021-09-30 20:35:35 +05:30
Sriharsha Chintalapani
7e16e39ca0
Fix metadata usage listing of tables (#620) 2021-09-29 22:54:52 -07:00
Sriharsha Chintalapani
8f5c5021d7
Fix redshift complex types (#619) 2021-09-30 11:02:52 +05:30