yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-08-17 13:36:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
10,673 Commits 728 Branches 153 Tags
Commit Graph

2916 Commits

Author SHA1 Message Date
Sriharsha Chintalapani
39a7b3e8c6
Fix #845: Fix FQN when the service is passed with Entity ID and Type only (#880) 2021-10-20 17:03:01 -07:00
Sriharsha Chintalapani
ee1ec15ef9
Fix #853: Ingestion: handle profiler returning None (#857) 
* Fix #853: Ingestion: handle profiler returning None

* Fix #853: Ingestion: handle profiler returning None
 2021-10-19 10:08:35 +05:30
Sriharsha Chintalapani
1db18a50cf
Issue-823: data profiler should have an option of configuring a profiler date (#831) 
* Issue-823: data profiler should have an option of configuring a profiler date

* Fix error logging, remove stack trace

* Issue-823: data profiler should have an option of configuring a profiler date

* Issue-823: Fix formatting
 2021-10-18 15:00:19 -07:00
Ayush Shah
9cf3a4045f
Sample Profile Data for Sample Tables (#815) 
* Sample Profile Data for Sample Tables

* Disabling Profile as Default

* Added Sample Profile Data to 3 additional sample tables
 2021-10-17 17:24:39 -07:00
Sriharsha Chintalapani
f43bf000cd
Issue-787: sql_source mark entities failed to ingest in warning (#796) 2021-10-16 22:50:36 +05:30
Pere Miquel Brull
3bf5999224
[Issue-759] - Model Entity Sample (#797) 
* Fix typo

* Prepare sample data for Model Entity

* Clean print

* Prepare sample model data ingestion

* Allow fstring in logging for style unification

* Update sink status signature

* Update sink status signature

* Return Model in REST

* Update workflow status info for readability
 2021-10-16 09:59:32 -07:00
Ayush Shah
0eb3e7b964
Changing Pydantic from 1.8.2 to 1.7.4 resolves the conflict (#788) 2021-10-16 07:58:55 -07:00
Pere Miquel Brull
1804de6d73
[Issue-760] - Precommit & isort (#772) 
* Update configs

* Update configs

* Update precommit black exclude

* Update precommit exclude isort

* Test precommit

* Revert test commit

* isort & black

* Test precommit

* Revert test precommit

* Update docs

* Apply recipe update

* Fix black_check recipe

* Grammar
 2021-10-14 15:22:59 -07:00
Ayush Shah
72e4b8831f
Removing Cron from all configs (#773) 
* Removing Cron from all configs

* Removing Cron from Documentation
 2021-10-14 15:21:47 -07:00
Pere Miquel Brull
9a362dafdc
[Issue 760] - Python tooling (#761) 
* Update python tooling with black

* black formatting

* Change ci config for python
 2021-10-14 07:48:42 -07:00
Ayush Shah
85b6b72848
Airflow docker (#762) 
* Airflow Docker implementation - Ingestion

* Dockerfiles modified
 2021-10-14 07:46:24 -07:00
Sriharsha Chintalapani
5644addc9a
Fix #748: Airflow lineage add inlet , outlet parsing to construct lineage (#749) 2021-10-13 09:33:15 +05:30
Sriharsha Chintalapani
c28665bca7
Sample lineage (#735) 
* Fix #727: Add sample lineage data and ingestion support
 2021-10-11 20:12:40 -07:00
Matt
78e97c81ca
Add exponentional backoff for ometa REST Client (#711) 
Co-authored-by: Mithun Mathew <matt@Mithuns-MacBook-Pro.local>
 2021-10-11 11:20:36 -07:00
Ayush Shah
323513755f
register_custom_type import path changed (#722) 2021-10-09 11:06:25 -07:00
Ayush Shah
319f7a795c
Docker local (#723) 
* added docker file for local build and ingestion

* Docker Modified for local setup

* Ingestion script optimized

* Added script to directly run Local Docker

Co-authored-by: Parth Panchal <parthpanchal@Parths-MacBook-Air.local>
 2021-10-09 10:58:33 -07:00
Ayush Shah
1650a4ef4a
Added support for struct in bigquery, pyhive and bigquery pypi packag… (#717) 
* Added support for struct in bigquery, pyhive and bigquery pypi packages modified

* Versions added, Naming changed, Newlines removed
 2021-10-09 07:15:41 -07:00
Ayush Shah
7ef490ed2d
Sample Data set to generate proper data for Currency column (#713) 2021-10-09 07:15:09 -07:00
Ayush Shah
a19c5bceca
Trino fix #705 (#706) 
* Trino fix #705

* Update trino.json to point to default database
 2021-10-07 12:25:03 -07:00
James
d455409cc9
issue-696: Added trino support for Openmetadata (#697) 
* issue-696: Added trino support for Openmetadata

* issue-696: fixed linting issues

* issue-696: not mentioning Trino for now as it will be part of 0.5 release

Co-authored-by: jbuoncri <jbuoncri@cisco.com>
 2021-10-07 11:15:34 -07:00
Sriharsha Chintalapani
e3cfb4dc65
For hive's complex data types parse raw type (#560) 
* For hive's complex data types parse raw type

* Complex Data type logic modification

* Complex Data Type parsing implemented

* Raw Data type helper modification

* handling unnamed/anonymous struct

* Complex Nested structure implementation

* print statements removed and reverted to raw_data_type

* Complex Structure Array & MAP logic implemented

* Raw Data Type Logic revamped

* Redshift Integration

* MAP and UnionType support added

* Redshift Pypi package updated

* dataLength validationError fix

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-10-04 23:36:35 +05:30
Sriharsha Chintalapani
1361bace11
sample entity generation (#646) 
* 1M entities to test scalability

* 1M entities to test scalability

* 1M entities test

* column name modification

* Fix service add and tags

* Add sample entity generation

Co-authored-by: parthp2107 <parth@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-10-04 22:56:28 +05:30
Sriharsha Chintalapani
37796c655b
Fix #650: Add support for pipelines in sample data & search integration (#651) 2021-10-04 17:48:45 +05:30
Sriharsha Chintalapani
08cefee56a
Fix #636: Pipeline/TaskEntity capture more metadata like start, end time, type etc.. (#637) 2021-10-02 17:12:18 -07:00
Sriharsha Chintalapani
ebfbd39cc2
Fix #624: Ingestion: Fix REST API calls to get service and create service (#625) 
* Fix metadata usage listing of tables

* Fix #624: Ingestion: Fix REST API calls to get service and create service
 2021-09-30 20:35:35 +05:30
Sriharsha Chintalapani
7e16e39ca0
Fix metadata usage listing of tables (#620) 2021-09-29 22:54:52 -07:00
Sriharsha Chintalapani
8f5c5021d7
Fix redshift complex types (#619) 2021-09-30 11:02:52 +05:30
Sriharsha Chintalapani
e84de1a310
Fix #611: ElasticSearch matches are case-sensitive (#617) 2021-09-29 17:43:09 -07:00
Julia Valenti
a514934f16
Fixes docs that reference messaging service instead of dashboard or pipeline (#613) (#614) 2021-09-29 16:21:31 -07:00
Sriharsha Chintalapani
bfec0bfbed
Ingestion: Airflow integration to ingest metadata about pipelines and tasks (#609) 
* [WIP] Airlfow integration

* [WIP] Airlfow integration

* [WIP] airflow integration

* [WIP] Airflow

* [WIP] Airflow

* Fix #608: Ingestion: Airflow integration to ingest metadata about pipelines and tasks

* Fix #608: Ingestion: Airflow integration to ingest metadata about pipelines and tasks

* Update DashboardServiceResource.java

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-09-29 11:32:09 -07:00
Sriharsha Chintalapani
f7da8045b4
Fix #605: Ingestion: metadata list apis should paginate (#606) 
* Fix #605: Ingestion: metadata list apis should paginate

* Fix #605: Ingestion: metadata list apis should paginate, Addressing review comments
 2021-09-29 00:06:08 +05:30
Sriharsha Chintalapani
0983ec7f1f
Add nested types to sample data (#601) 
* Add nested types to sample data

* Add nested types to sample data
 2021-09-28 12:23:34 +05:30
Sriharsha Chintalapani
3d10347aec
Fix #596: Update ingestion to new OpenMetadata column entity changes (#597) 2021-09-27 23:11:19 +05:30
Sriharsha Chintalapani
eb2717b0e3
Fix #587: Ingestion: Add standalone report process to generate datasets, usage & profile and serve from standalone server (#588) 
* Fix #587: Ingestion: Add standalone report process to generate datasets, usage & profile and serve from standalone server

* add localhost
 2021-09-27 08:43:38 -07:00
Ayush Shah
627481f181
Status record Json encoding bug fixed and pandas not found fixed (#584) 2021-09-25 13:54:04 -07:00
Sriharsha Chintalapani
745ae0c253
Fix #577: Users API should support put op (#578) 
* Fix #577: Users API should support put op
 2021-09-24 17:55:26 -07:00
parthp2107
06810cdec1
Fix #432:Added Redash Connector (#444) 
* added redash connector

* added redash connector

* Added Redash Connector

* minor changes

Co-authored-by: parthp2107 <parth.panchal@deuexsoultions.com>
Co-authored-by: parthp2107 <parth@getcollate.io>
 2021-09-22 15:09:24 +05:30
Ayush Shah
7bc2b14178
Added and Modified Redshift Pipeline (#528) 
* Pylint build failure fixed

* Redshift workflow added
 2021-09-19 21:16:50 -07:00
Ayush Shah
7652baa00d
Setup.py Refactored, ES port fix (#521) 
* Pylint build failure fixed

* Setup & dependency modified, Data profiler default to False, ES port fix

* Profiler requirements refactored

* Setup.py requirement fix

* openmetadata-ingestion version upgrade
 2021-09-19 13:59:14 +05:30
Sriharsha Chintalapani
4c6c8fd446
Fix #515: Ingestion: Add ES configuration to allow port (#516) 2021-09-17 08:57:41 -07:00
Sriharsha Chintalapani
8c103bd2ad
Profiler (#496) 
* profiler code

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-09-15 15:49:26 -07:00
Ayush Shah
9b04d781e0
Fixed Bigquery to support data profiling (#494) 2021-09-14 22:07:51 -07:00
Sriharsha Chintalapani
20a98aca81
Fix #396: Elastic search is not indexing usage for dashboard (#492) 2021-09-15 09:47:14 +05:30
Sriharsha Chintalapani
b7adb5dc6b
Fix #469: Add Vertica Connector (#470) 2021-09-12 21:59:31 -07:00
parthp2107
191ecb7332
added unit tests to helper and workflow classes in ingestion (#391) 
* added unit tests to helper and workflow classes in ingestion

* Addressing comments

* Adressing comments

* Adressing comments

* Failing Unit tests fixed

* Workflow test and config file modified

Co-authored-by: parthp2107 <parth.panchal@deuexsoultions.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-09-12 01:07:13 +05:30
Ayush Shah
d2df40cf2b
Fix #355: Tableau Implemented (#468) 
* Fix #355: Tableau Implemented

* Tableau pipeline location modification
 2021-09-11 11:46:10 -07:00
Sriharsha Chintalapani
ed698dc40f
Fix #466: Fix airflow example dag (#467) 2021-09-10 22:28:23 -07:00
Sriharsha Chintalapani
1c80dc246e
Fix #456: Make PII-Processor optional and independent install (#457) 2021-09-10 10:41:19 +05:30
Suresh Srinivas
30c1e5a14c
Merge pull request #447 from open-metadata/Issue-446 
Fix #446: Add DataProfiler to ingestion and APIs
 2021-09-09 11:25:56 -07:00
Sriharsha Chintalapani
4998d49167 Fix #446: Add DataProfiler to ingestion and APIs 2021-09-09 10:51:07 -07:00