yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-10-31 02:29:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,829 Commits 865 Branches 177 Tags
Commit Graph

238 Commits

Author SHA1 Message Date
Ayush Shah
219246b78e
Glue pagination added (#1282) 2021-11-20 12:46:18 -08:00
Sriharsha Chintalapani
f3054658f5
Fix #968: Add DBT Connector (#1200) 2021-11-16 01:02:45 -08:00
Sriharsha Chintalapani
f018664e0c
Issue-1197: Add MariaDB Connector (#1198) 2021-11-16 00:25:24 -08:00
parthp2107
4a5f5e69b3
uncommenting add-user test (#1097) 
* uncommenting add-user test

* quick-fix

* Ingestion script modified

* Airflow sample_usage modified, added sleep in curl commands

* adding chrome options for failing tests

* addressing reviewdog finding

* testing

* addressing reviewdog findings

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-11-14 20:16:39 -08:00
Ayush Shah
5dc3bb9297
Docker Support from Python added (#1158) 
* Script modified - supports running from different locations

* Docker support from Python CLI

* Docker plugin setup.py

* Paths for latest and local dockers updated

* Resolved Comments - Docker CLI optimized, timestamp added

* help attribute added to options

* Docker clean code refactored
 2021-11-12 10:30:28 -08:00
Ayush Shah
fa3c9b1758
Reduced the sample-users to 100 (#1156) 2021-11-11 07:31:03 -08:00
Ayush Shah
c95f1542eb
Docker fix (#1101) 
* Docker Fix, User Ingestion Fix

* Docker file optimized

* ES docker version rolledback

* Healthcheck removed - incompatible with 3.9

* Docker & Airflow Config files updated with - resolved comments

* Openmetadata managed Airflow apis support added

* Dockerfile updated

* Docker file Updated with release package

* Elasticsearch allocated memory increased
 2021-11-10 21:22:32 -08:00
Ayush Shah
9839191242
Issue 483 - Glue Implementation (#1124) 
* Glue Tables and Pipeline workflows implemented

* Glue Config Added

* Relative imports changed to absolute

* Resolving Comments - Changed Imports, serviceTypes

* Type fixed in setup.py
 2021-11-10 07:28:13 -08:00
Akash Jain
06b2f70e3e
kubernetes docker image compatibility for #1026 (#1042) 2021-11-03 08:18:52 -07:00
Sriharsha Chintalapani
9b7d3e313f
Fix #1004: Update Pipeline Ingestion and Indexing to accommodate new API changes (#1005) 2021-11-01 07:22:01 -07:00
Ayush Shah
93921814af
Docker fix - latest release changes (#983) 
* Docker fix - Architecture, MySQL

* Docker Airflow API Dagrun support

* Docker latest changes modified
 2021-10-30 09:05:30 -07:00
Pere Miquel Brull
710675d51a
OpenMetadata High Level API (#942) 
* update arg name for fqdn

* fix suite setup and teardown

* Update list logic

* Add Chart tests

* Add Dashboard tests

* Add pipeline tests

* Add table tests

* Simplify create API

* Formatting tests

* Update create API

* Add Tasks and Topics tests

* Rename OpenMetadata

* Add client close

* Use Optional for GET id and name

* Fix #931 fetching dashboards using client

* Add tests

* Fix sample users ingestion

* Remove lineage test

* Rename services for independence
 2021-10-26 14:18:43 -07:00
Ayush Shah
759574a8de
Ingestion Optimization - Sample Users, Dockerfiles, Removal of Pandas (#935) 
* Sample Profile Data for Sample Tables (#815)

* Sample Profile Data for Sample Tables

* Disabling Profile as Default

* Added Sample Profile Data to 3 additional sample tables

* Sample Tables fixed (#850)

* Pydantic fix, Docker update (#860)

* Setup.py Modified with openmetadata-airflow package, docker update

* Setup.py Modified

* Update setup.py

* Removed Pandas from Sample Data

* Sample Users added under sample data

* Sample User Standalone pipelines and modules removed

* Docker release package updated

* Dockerfile updated, removed redundant files

* Setup.py removed from ingestion src directory

* User Resource failing check resolved

* Modifying Usage Columns Datatype
 2021-10-26 09:14:24 -07:00
Ayush Shah
9cf3a4045f
Sample Profile Data for Sample Tables (#815) 
* Sample Profile Data for Sample Tables

* Disabling Profile as Default

* Added Sample Profile Data to 3 additional sample tables
 2021-10-17 17:24:39 -07:00
Pere Miquel Brull
3bf5999224
[Issue-759] - Model Entity Sample (#797) 
* Fix typo

* Prepare sample data for Model Entity

* Clean print

* Prepare sample model data ingestion

* Allow fstring in logging for style unification

* Update sink status signature

* Update sink status signature

* Return Model in REST

* Update workflow status info for readability
 2021-10-16 09:59:32 -07:00
Ayush Shah
72e4b8831f
Removing Cron from all configs (#773) 
* Removing Cron from all configs

* Removing Cron from Documentation
 2021-10-14 15:21:47 -07:00
Ayush Shah
85b6b72848
Airflow docker (#762) 
* Airflow Docker implementation - Ingestion

* Dockerfiles modified
 2021-10-14 07:46:24 -07:00
Sriharsha Chintalapani
5644addc9a
Fix #748: Airflow lineage add inlet , outlet parsing to construct lineage (#749) 2021-10-13 09:33:15 +05:30
Sriharsha Chintalapani
c28665bca7
Sample lineage (#735) 
* Fix #727: Add sample lineage data and ingestion support
 2021-10-11 20:12:40 -07:00
Ayush Shah
a19c5bceca
Trino fix #705 (#706) 
* Trino fix #705

* Update trino.json to point to default database
 2021-10-07 12:25:03 -07:00
James
d455409cc9
issue-696: Added trino support for Openmetadata (#697) 
* issue-696: Added trino support for Openmetadata

* issue-696: fixed linting issues

* issue-696: not mentioning Trino for now as it will be part of 0.5 release

Co-authored-by: jbuoncri <jbuoncri@cisco.com>
 2021-10-07 11:15:34 -07:00
Sriharsha Chintalapani
1361bace11
sample entity generation (#646) 
* 1M entities to test scalability

* 1M entities to test scalability

* 1M entities test

* column name modification

* Fix service add and tags

* Add sample entity generation

Co-authored-by: parthp2107 <parth@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-10-04 22:56:28 +05:30
Sriharsha Chintalapani
37796c655b
Fix #650: Add support for pipelines in sample data & search integration (#651) 2021-10-04 17:48:45 +05:30
Sriharsha Chintalapani
ebfbd39cc2
Fix #624: Ingestion: Fix REST API calls to get service and create service (#625) 
* Fix metadata usage listing of tables

* Fix #624: Ingestion: Fix REST API calls to get service and create service
 2021-09-30 20:35:35 +05:30
Sriharsha Chintalapani
bfec0bfbed
Ingestion: Airflow integration to ingest metadata about pipelines and tasks (#609) 
* [WIP] Airlfow integration

* [WIP] Airlfow integration

* [WIP] airflow integration

* [WIP] Airflow

* [WIP] Airflow

* Fix #608: Ingestion: Airflow integration to ingest metadata about pipelines and tasks

* Fix #608: Ingestion: Airflow integration to ingest metadata about pipelines and tasks

* Update DashboardServiceResource.java

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-09-29 11:32:09 -07:00
Sriharsha Chintalapani
0983ec7f1f
Add nested types to sample data (#601) 
* Add nested types to sample data

* Add nested types to sample data
 2021-09-28 12:23:34 +05:30
Sriharsha Chintalapani
3d10347aec
Fix #596: Update ingestion to new OpenMetadata column entity changes (#597) 2021-09-27 23:11:19 +05:30
Sriharsha Chintalapani
eb2717b0e3
Fix #587: Ingestion: Add standalone report process to generate datasets, usage & profile and serve from standalone server (#588) 
* Fix #587: Ingestion: Add standalone report process to generate datasets, usage & profile and serve from standalone server

* add localhost
 2021-09-27 08:43:38 -07:00
parthp2107
06810cdec1
Fix #432:Added Redash Connector (#444) 
* added redash connector

* added redash connector

* Added Redash Connector

* minor changes

Co-authored-by: parthp2107 <parth.panchal@deuexsoultions.com>
Co-authored-by: parthp2107 <parth@getcollate.io>
 2021-09-22 15:09:24 +05:30
Ayush Shah
7bc2b14178
Added and Modified Redshift Pipeline (#528) 
* Pylint build failure fixed

* Redshift workflow added
 2021-09-19 21:16:50 -07:00
Ayush Shah
7652baa00d
Setup.py Refactored, ES port fix (#521) 
* Pylint build failure fixed

* Setup & dependency modified, Data profiler default to False, ES port fix

* Profiler requirements refactored

* Setup.py requirement fix

* openmetadata-ingestion version upgrade
 2021-09-19 13:59:14 +05:30
Sriharsha Chintalapani
b7adb5dc6b
Fix #469: Add Vertica Connector (#470) 2021-09-12 21:59:31 -07:00
Ayush Shah
d2df40cf2b
Fix #355: Tableau Implemented (#468) 
* Fix #355: Tableau Implemented

* Tableau pipeline location modification
 2021-09-11 11:46:10 -07:00
Sriharsha Chintalapani
ed698dc40f
Fix #466: Fix airflow example dag (#467) 2021-09-10 22:28:23 -07:00
Sriharsha Chintalapani
1c80dc246e
Fix #456: Make PII-Processor optional and independent install (#457) 2021-09-10 10:41:19 +05:30
Suresh Srinivas
30c1e5a14c
Merge pull request #447 from open-metadata/Issue-446 
Fix #446: Add DataProfiler to ingestion and APIs
 2021-09-09 11:25:56 -07:00
Ayush Shah
bdad904e82
Sample Data ingestion fix (#437) 2021-09-08 16:52:54 +05:30
Suresh Srinivas
328658ebea [WIP] profiler 2021-09-07 22:03:57 -07:00
Ayush Shah
2b61c8ea8c
fix Issue-404 replicationFactor and Message Maximum Byte added (#419) 
* fix Issue-404 replicationFactor and Message Maximum Byte added

* Cleanup Policy values changed in sample topic data
 2021-09-07 12:22:51 -07:00
parthp2107
3965b030a9
Added Salesforce Connector (#423) 
* Added salesforce-connector

* minor changes

* Added Salesforce-connector

* Added Salesforce-connector

* Salesforce sample data implemented

Co-authored-by: parthp2107 <parth.panchal@deuexsoultions.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-09-08 00:37:13 +05:30
Ayush Shah
657962bc4f
MSSQL sample-data query fix (#375) 
* MSSQL sample-data query fix

* Query Format as per Database implemented
 2021-09-06 21:03:04 -07:00
Sriharsha Chintalapani
d0dbcc19b7
Fix #401: Merge sample data generation into single connector (#402) 
* Fix #401: Merge sample data generation into single connector

* Path for datasets modified

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-09-05 22:35:02 +05:30
Sriharsha Chintalapani
eff1de87ef
Fix #381: SQLAlchemy URL is causing issues in creating service (#383) 2021-09-02 20:36:18 -07:00
Ayush Shah
c9ada4ca1a
Looker Dashboard Connecter Added (#351) 
* Looker Dashboard Connecter Added

* Dashboard yield fixed

* Looker Connector Method modifications
 2021-09-02 20:32:03 -07:00
parthp2107
d3d0ac19d0
Removed serviceType as a configuration option from workflow configs (#371) 
* Removed serviceType as a configuration option from workflow configs

* Removed serviceType as a configuration option from workflow configs

Co-authored-by: parthp2107 <parth.panchal@deuexsoultions.com>
 2021-09-02 19:45:41 +05:30
Sriharsha Chintalapani
193c6a782a Fix #346: Add displayName to Chart and Dashboard entities and use the dashboard and chart ids/unique identifiers in name field 2021-08-31 23:49:01 -07:00
Suresh Srinivas
fd99dbdb1a Fix #339: Add sample dashboards & charts 2021-08-30 22:13:45 -07:00
Suresh Srinivas
bd0c65f3e4 Fix #331: Integrate superset connector to OpenMetadata APIs 2021-08-27 21:33:42 -07:00
Suresh Srinivas
c36727aaf9
Merge pull request #307 from open-metadata/bigqueryusage-pipeline 
Bigquery Usage Pipeline added
 2021-08-26 22:24:09 -07:00
Suresh Srinivas
aa0ab362e6
Merge pull request #289 from open-metadata/issue-288 
Ingestion of Sample Data fixed - #288
 2021-08-26 22:23:22 -07:00