yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-07-23 01:12:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,072 Commits 688 Branches 147 Tags
Commit Graph

251 Commits

Author SHA1 Message Date
Sriharsha Chintalapani
7b637d4628
Fix #1665: Oracle connector add an option to configure oracle service name (#1666) 
* Fix #1665: Oracle connector add an option to configure oracle service name

* Fixed removal of semi colon

Fixes SQL Command Warning while ingesting

Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
 2021-12-10 16:37:19 +05:30
Sriharsha Chintalapani
6f0ef68701
Fix #1392: parse exclude/include filters from UI generated config (#1634) 2021-12-10 15:17:43 +05:30
Sriharsha Chintalapani
c8c57de37e
Issue 1658: Ingestion changes to add dbtModel as part of Table entity (#1659) 
* Fix #1658: Ingestion changes to add dbtModel as part of Table entity

* Fixes #1652 Remove DBTModel as top level entity and capture information from DBT in existing Table entity

Co-authored-by: sureshms <suresh@getcollate.io>
 2021-12-09 20:32:17 -08:00
Ayush Shah
7263510124
Query Log fixed (#1615) 
* Query Log fixed

* Wrong ColumnName fixed

* Added Service Name in usage configs
 2021-12-08 07:36:45 -08:00
parthp2107
64386b035e
added new tests for tags page (#1621) 
* added new tests for tags page

* addressing reviewdog findings

* Added Sample-usage service_name

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-12-08 16:54:48 +05:30
Ayush Shah
67433f559c
Trino JDBC updated (#1614) 2021-12-08 13:14:38 +05:30
Alberto Miorin
8621ee3fa0
Fix #1533, #1522: (#1575) 
- Glue works with columns containing dots.
- Glue works with temporary credentials.
- Fix bug in TableRepository, the EntityType was database instead of table.
- only region_name is mandatory in Glue config.
- first integration test for Glue. The CI integration with AWS is still missing.
 2021-12-06 16:03:05 -08:00
Alberto Miorin
f9436c65f0
Fix #1306: Evolve Location entity integration (#1442) 2021-12-03 02:13:13 +01:00
Ayush Shah
0e205d93dd
Added DBT to airflow, modified tables.json - added profile data, fixed user's displayName (#1501) 
* User's Name fixed, added profile data, added dbt pipeline under airflow

* service name changed
 2021-12-01 08:32:29 -08:00
parthp2107
a9e70e6a89
License header update (#1498) 
* updated liscense header

* updated license header

* updated license header

* updated license header

* addressing stylecheck findings
 2021-12-01 12:46:28 +05:30
Ayush Shah
47a97e7be0 FQN changed from bigquery to bigquery_gcp 2021-11-29 16:57:39 +05:30
pmbrull
98066bf986 Rename FQN for bigquery 2021-11-28 16:09:32 +01:00
Sriharsha Chintalapani
e4fa0247f5
Fix #1280: Amundsen connector to import metadata (#1281) 
* Fix #1280: Amundsen connector to import metadata
 2021-11-20 14:08:27 -08:00
Ayush Shah
219246b78e
Glue pagination added (#1282) 2021-11-20 12:46:18 -08:00
Sriharsha Chintalapani
f3054658f5
Fix #968: Add DBT Connector (#1200) 2021-11-16 01:02:45 -08:00
Sriharsha Chintalapani
f018664e0c
Issue-1197: Add MariaDB Connector (#1198) 2021-11-16 00:25:24 -08:00
parthp2107
4a5f5e69b3
uncommenting add-user test (#1097) 
* uncommenting add-user test

* quick-fix

* Ingestion script modified

* Airflow sample_usage modified, added sleep in curl commands

* adding chrome options for failing tests

* addressing reviewdog finding

* testing

* addressing reviewdog findings

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-11-14 20:16:39 -08:00
Ayush Shah
5dc3bb9297
Docker Support from Python added (#1158) 
* Script modified - supports running from different locations

* Docker support from Python CLI

* Docker plugin setup.py

* Paths for latest and local dockers updated

* Resolved Comments - Docker CLI optimized, timestamp added

* help attribute added to options

* Docker clean code refactored
 2021-11-12 10:30:28 -08:00
Ayush Shah
fa3c9b1758
Reduced the sample-users to 100 (#1156) 2021-11-11 07:31:03 -08:00
Ayush Shah
c95f1542eb
Docker fix (#1101) 
* Docker Fix, User Ingestion Fix

* Docker file optimized

* ES docker version rolledback

* Healthcheck removed - incompatible with 3.9

* Docker & Airflow Config files updated with - resolved comments

* Openmetadata managed Airflow apis support added

* Dockerfile updated

* Docker file Updated with release package

* Elasticsearch allocated memory increased
 2021-11-10 21:22:32 -08:00
Ayush Shah
9839191242
Issue 483 - Glue Implementation (#1124) 
* Glue Tables and Pipeline workflows implemented

* Glue Config Added

* Relative imports changed to absolute

* Resolving Comments - Changed Imports, serviceTypes

* Type fixed in setup.py
 2021-11-10 07:28:13 -08:00
Akash Jain
06b2f70e3e
kubernetes docker image compatibility for #1026 (#1042) 2021-11-03 08:18:52 -07:00
Sriharsha Chintalapani
9b7d3e313f
Fix #1004: Update Pipeline Ingestion and Indexing to accommodate new API changes (#1005) 2021-11-01 07:22:01 -07:00
Ayush Shah
93921814af
Docker fix - latest release changes (#983) 
* Docker fix - Architecture, MySQL

* Docker Airflow API Dagrun support

* Docker latest changes modified
 2021-10-30 09:05:30 -07:00
Pere Miquel Brull
710675d51a
OpenMetadata High Level API (#942) 
* update arg name for fqdn

* fix suite setup and teardown

* Update list logic

* Add Chart tests

* Add Dashboard tests

* Add pipeline tests

* Add table tests

* Simplify create API

* Formatting tests

* Update create API

* Add Tasks and Topics tests

* Rename OpenMetadata

* Add client close

* Use Optional for GET id and name

* Fix #931 fetching dashboards using client

* Add tests

* Fix sample users ingestion

* Remove lineage test

* Rename services for independence
 2021-10-26 14:18:43 -07:00
Ayush Shah
759574a8de
Ingestion Optimization - Sample Users, Dockerfiles, Removal of Pandas (#935) 
* Sample Profile Data for Sample Tables (#815)

* Sample Profile Data for Sample Tables

* Disabling Profile as Default

* Added Sample Profile Data to 3 additional sample tables

* Sample Tables fixed (#850)

* Pydantic fix, Docker update (#860)

* Setup.py Modified with openmetadata-airflow package, docker update

* Setup.py Modified

* Update setup.py

* Removed Pandas from Sample Data

* Sample Users added under sample data

* Sample User Standalone pipelines and modules removed

* Docker release package updated

* Dockerfile updated, removed redundant files

* Setup.py removed from ingestion src directory

* User Resource failing check resolved

* Modifying Usage Columns Datatype
 2021-10-26 09:14:24 -07:00
Ayush Shah
9cf3a4045f
Sample Profile Data for Sample Tables (#815) 
* Sample Profile Data for Sample Tables

* Disabling Profile as Default

* Added Sample Profile Data to 3 additional sample tables
 2021-10-17 17:24:39 -07:00
Pere Miquel Brull
3bf5999224
[Issue-759] - Model Entity Sample (#797) 
* Fix typo

* Prepare sample data for Model Entity

* Clean print

* Prepare sample model data ingestion

* Allow fstring in logging for style unification

* Update sink status signature

* Update sink status signature

* Return Model in REST

* Update workflow status info for readability
 2021-10-16 09:59:32 -07:00
Ayush Shah
72e4b8831f
Removing Cron from all configs (#773) 
* Removing Cron from all configs

* Removing Cron from Documentation
 2021-10-14 15:21:47 -07:00
Ayush Shah
85b6b72848
Airflow docker (#762) 
* Airflow Docker implementation - Ingestion

* Dockerfiles modified
 2021-10-14 07:46:24 -07:00
Sriharsha Chintalapani
5644addc9a
Fix #748: Airflow lineage add inlet , outlet parsing to construct lineage (#749) 2021-10-13 09:33:15 +05:30
Sriharsha Chintalapani
c28665bca7
Sample lineage (#735) 
* Fix #727: Add sample lineage data and ingestion support
 2021-10-11 20:12:40 -07:00
Ayush Shah
a19c5bceca
Trino fix #705 (#706) 
* Trino fix #705

* Update trino.json to point to default database
 2021-10-07 12:25:03 -07:00
James
d455409cc9
issue-696: Added trino support for Openmetadata (#697) 
* issue-696: Added trino support for Openmetadata

* issue-696: fixed linting issues

* issue-696: not mentioning Trino for now as it will be part of 0.5 release

Co-authored-by: jbuoncri <jbuoncri@cisco.com>
 2021-10-07 11:15:34 -07:00
Sriharsha Chintalapani
1361bace11
sample entity generation (#646) 
* 1M entities to test scalability

* 1M entities to test scalability

* 1M entities test

* column name modification

* Fix service add and tags

* Add sample entity generation

Co-authored-by: parthp2107 <parth@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-10-04 22:56:28 +05:30
Sriharsha Chintalapani
37796c655b
Fix #650: Add support for pipelines in sample data & search integration (#651) 2021-10-04 17:48:45 +05:30
Sriharsha Chintalapani
ebfbd39cc2
Fix #624: Ingestion: Fix REST API calls to get service and create service (#625) 
* Fix metadata usage listing of tables

* Fix #624: Ingestion: Fix REST API calls to get service and create service
 2021-09-30 20:35:35 +05:30
Sriharsha Chintalapani
bfec0bfbed
Ingestion: Airflow integration to ingest metadata about pipelines and tasks (#609) 
* [WIP] Airlfow integration

* [WIP] Airlfow integration

* [WIP] airflow integration

* [WIP] Airflow

* [WIP] Airflow

* Fix #608: Ingestion: Airflow integration to ingest metadata about pipelines and tasks

* Fix #608: Ingestion: Airflow integration to ingest metadata about pipelines and tasks

* Update DashboardServiceResource.java

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-09-29 11:32:09 -07:00
Sriharsha Chintalapani
0983ec7f1f
Add nested types to sample data (#601) 
* Add nested types to sample data

* Add nested types to sample data
 2021-09-28 12:23:34 +05:30
Sriharsha Chintalapani
3d10347aec
Fix #596: Update ingestion to new OpenMetadata column entity changes (#597) 2021-09-27 23:11:19 +05:30
Sriharsha Chintalapani
eb2717b0e3
Fix #587: Ingestion: Add standalone report process to generate datasets, usage & profile and serve from standalone server (#588) 
* Fix #587: Ingestion: Add standalone report process to generate datasets, usage & profile and serve from standalone server

* add localhost
 2021-09-27 08:43:38 -07:00
parthp2107
06810cdec1
Fix #432:Added Redash Connector (#444) 
* added redash connector

* added redash connector

* Added Redash Connector

* minor changes

Co-authored-by: parthp2107 <parth.panchal@deuexsoultions.com>
Co-authored-by: parthp2107 <parth@getcollate.io>
 2021-09-22 15:09:24 +05:30
Ayush Shah
7bc2b14178
Added and Modified Redshift Pipeline (#528) 
* Pylint build failure fixed

* Redshift workflow added
 2021-09-19 21:16:50 -07:00
Ayush Shah
7652baa00d
Setup.py Refactored, ES port fix (#521) 
* Pylint build failure fixed

* Setup & dependency modified, Data profiler default to False, ES port fix

* Profiler requirements refactored

* Setup.py requirement fix

* openmetadata-ingestion version upgrade
 2021-09-19 13:59:14 +05:30
Sriharsha Chintalapani
b7adb5dc6b
Fix #469: Add Vertica Connector (#470) 2021-09-12 21:59:31 -07:00
Ayush Shah
d2df40cf2b
Fix #355: Tableau Implemented (#468) 
* Fix #355: Tableau Implemented

* Tableau pipeline location modification
 2021-09-11 11:46:10 -07:00
Sriharsha Chintalapani
ed698dc40f
Fix #466: Fix airflow example dag (#467) 2021-09-10 22:28:23 -07:00
Sriharsha Chintalapani
1c80dc246e
Fix #456: Make PII-Processor optional and independent install (#457) 2021-09-10 10:41:19 +05:30
Suresh Srinivas
30c1e5a14c
Merge pull request #447 from open-metadata/Issue-446 
Fix #446: Add DataProfiler to ingestion and APIs
 2021-09-09 11:25:56 -07:00
Ayush Shah
bdad904e82
Sample Data ingestion fix (#437) 2021-09-08 16:52:54 +05:30