157 Commits

Author SHA1 Message Date
Sriharsha Chintalapani
df6232ef52
Fix #1808: check for None in description (#1809) 2021-12-16 21:15:42 +01:00
Ayush Shah
d71c75aaa1
Update snowflake.py (#1803)
* Update snowflake.py

* snowflake needs database and warehouse
2021-12-16 10:46:49 -08:00
Ayush Shah
06712fa425
Metabase and Snowflake Fixed (#1788)
* Metabase and Snowflake Fixed

* Removed DBT
2021-12-16 11:57:43 +05:30
Ayush Shah
07924f80c3
Added Metabase Connector (#1726)
* Added Metabase Connector

* Metabase Added support for lineage

* Modified Metabase as per comments

* Metabase filter patterns added

* Update databaseService.json
2021-12-15 09:53:55 -08:00
Sriharsha Chintalapani
e3d1a95d2c
Fix #1737: Add separate filter patterns for database/schema and tables (#1739)
* Fix #1737: Add separate filter patterns for database/schema and tables
2021-12-14 08:50:05 -08:00
Alberto Miorin
93276079ec
Fix #1729: Connector Trino works with username. (#1732)
Implementation details
I have decided to rename schema_name to database and make it mandatory. Without database there is an error while scanning all available tables. The connector doesn't support multiple databases at the moment. It has to be tested with passwords. Trino requires SSL if you use passwords. It has to be tested with impersonation. I have removed quote_plus because I don't think it's needed.

- [x] Support username
- [ ] There is an integration test
- [ ] Support impersonation
- [ ] Support passwords
- [ ] Support tokens
- [ ] Support multiple databases
2021-12-13 22:05:28 +01:00
Sriharsha Chintalapani
a9198c72f2
Fix #1690: Ingestion: Apply DBT models to views (#1691)
* Fix #1690: Ingestion: Apply DBT models to views

* Fix #1690: Ingestion: Apply DBT models to views
2021-12-12 20:53:08 +05:30
Pere Miquel Brull
a7c8056bf5
[issue-1079] - MlFlow Connector (#1677)
* Prepare infra

* Store experiment information in MySQL & MinIO

* Use CreateMlModelEntityRequest instead of MlModel for PUT operations

* Update MlFlow infra

* Prepare MlFlow source

* Prepare Mlflow workflow

* Simplify test and prepare README

* Revert compose

* Fix compose

* Prepare warnings and fix features

* Use non-default port for integration test

* Use mlflow-skinny for the client application
2021-12-11 18:40:19 -08:00
Sriharsha Chintalapani
7b637d4628
Fix #1665: Oracle connector add an option to configure oracle service name (#1666)
* Fix #1665: Oracle connector add an option to configure oracle service name

* Fixed removal of semi colon

Fixes SQL Command Warning while ingesting

Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
2021-12-10 16:37:19 +05:30
Sriharsha Chintalapani
6f0ef68701
Fix #1392: parse exclude/include filters from UI generated config (#1634) 2021-12-10 15:17:43 +05:30
Sriharsha Chintalapani
c8c57de37e
Issue 1658: Ingestion changes to add dbtModel as part of Table entity (#1659)
* Fix #1658: Ingestion changes to add dbtModel as part of Table entity

* Fixes #1652 Remove DBTModel as top level entity and capture information from DBT in existing Table entity

Co-authored-by: sureshms <suresh@getcollate.io>
2021-12-09 20:32:17 -08:00
Ayush Shah
7263510124
Query Log fixed (#1615)
* Query Log fixed

* Wrong ColumnName fixed

* Added Service Name in usage configs
2021-12-08 07:36:45 -08:00
Ayush Shah
67433f559c
Trino JDBC updated (#1614) 2021-12-08 13:14:38 +05:30
Alberto Miorin
8621ee3fa0
Fix #1533, #1522: (#1575)
- Glue works with columns containing dots.
- Glue works with temporary credentials.
- Fix bug in TableRepository, the EntityType was database instead of table.
- only region_name is mandatory in Glue config.
- first integration test for Glue. The CI integration with AWS is still missing.
2021-12-06 16:03:05 -08:00
Sriharsha Chintalapani
e4fa0247f5
Fix #1280: Amundsen connector to import metadata (#1281)
* Fix #1280: Amundsen connector to import metadata
2021-11-20 14:08:27 -08:00
Ayush Shah
219246b78e
Glue pagination added (#1282) 2021-11-20 12:46:18 -08:00
Sriharsha Chintalapani
f3054658f5
Fix #968: Add DBT Connector (#1200) 2021-11-16 01:02:45 -08:00
Sriharsha Chintalapani
f018664e0c
Issue-1197: Add MariaDB Connector (#1198) 2021-11-16 00:25:24 -08:00
Ayush Shah
9839191242
Issue 483 - Glue Implementation (#1124)
* Glue Tables and Pipeline workflows implemented

* Glue Config Added

* Relative imports changed to absolute

* Resolving Comments - Changed Imports, serviceTypes

* Type fixed in setup.py
2021-11-10 07:28:13 -08:00
Ayush Shah
72e4b8831f
Removing Cron from all configs (#773)
* Removing Cron from all configs

* Removing Cron from Documentation
2021-10-14 15:21:47 -07:00
Ayush Shah
a19c5bceca
Trino fix #705 (#706)
* Trino fix #705

* Update trino.json to point to default database
2021-10-07 12:25:03 -07:00
James
d455409cc9
issue-696: Added trino support for Openmetadata (#697)
* issue-696: Added trino support for Openmetadata

* issue-696: fixed linting issues

* issue-696: not mentioning Trino for now as it will be part of 0.5 release

Co-authored-by: jbuoncri <jbuoncri@cisco.com>
2021-10-07 11:15:34 -07:00
Sriharsha Chintalapani
1361bace11
sample entity generation (#646)
* 1M entities to test scalability

* 1M entities to test scalability

* 1M entities test

* column name modification

* Fix service add and tags

* Add sample entity generation

Co-authored-by: parthp2107 <parth@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-10-04 22:56:28 +05:30
Sriharsha Chintalapani
eb2717b0e3
Fix #587: Ingestion: Add standalone report process to generate datasets, usage & profile and serve from standalone server (#588)
* Fix #587: Ingestion: Add standalone report process to generate datasets, usage & profile and serve from standalone server

* add localhost
2021-09-27 08:43:38 -07:00
parthp2107
06810cdec1
Fix #432:Added Redash Connector (#444)
* added redash connector

* added redash connector

* Added Redash Connector

* minor changes

Co-authored-by: parthp2107 <parth.panchal@deuexsoultions.com>
Co-authored-by: parthp2107 <parth@getcollate.io>
2021-09-22 15:09:24 +05:30
Ayush Shah
7bc2b14178
Added and Modified Redshift Pipeline (#528)
* Pylint build failure fixed

* Redshift workflow added
2021-09-19 21:16:50 -07:00
Ayush Shah
7652baa00d
Setup.py Refactored, ES port fix (#521)
* Pylint build failure fixed

* Setup & dependency modified, Data profiler default to False, ES port fix

* Profiler requirements refactored

* Setup.py requirement fix

* openmetadata-ingestion version upgrade
2021-09-19 13:59:14 +05:30
Sriharsha Chintalapani
b7adb5dc6b
Fix #469: Add Vertica Connector (#470) 2021-09-12 21:59:31 -07:00
Ayush Shah
d2df40cf2b
Fix #355: Tableau Implemented (#468)
* Fix #355: Tableau Implemented

* Tableau pipeline location modification
2021-09-11 11:46:10 -07:00
Sriharsha Chintalapani
1c80dc246e
Fix #456: Make PII-Processor optional and independent install (#457) 2021-09-10 10:41:19 +05:30
Suresh Srinivas
328658ebea [WIP] profiler 2021-09-07 22:03:57 -07:00
parthp2107
3965b030a9
Added Salesforce Connector (#423)
* Added salesforce-connector

* minor changes

* Added Salesforce-connector

* Added Salesforce-connector

* Salesforce sample data implemented

Co-authored-by: parthp2107 <parth.panchal@deuexsoultions.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-09-08 00:37:13 +05:30
Ayush Shah
657962bc4f
MSSQL sample-data query fix (#375)
* MSSQL sample-data query fix

* Query Format as per Database implemented
2021-09-06 21:03:04 -07:00
Sriharsha Chintalapani
eff1de87ef
Fix #381: SQLAlchemy URL is causing issues in creating service (#383) 2021-09-02 20:36:18 -07:00
Ayush Shah
c9ada4ca1a
Looker Dashboard Connecter Added (#351)
* Looker Dashboard Connecter Added

* Dashboard yield fixed

* Looker Connector Method modifications
2021-09-02 20:32:03 -07:00
parthp2107
d3d0ac19d0
Removed serviceType as a configuration option from workflow configs (#371)
* Removed serviceType as a configuration option from workflow configs

* Removed serviceType as a configuration option from workflow configs

Co-authored-by: parthp2107 <parth.panchal@deuexsoultions.com>
2021-09-02 19:45:41 +05:30
Suresh Srinivas
bd0c65f3e4 Fix #331: Integrate superset connector to OpenMetadata APIs 2021-08-27 21:33:42 -07:00
Suresh Srinivas
c36727aaf9
Merge pull request #307 from open-metadata/bigqueryusage-pipeline
Bigquery Usage Pipeline added
2021-08-26 22:24:09 -07:00
Suresh Srinivas
c790d4fc75
Merge pull request #291 from open-metadata/bigquery-optimize
Added credentials path to the bigquery config
2021-08-26 22:22:57 -07:00
Ayush Shah
820af4629c RedshiftConfig Filter Modified 2021-08-26 01:42:19 +05:30
Ayush Shah
a58cae7286 Pipeline config changes 2021-08-26 01:19:32 +05:30
Ayush Shah
bd40ae45ce Bigquery Usage Pipeline added 2021-08-26 01:15:02 +05:30
Suresh Srinivas
6a28ae988f [WIP] Issue #285: Add support for Dashboard Entities; Superset connector 2021-08-24 13:47:41 -07:00
Ayush Shah
c813e8f71b Added credentials path to the bigquery config 2021-08-24 23:53:34 +05:30
Ayush Shah
07dd7b36f4 Pipeline Ingestion refactoring 2021-08-24 18:49:53 +05:30
Ayush Shah
f543c3f99b Pipeline Fixes 2021-08-24 18:49:53 +05:30
Suresh Srinivas
4533bc49cb
Merge pull request #258 from open-metadata/kafka-pulsar
Kafka pulsar APIs & Kafka connector
2021-08-21 20:17:54 -07:00
Suresh Srinivas
19151dcac7 Ingestion: Add Kafka Connector 2021-08-21 17:52:24 -07:00
Suresh Srinivas
dc7e05dd74 Ingestion: Add Confluent Kafka topic and schema connector 2021-08-21 13:16:40 -07:00
Suresh Srinivas
be2d55ba1e Ingestion: close sql connection 2021-08-20 11:18:29 -07:00