181 Commits

Author SHA1 Message Date
Ayush Shah
568845b65c
Power BI connector (#3019)
* Power BI Dashboard and Tiles added

* Added Powerbi dependency

* Modified Powerbi debug logs

* Bump up _version.py

* Modified Power BI - added failure status, resolved comments
2022-03-01 10:09:35 +05:30
Mayur Singal
e79be68bea
Fix #2937: added clickhouse usage connector (#2987) 2022-02-26 08:18:49 -08:00
Mayur Singal
4739aaa096
Fix #2905: Added MSSQL Usage Connector (#2948)
* Fix #2509: added mssql usage connector

* removed comment

* removed unused variables

* fixed code smell

* renamed sql variable
2022-02-24 01:01:03 +05:30
Sriharsha Chintalapani
165f5bb0e9
Fix #2819: Kafka connector security settings along with all configs to be retrieved (#2823)
* Fix #2811: Log level option for running metadata

* Fix #2819: Kafka connector security settings along with all configs to be retrieved

* Fix #2819: Kafka connector security settings along with all configs to be retrieved
2022-02-17 15:17:43 +05:30
Mayur Singal
419448a28f
Issue 2727: Added Databricks Connector (#2745)
* Issue #2727: Added Databricks Connector

* optional db and licence info

* make core_bump_version_dev

* host_port fix

* fixed service_type

* change package to sqlalchemy-databricks

* Update databaseService.json

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-02-17 11:22:13 +05:30
codingwithabhi
088dba3a1f
azuresql-config-added (#2711)
* azuresql-config-added

* rebased-with-main

* tables-ingestion-added

* duplicate-files-removed

* Update azuresql.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-02-17 09:23:07 +05:30
codingwithabhi
9011b27bf6
table-filter-added (#2770)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-02-16 19:26:40 +05:30
Mayur Singal
5e4cd516bf
issue #2712: Added SingleStore Connector (#2747) 2022-02-15 21:11:46 -08:00
Mayur Singal
88afb17ee8
issue #2611: added clickhouse connector (#2679)
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-10 08:32:01 -08:00
Ayush Shah
4f5f63b226
Issue-2672: Added support for personal access token name and secret and upgraded tableau version (#2682) 2022-02-08 23:02:56 +05:30
Ayush Shah
18d963c3d8
Issue-969: Superset Lineage Implemented (#2659) 2022-02-07 16:16:38 -08:00
codingwithabhi
66c3f43bbb
Fix #2537: Ingestion Connector - DynamoDB 2022-02-07 10:52:11 -08:00
Mayur Singal
2d25da149d
Fix #2626: add db2 connector (#2661)
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-07 10:49:59 -08:00
Ayush Shah
6f5a4eafe7
Bigquery support for env without passing any config (#2160)
* Bigquery support for env without passing any config

* Bigquery File formatted
2022-01-22 22:25:48 -08:00
codingwithabhi
ddf4ae261b
Bigquery credential configuration updated (#2041)
* bigquery-credentials-in-bigquery-json

* ui-changes-reverted

* credentials-moved-to-options

* Update bigquery.json

* Update bigquery.py

* delete-temp-code-added

* delete-temp-code-added

* removed-init-from-config

* Update datatypes_test.py

* Update column_helpers.py

* code-formatted

Co-authored-by: Abhishek <abhishek@Abhisheks-MacBook-Air.local>
Co-authored-by: = <=>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-21 17:23:02 +05:30
Matt
2a80caa8fe
Add logic to initialize relationships from seed data during application startup (#2307)
* Add logic to initialize relationships from seed data during application startup

* Remove ingestion related code for access control policies

* Move PolicyEvaluator init to PolicyResource
2022-01-20 22:18:03 -08:00
Sriharsha Chintalapani
0d3ded0742
Data Profiler Integration (#2235)
* Fix 2234: Data profiler integration
2022-01-18 20:25:43 -08:00
Sriharsha Chintalapani
cf6f438531
Fix #1994: Add support for marking dataset entities as deleted (#2141)
* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted
2022-01-11 11:42:53 +05:30
Ayush Shah
19e5f737ff
Usage fix - removed service_name from bulk_sink (#1956)
* Query Usage fixed

* Usage updated

* Reformatted change

* Usage files reformatted
2022-01-04 17:12:10 +05:30
Matt
ab4c9ede25
Add ingestion for access_control_policies to support RBAC (#1969) 2021-12-30 08:25:09 -08:00
Matt
f7653f4122
Add support for GCS Lifecycle Policy Ingestion (#1923)
- Add example gcs.json
- Add support for MoveAction and DeleteAction
2021-12-26 20:26:33 -08:00
Matt
4588121520
Add support for DeleteAction in S3 Lifecycle Policy (#1916)
- Support Delete Action for S3
- Add Example s3.json
- Create AWSClient util
- Use AWSClient util in S3 ingestion source
- Remove ambiguity in policy filters by removing array with different types
2021-12-25 16:29:10 -08:00
Sriharsha Chintalapani
485fdea7b1
Fix #1854: Support DBT 1.0 format (#1855)
* Fix #1854: Support DBT 1.0 format
2021-12-20 12:25:26 -08:00
Sriharsha Chintalapani
3c254440a5
Fix #1841: Presto Connector issues (#1842) 2021-12-18 23:27:51 -08:00
Sriharsha Chintalapani
df6232ef52
Fix #1808: check for None in description (#1809) 2021-12-16 21:15:42 +01:00
Ayush Shah
d71c75aaa1
Update snowflake.py (#1803)
* Update snowflake.py

* snowflake needs database and warehouse
2021-12-16 10:46:49 -08:00
Ayush Shah
06712fa425
Metabase and Snowflake Fixed (#1788)
* Metabase and Snowflake Fixed

* Removed DBT
2021-12-16 11:57:43 +05:30
Ayush Shah
07924f80c3
Added Metabase Connector (#1726)
* Added Metabase Connector

* Metabase Added support for lineage

* Modified Metabase as per comments

* Metabase filter patterns added

* Update databaseService.json
2021-12-15 09:53:55 -08:00
Sriharsha Chintalapani
e3d1a95d2c
Fix #1737: Add separate filter patterns for database/schema and tables (#1739)
* Fix #1737: Add separate filter patterns for database/schema and tables
2021-12-14 08:50:05 -08:00
Alberto Miorin
93276079ec
Fix #1729: Connector Trino works with username. (#1732)
Implementation details
I have decided to rename schema_name to database and make it mandatory. Without database there is an error while scanning all available tables. The connector doesn't support multiple databases at the moment. It has to be tested with passwords. Trino requires SSL if you use passwords. It has to be tested with impersonation. I have removed quote_plus because I don't think it's needed.

- [x] Support username
- [ ] There is an integration test
- [ ] Support impersonation
- [ ] Support passwords
- [ ] Support tokens
- [ ] Support multiple databases
2021-12-13 22:05:28 +01:00
Sriharsha Chintalapani
a9198c72f2
Fix #1690: Ingestion: Apply DBT models to views (#1691)
* Fix #1690: Ingestion: Apply DBT models to views

* Fix #1690: Ingestion: Apply DBT models to views
2021-12-12 20:53:08 +05:30
Pere Miquel Brull
a7c8056bf5
[issue-1079] - MlFlow Connector (#1677)
* Prepare infra

* Store experiment information in MySQL & MinIO

* Use CreateMlModelEntityRequest instead of MlModel for PUT operations

* Update MlFlow infra

* Prepare MlFlow source

* Prepare Mlflow workflow

* Simplify test and prepare README

* Revert compose

* Fix compose

* Prepare warnings and fix features

* Use non-default port for integration test

* Use mlflow-skinny for the client application
2021-12-11 18:40:19 -08:00
Sriharsha Chintalapani
7b637d4628
Fix #1665: Oracle connector add an option to configure oracle service name (#1666)
* Fix #1665: Oracle connector add an option to configure oracle service name

* Fixed removal of semi colon

Fixes SQL Command Warning while ingesting

Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
2021-12-10 16:37:19 +05:30
Sriharsha Chintalapani
6f0ef68701
Fix #1392: parse exclude/include filters from UI generated config (#1634) 2021-12-10 15:17:43 +05:30
Sriharsha Chintalapani
c8c57de37e
Issue 1658: Ingestion changes to add dbtModel as part of Table entity (#1659)
* Fix #1658: Ingestion changes to add dbtModel as part of Table entity

* Fixes #1652 Remove DBTModel as top level entity and capture information from DBT in existing Table entity

Co-authored-by: sureshms <suresh@getcollate.io>
2021-12-09 20:32:17 -08:00
Ayush Shah
7263510124
Query Log fixed (#1615)
* Query Log fixed

* Wrong ColumnName fixed

* Added Service Name in usage configs
2021-12-08 07:36:45 -08:00
Ayush Shah
67433f559c
Trino JDBC updated (#1614) 2021-12-08 13:14:38 +05:30
Alberto Miorin
8621ee3fa0
Fix #1533, #1522: (#1575)
- Glue works with columns containing dots.
- Glue works with temporary credentials.
- Fix bug in TableRepository, the EntityType was database instead of table.
- only region_name is mandatory in Glue config.
- first integration test for Glue. The CI integration with AWS is still missing.
2021-12-06 16:03:05 -08:00
Sriharsha Chintalapani
e4fa0247f5
Fix #1280: Amundsen connector to import metadata (#1281)
* Fix #1280: Amundsen connector to import metadata
2021-11-20 14:08:27 -08:00
Ayush Shah
219246b78e
Glue pagination added (#1282) 2021-11-20 12:46:18 -08:00
Sriharsha Chintalapani
f3054658f5
Fix #968: Add DBT Connector (#1200) 2021-11-16 01:02:45 -08:00
Sriharsha Chintalapani
f018664e0c
Issue-1197: Add MariaDB Connector (#1198) 2021-11-16 00:25:24 -08:00
Ayush Shah
9839191242
Issue 483 - Glue Implementation (#1124)
* Glue Tables and Pipeline workflows implemented

* Glue Config Added

* Relative imports changed to absolute

* Resolving Comments - Changed Imports, serviceTypes

* Type fixed in setup.py
2021-11-10 07:28:13 -08:00
Ayush Shah
72e4b8831f
Removing Cron from all configs (#773)
* Removing Cron from all configs

* Removing Cron from Documentation
2021-10-14 15:21:47 -07:00
Ayush Shah
a19c5bceca
Trino fix #705 (#706)
* Trino fix #705

* Update trino.json to point to default database
2021-10-07 12:25:03 -07:00
James
d455409cc9
issue-696: Added trino support for Openmetadata (#697)
* issue-696: Added trino support for Openmetadata

* issue-696: fixed linting issues

* issue-696: not mentioning Trino for now as it will be part of 0.5 release

Co-authored-by: jbuoncri <jbuoncri@cisco.com>
2021-10-07 11:15:34 -07:00
Sriharsha Chintalapani
1361bace11
sample entity generation (#646)
* 1M entities to test scalability

* 1M entities to test scalability

* 1M entities test

* column name modification

* Fix service add and tags

* Add sample entity generation

Co-authored-by: parthp2107 <parth@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-10-04 22:56:28 +05:30
Sriharsha Chintalapani
eb2717b0e3
Fix #587: Ingestion: Add standalone report process to generate datasets, usage & profile and serve from standalone server (#588)
* Fix #587: Ingestion: Add standalone report process to generate datasets, usage & profile and serve from standalone server

* add localhost
2021-09-27 08:43:38 -07:00
parthp2107
06810cdec1
Fix #432:Added Redash Connector (#444)
* added redash connector

* added redash connector

* Added Redash Connector

* minor changes

Co-authored-by: parthp2107 <parth.panchal@deuexsoultions.com>
Co-authored-by: parthp2107 <parth@getcollate.io>
2021-09-22 15:09:24 +05:30
Ayush Shah
7bc2b14178
Added and Modified Redshift Pipeline (#528)
* Pylint build failure fixed

* Redshift workflow added
2021-09-19 21:16:50 -07:00