251 Commits

Author SHA1 Message Date
Alberto Miorin
fe5618c8f1
Fix #3037: metadata --version doesn't work (#3038) 2022-03-01 12:19:36 +01:00
codingwithabhi
06d9329ae3
Atlas connector (#2855)
* Fix #1161: Add Atlas Connector

* Atlast Decimal type handled

* Atlas col_data_type handled

* Atlas columns exception handling implemented

* atlas-services-added-into-single-file

* import-updated

* authentication-fix

* atlas-connector-services-added

* code-smell-removed

* code-smell-removed

* code-smell-removed

* file-exist-check-added

* updated-mapping-json-logic

* pr-comment-changes-completed

* atlas-linegae-added

* atlas-linegae-added

* pr-changes-done

* code-formatted

* bug-resolved

* topic-bug-resolved

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-03-01 11:50:14 +05:30
Ayush Shah
568845b65c
Power BI connector (#3019)
* Power BI Dashboard and Tiles added

* Added Powerbi dependency

* Modified Powerbi debug logs

* Bump up _version.py

* Modified Power BI - added failure status, resolved comments
2022-03-01 10:09:35 +05:30
Mayur Singal
c49af971a7
Fix #2984: added azure sso auth (#3002) 2022-02-28 11:44:28 -08:00
Ayush Shah
412d61a875
Fix Okta Authentication and Validation - Ingestion (#2955) 2022-02-26 08:19:36 -08:00
Mayur Singal
e79be68bea
Fix #2937: added clickhouse usage connector (#2987) 2022-02-26 08:18:49 -08:00
Pere Miquel Brull
4d928bab80
Fix #2965 - Additional Properties set to False (#2976) 2022-02-24 08:38:56 -08:00
Mayur Singal
4739aaa096
Fix #2905: Added MSSQL Usage Connector (#2948)
* Fix #2509: added mssql usage connector

* removed comment

* removed unused variables

* fixed code smell

* renamed sql variable
2022-02-24 01:01:03 +05:30
Ayush Shah
d2c64007cb
Table Constraints Added - Ingestion (#2854) 2022-02-19 09:16:15 -08:00
Sriharsha Chintalapani
891ff465a0
Fix #2838: Fix airflow example DAG to import airflow libs (#2839) 2022-02-18 10:47:07 +05:30
Sriharsha Chintalapani
165f5bb0e9
Fix #2819: Kafka connector security settings along with all configs to be retrieved (#2823)
* Fix #2811: Log level option for running metadata

* Fix #2819: Kafka connector security settings along with all configs to be retrieved

* Fix #2819: Kafka connector security settings along with all configs to be retrieved
2022-02-17 15:17:43 +05:30
Mayur Singal
419448a28f
Issue 2727: Added Databricks Connector (#2745)
* Issue #2727: Added Databricks Connector

* optional db and licence info

* make core_bump_version_dev

* host_port fix

* fixed service_type

* change package to sqlalchemy-databricks

* Update databaseService.json

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-02-17 11:22:13 +05:30
codingwithabhi
088dba3a1f
azuresql-config-added (#2711)
* azuresql-config-added

* rebased-with-main

* tables-ingestion-added

* duplicate-files-removed

* Update azuresql.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-02-17 09:23:07 +05:30
codingwithabhi
9011b27bf6
table-filter-added (#2770)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-02-16 19:26:40 +05:30
Mayur Singal
5e4cd516bf
issue #2712: Added SingleStore Connector (#2747) 2022-02-15 21:11:46 -08:00
Suresh Srinivas
28ba1a3c04
Fixes #2760 - Add entities for Glossary and initial API (#2761)
* Fixes #2760 - Add entities for Glossary and initial API

* Fixing merge issues
2022-02-15 20:54:46 -08:00
Mayur Singal
44cca4d020
fix #2663: added table queries in sample data (#2763) 2022-02-15 15:45:58 +05:30
Mayur Singal
88afb17ee8
issue #2611: added clickhouse connector (#2679)
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-10 08:32:01 -08:00
Ayush Shah
4f5f63b226
Issue-2672: Added support for personal access token name and secret and upgraded tableau version (#2682) 2022-02-08 23:02:56 +05:30
Ayush Shah
18d963c3d8
Issue-969: Superset Lineage Implemented (#2659) 2022-02-07 16:16:38 -08:00
codingwithabhi
66c3f43bbb
Fix #2537: Ingestion Connector - DynamoDB 2022-02-07 10:52:11 -08:00
Mayur Singal
2d25da149d
Fix #2626: add db2 connector (#2661)
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-07 10:49:59 -08:00
Ayush Shah
7013b070d2
Minor Data Type Display modification (#2654)
Characters modified from unicode for e.g. &lt; to <
2022-02-07 18:07:08 +05:30
Pere Miquel Brull
7aceab41a2
Fix #2412 - Airflow Lineage Docs (#2618) 2022-02-06 12:51:05 -08:00
Ayush Shah
6f5a4eafe7
Bigquery support for env without passing any config (#2160)
* Bigquery support for env without passing any config

* Bigquery File formatted
2022-01-22 22:25:48 -08:00
Pere Miquel Brull
6693b4ae40
[issue-2262] - Simplify sample data DAG (#2337)
* Simplify sample data DAG

* Remove mkdir

* Generate sources before running compose

* Generate sources to install models to ingestion image

* Add python-on-whales for docker --start

* Remove python-on-whales from base

* Install venv

* Setup python
2022-01-22 16:37:56 +05:30
Sriharsha Chintalapani
10e71a7e09
Fix #2340: Ingestion pipeline deployment refactor (#2341)
* Fix #2340: Ingestion pipeline deployment refactor
2022-01-21 22:06:14 -08:00
codingwithabhi
ddf4ae261b
Bigquery credential configuration updated (#2041)
* bigquery-credentials-in-bigquery-json

* ui-changes-reverted

* credentials-moved-to-options

* Update bigquery.json

* Update bigquery.py

* delete-temp-code-added

* delete-temp-code-added

* removed-init-from-config

* Update datatypes_test.py

* Update column_helpers.py

* code-formatted

Co-authored-by: Abhishek <abhishek@Abhisheks-MacBook-Air.local>
Co-authored-by: = <=>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-21 17:23:02 +05:30
Matt
2a80caa8fe
Add logic to initialize relationships from seed data during application startup (#2307)
* Add logic to initialize relationships from seed data during application startup

* Remove ingestion related code for access control policies

* Move PolicyEvaluator init to PolicyResource
2022-01-20 22:18:03 -08:00
Sriharsha Chintalapani
0d3ded0742
Data Profiler Integration (#2235)
* Fix 2234: Data profiler integration
2022-01-18 20:25:43 -08:00
Matt
276128e8b0
Add MetadataOperations RBAC support for PATCH APIs (#2128)
* Add RBAC for PATCH APIs

* Expand scope to all resources except a few (Policy, User, Role, Team, resources that doesn't support PATCH)

* Fix code smells
2022-01-16 09:24:26 -08:00
Matt
f323afe0cd
Support role ingestion from sample data (#2167) 2022-01-11 16:51:18 -08:00
Sriharsha Chintalapani
cf6f438531
Fix #1994: Add support for marking dataset entities as deleted (#2141)
* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted
2022-01-11 11:42:53 +05:30
Alberto Miorin
c8de136cca
First part of #2091: Python code in branch main is always formatted correctly (#2093) 2022-01-07 06:17:48 +01:00
Ayush Shah
19e5f737ff
Usage fix - removed service_name from bulk_sink (#1956)
* Query Usage fixed

* Usage updated

* Reformatted change

* Usage files reformatted
2022-01-04 17:12:10 +05:30
Ayush Shah
6175053dd4
Sample Topics fixed (#2005) 2022-01-03 11:00:04 -08:00
Matt
ab4c9ede25
Add ingestion for access_control_policies to support RBAC (#1969) 2021-12-30 08:25:09 -08:00
Matt
f7653f4122
Add support for GCS Lifecycle Policy Ingestion (#1923)
- Add example gcs.json
- Add support for MoveAction and DeleteAction
2021-12-26 20:26:33 -08:00
Matt
4588121520
Add support for DeleteAction in S3 Lifecycle Policy (#1916)
- Support Delete Action for S3
- Add Example s3.json
- Create AWSClient util
- Use AWSClient util in S3 ingestion source
- Remove ambiguity in policy filters by removing array with different types
2021-12-25 16:29:10 -08:00
Sriharsha Chintalapani
485fdea7b1
Fix #1854: Support DBT 1.0 format (#1855)
* Fix #1854: Support DBT 1.0 format
2021-12-20 12:25:26 -08:00
Sriharsha Chintalapani
3c254440a5
Fix #1841: Presto Connector issues (#1842) 2021-12-18 23:27:51 -08:00
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup (#1730)
* Fix typo

* Clean setup

* Update ingestion local image to be barebone on connector dependencies

* Prepare ingestion connectors base image

* Add system dependencies

* Prepare docker CLI

* Add docker provider

* Prepare entrypoint for the image

* Remove DBT pipeline as per Issue 1658

* Add TODO for ingestion build

* Bind docker socket

* Update comment

* Update README

* Use DockerOperator in sample data

* Build images with latest tag

* Prepare symlink to pass the volume to the DockerOperator

* Update README

* Prepare Base image for CI

* COPY multiple files into dir

* COPY multiple files into dir

* Remove DBT source as is now part of table ingestion

* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
Sriharsha Chintalapani
df6232ef52
Fix #1808: check for None in description (#1809) 2021-12-16 21:15:42 +01:00
Ayush Shah
d71c75aaa1
Update snowflake.py (#1803)
* Update snowflake.py

* snowflake needs database and warehouse
2021-12-16 10:46:49 -08:00
Ayush Shah
06712fa425
Metabase and Snowflake Fixed (#1788)
* Metabase and Snowflake Fixed

* Removed DBT
2021-12-16 11:57:43 +05:30
Ayush Shah
07924f80c3
Added Metabase Connector (#1726)
* Added Metabase Connector

* Metabase Added support for lineage

* Modified Metabase as per comments

* Metabase filter patterns added

* Update databaseService.json
2021-12-15 09:53:55 -08:00
Sriharsha Chintalapani
e3d1a95d2c
Fix #1737: Add separate filter patterns for database/schema and tables (#1739)
* Fix #1737: Add separate filter patterns for database/schema and tables
2021-12-14 08:50:05 -08:00
Alberto Miorin
93276079ec
Fix #1729: Connector Trino works with username. (#1732)
Implementation details
I have decided to rename schema_name to database and make it mandatory. Without database there is an error while scanning all available tables. The connector doesn't support multiple databases at the moment. It has to be tested with passwords. Trino requires SSL if you use passwords. It has to be tested with impersonation. I have removed quote_plus because I don't think it's needed.

- [x] Support username
- [ ] There is an integration test
- [ ] Support impersonation
- [ ] Support passwords
- [ ] Support tokens
- [ ] Support multiple databases
2021-12-13 22:05:28 +01:00
Sriharsha Chintalapani
a9198c72f2
Fix #1690: Ingestion: Apply DBT models to views (#1691)
* Fix #1690: Ingestion: Apply DBT models to views

* Fix #1690: Ingestion: Apply DBT models to views
2021-12-12 20:53:08 +05:30
Pere Miquel Brull
a7c8056bf5
[issue-1079] - MlFlow Connector (#1677)
* Prepare infra

* Store experiment information in MySQL & MinIO

* Use CreateMlModelEntityRequest instead of MlModel for PUT operations

* Update MlFlow infra

* Prepare MlFlow source

* Prepare Mlflow workflow

* Simplify test and prepare README

* Revert compose

* Fix compose

* Prepare warnings and fix features

* Use non-default port for integration test

* Use mlflow-skinny for the client application
2021-12-11 18:40:19 -08:00