Ayush Shah
8cc91c3405
Modified Ingestion to support 3.6 and later ( #5003 )
2022-05-17 21:38:57 +05:30
Milan Bariya
0acf88e1ba
ISSUE-1072: Add Apache Pinot support ( #4649 )
...
* ISSUE-1072: Add Apache Pinot support
* ISSUE-1072: Add Apache Pinot support
* Rename pinotdbConnection.json to pinotDBConnection.json
2022-05-11 11:13:28 +02:00
Ayush Shah
c94de59a40
Missing pymssql dependency ( #4841 )
2022-05-10 19:28:16 +05:30
Sriharsha Chintalapani
34858f3b32
Topics: Add Sample data ( #4787 )
...
* Topics: Add Sample data
* Topics: Add Sample data
* Topics: Add Sample data
* Topics: Add Sample data
* Topics: Add Sample data
2022-05-09 12:50:54 +02:00
Onkar Ravgan
6a5ffb0626
added google cloud storage dependency ( #4651 )
...
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-05-03 15:13:46 +05:30
Ayush Shah
88cf0930e2
Fix Dependency issue ( #4523 )
2022-04-27 14:54:47 +05:30
Akash Jain
55e1be9ca4
feat: Prepare 0.11.0-snapshot version changes ( #4505 )
...
* updated version to 0.11.0-SNAPSHOT
* updated version to 0.11.0.dev* for ingestion
* docker version updates for metadata docker cli
* OM Server Env Variables from #4197 #4261 #4473
2022-04-27 00:41:17 +05:30
Pere Miquel Brull
bd150d8124
Fix #4457 - BigQuery requirements to handle keys ( #4464 )
...
* Add defaults
* Add crypto req for bigquery
* Crypto is in base
2022-04-25 11:54:45 +02:00
Onkar Ravgan
c4aa07858d
ISSUE-2925: Added support to get dbt files via s3, gcs and http sources ( #3736 )
2022-04-19 19:36:24 +05:30
Pere Miquel Brull
91af412cbf
Remove airflow plugin ( #4217 )
2022-04-19 07:54:27 +02:00
Ayush Shah
4f5c5a9e23
Ingestion - fix pytest workflow errors and modify airflow pipeline ( #4178 )
2022-04-18 11:36:21 +05:30
Ayush Shah
6cea695e23
Added ES mixin - supports Lineage creation ( #4163 )
2022-04-15 23:16:19 -07:00
Ayush Shah
1f3667b46f
Fix Python test ( #4095 )
2022-04-13 21:03:33 +05:30
Mayur Singal
726da97dd8
Fix #3654 : Fixed TypeError in Deltalake ( #3710 )
...
* Fix #3654 : fixed minor bugs
* Fixed table ingestion issues
* Added sample script for deltalake
* Added Hive Metastore support
* Added support for local metastore
* removed code smell
* Resolved review comments
* fixed display data type logic
* Fixed Code Smell and Resolved conflicts
* Fixed Delta lake source
* removed set spark
* Updated prepare method
* id fix
* Metatore exception added
2022-04-11 10:45:59 +05:30
dvecerdea
23d68b24d9
Replace sqlalchemy-trino with trino[sqlalchemy] ( #3800 )
2022-04-01 12:12:03 -07:00
Pere Miquel Brull
3b7b65d46c
Upper bound click req ( #3724 )
2022-03-29 11:28:41 +05:30
Pere Miquel Brull
1c21abebdd
Fix #3645 - Revisit lineage for Airflow 1.10.15 ( #3696 )
...
Fix #3645 - Revisit lineage for Airflow 1.10.15 (#3696 )
2022-03-28 08:33:02 +02:00
Mayur Singal
3aa8755a7f
Fix #3465 : Added trino version ( #3466 )
2022-03-16 09:00:08 -07:00
Ayush Shah
28a1918ed8
Fixed Looker - Dashboard and Dashboard Elements ( #3441 )
...
* Looker fix
* Update looker.py
2022-03-16 11:52:29 +05:30
Onkar Ravgan
ad04136847
Fix #3095 : Added support for private key ( #3408 )
...
* ISSUE-3095: Add support for private key in snowflake
rebase main branch
* Fix #3095 : Added support for private key
* Fix #3095 : Added support for private key and review changes
* Fix #3095 : Added support for private key and review changes
* changed direct assignment of dictionary
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-03-15 18:19:02 +05:30
Pere Miquel Brull
130bbb0c5c
Fix #3104 - Remove unused imports with pycln ( #3370 )
...
Fix #3104 - Remove unused imports with pycln (#3370 )
2022-03-14 06:59:15 +01:00
Ayush Shah
7c84f063e6
Minor Fixes - Ingestion - Datetime format, Looker fix ( #3365 )
...
* Minor Fixes - UTC timezone for usage, tag method modification, looker fix
* Modified core version
* Snowflake Usage Result limit and Makefile ignore env fix
2022-03-11 13:31:42 +05:30
Akash Jain
211a8a55e8
feat: Prepare 0.9.1-SNAPSHOT ( #3297 )
...
* prepare snapshot version for 0.9.1
* prepare dev version for ingestion 0.9.1
* prepare dev version for ingestion-core 0.9.1
* prepare Docker CI workflows for 0.9.1
2022-03-09 12:08:42 +05:30
Pere Miquel Brull
a9290bf1a0
Fix #3181 - Align ingestion requirements
...
Fix ES and bump down redash (#3182 )
2022-03-06 09:00:58 +01:00
codingwithabhi
ad341e8259
fix ISSUE-3137: updated-redash-library-and-function ( #3157 )
2022-03-04 12:02:08 -08:00
Ayush Shah
c3cc685663
Excluded db2 from all plugins as it was creating issues with docker image creation ( #3097 )
2022-03-03 14:55:40 +05:30
Pere Miquel Brull
71207de362
Fix #2875 - Profiler API Sink ( #3011 )
...
Fix #2875 - Profiler API Sink
2022-03-02 16:46:28 +01:00
Alberto Miorin
fe5618c8f1
Fix #3037 : metadata --version doesn't work ( #3038 )
2022-03-01 12:19:36 +01:00
codingwithabhi
06d9329ae3
Atlas connector ( #2855 )
...
* Fix #1161 : Add Atlas Connector
* Atlast Decimal type handled
* Atlas col_data_type handled
* Atlas columns exception handling implemented
* atlas-services-added-into-single-file
* import-updated
* authentication-fix
* atlas-connector-services-added
* code-smell-removed
* code-smell-removed
* code-smell-removed
* file-exist-check-added
* updated-mapping-json-logic
* pr-comment-changes-completed
* atlas-linegae-added
* atlas-linegae-added
* pr-changes-done
* code-formatted
* bug-resolved
* topic-bug-resolved
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-03-01 11:50:14 +05:30
Ayush Shah
568845b65c
Power BI connector ( #3019 )
...
* Power BI Dashboard and Tiles added
* Added Powerbi dependency
* Modified Powerbi debug logs
* Bump up _version.py
* Modified Power BI - added failure status, resolved comments
2022-03-01 10:09:35 +05:30
Mayur Singal
c49af971a7
Fix #2984 : added azure sso auth ( #3002 )
2022-02-28 11:44:28 -08:00
Pere Miquel Brull
00552f6d36
Fix #3012 - Dependency breaking Airflow Container ( #3015 )
...
Fix #3012 - Dependency breaking Airflow Container
2022-02-28 17:11:18 +01:00
Ayush Shah
412d61a875
Fix Okta Authentication and Validation - Ingestion ( #2955 )
2022-02-26 08:19:36 -08:00
Mayur Singal
e79be68bea
Fix #2937 : added clickhouse usage connector ( #2987 )
2022-02-26 08:18:49 -08:00
Mayur Singal
df98931ed4
fix #2809 : Auto populate View Lineage to Tables ( #2977 )
2022-02-24 08:28:40 -08:00
Mayur Singal
6a2180e975
Fix #2539 : redshift lineage mapping ( #2828 )
...
* py_format
* Fixed code smells
* used util method to ingest lineage
* removed sample sql query
2022-02-24 13:20:09 +05:30
codingwithabhi
2ef2edbf17
Bigquery lineage mapping ( #2849 )
...
* lineage-boilerplate-code-added
* bigquery-lineage-completed
* lineage-code-optimised
* code-cleaned
* code-smell-fixed
* code-refined
* added-database-name-in-fqdn
* code-smell-removed
* exception-handled
* removed-print-statement
* formatted-setup.py
* added-sqllineage-version
* included-valid-type-none
* debug-added-in-logger
* lineage-logic-updated
* updated-lineage-params
* global-config-variables-removed
* return-type-updated-in-entity
* updated-conditionin-from-entity
* imported-sqllineage-inside-method
* code-smell-removed
* code-formatted
2022-02-24 11:59:00 +05:30
Ayush Shah
959d441661
Glossary Entity Added - Ingestion fix ( #2794 )
...
* Added Glossary
* Modified Glossary: Mixin and suffix
* Glossary Mixin Methods
* Sample Data and Mixin for Glossary
* Modified comments
* Glossary Mixin Updated
* Added Glossary import
* Import issues fixed, removed upper limit for jinja2 as it conflicted
* Removed Glossary Import
2022-02-18 12:50:55 +05:30
Pere Miquel Brull
f304d290b4
Fix #2751 - Init ORM Profiler ( #2831 )
...
* ORM Profiler skeleton
* Fix table name within service
* Add license
* Prepare custom types
* Fix converter
* Compute stddev only on numeric
* Prepare smart registries
* Update tests
* Update results retrieval
* Fix composed metrics result
* Format
* Add missing type
* Add _label decorator
* clean readme
* clean readme
* Filter types when profiler runs not allowed metric types
* Fix null ratio
* Add proper type
* RuleMetric skeleton
* Prepare table metrics
* Update simple profiler
* Format
* Define test expression grammar and node visiting
* Unify metric registry
* Prepare validation core
* Add grammar lib
* Add safe get
* Format
* Allow decimals in grammar
* Test validation conversion
* Fix validation conversion and test
* Rename to row_number
* Update READMEs
* Format
* Row number naming
* Fix rename
2022-02-18 07:48:38 +01:00
Mayur Singal
419448a28f
Issue 2727: Added Databricks Connector ( #2745 )
...
* Issue #2727 : Added Databricks Connector
* optional db and licence info
* make core_bump_version_dev
* host_port fix
* fixed service_type
* change package to sqlalchemy-databricks
* Update databaseService.json
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-02-17 11:22:13 +05:30
codingwithabhi
088dba3a1f
azuresql-config-added ( #2711 )
...
* azuresql-config-added
* rebased-with-main
* tables-ingestion-added
* duplicate-files-removed
* Update azuresql.py
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-02-17 09:23:07 +05:30
Mayur Singal
5e4cd516bf
issue #2712 : Added SingleStore Connector ( #2747 )
2022-02-15 21:11:46 -08:00
Mayur Singal
88afb17ee8
issue #2611 : added clickhouse connector ( #2679 )
...
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-10 08:32:01 -08:00
Ayush Shah
4f5f63b226
Issue-2672: Added support for personal access token name and secret and upgraded tableau version ( #2682 )
2022-02-08 23:02:56 +05:30
codingwithabhi
66c3f43bbb
Fix #2537 : Ingestion Connector - DynamoDB
2022-02-07 10:52:11 -08:00
Mayur Singal
2d25da149d
Fix #2626 : add db2 connector ( #2661 )
...
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-07 10:49:59 -08:00
Pere Miquel Brull
08a46814e6
Fix #2625 : Use pytest 7.0.0 ( #2640 )
2022-02-06 14:41:56 -08:00
Ayush Shah
21491559de
Fix #2587 : Bigquery ADC and Policy Tag implementation
2022-02-06 10:36:10 -08:00
Pere Miquel Brull
c2ad7f6373
Fix #2317 : Override Pipeline tasks from client
2022-02-06 10:17:08 -08:00
Pere Miquel Brull
6ea6f0c43d
Fix #2383 - Multiple validation workflows ( #2500 )
...
* Clean duplicated validation
* Move black dep to dev
* Fix black version for ingestion core
* Validate core generated in the same workflow
2022-01-30 12:26:41 -08:00
