Steve Russo
3a3f248edd
Issue 5592: Add Databricks Pipeline Connector ( #9554 )
...
* feat: update databricks_client
* feat: create pipelineconnection & connection
* test: add unit test
* feat: add examples file
* fix: remove unused imports
* style: fix python style
* fix: code review adjustments
* fix: remove unneeded schema ref
* fix: format requirements
* fix: change context calls
* test: add pipeline_status test
* format: pylint format change
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-01-05 14:14:12 +01:00
Ayush Shah
1d930ad14b
Fix security vulnerability ( #9580 )
2023-01-05 12:36:00 +05:30
Mayur Singal
0a34e18e26
Fix #9562 : Add duration in usage ingestion ( #9564 )
2023-01-05 06:33:07 +00:00
NiharDoshi99
9145054dc6
Fix: refractor datalake datatypes and s3 for parquet ( #9578 )
...
* Fix: refractor datalake datatypes
* Fix: s3 files for parquet
2023-01-04 22:12:00 +05:30
Milan Bariya
625c15c5d3
Fix: Table description error for table ( #9584 )
...
* Fix: Table description error for table
* add schema name in a query
* Add: Function for Column comments
* Mssql qery import path changed
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-01-04 21:57:56 +05:30
Mayur Singal
67ea711319
Fix #8422 : Fix False usage query count ( #9587 )
2023-01-04 14:01:07 +05:30
Pere Miquel Brull
1f49ea46b5
Fix #9149 - Postgres foreign tables & reorganize queries for sources ( #9588 )
...
Fix #9149 - Postgres foreign tables & reorganize queries for sources (#9588 )
2023-01-04 08:50:53 +01:00
Mayur Singal
0a7fe62552
Redshift: varchar datatype fix ( #9572 )
2023-01-03 14:15:42 +05:30
8vicat
7e7528e13b
Improve SQL lineage parsing ( #9537 )
...
* CTEs support
* Supports inline comments
* Expands `*` in sources
2023-01-03 09:27:45 +01:00
NiharDoshi99
8f69386076
Fix: refactor get_column_type ( #9559 )
...
* Fix: refactor get_column_type
* Fix: added changes as per comments
* Fix: pylint
* Fix: minor changes
* Fix: minor changes
2023-01-03 10:28:38 +05:30
Pere Miquel Brull
7f21a7bced
Fix #8088 - Restructure source connections & clients ( #9545 )
2023-01-02 13:52:27 +01:00
Onkar Ravgan
420fa1181a
Fixed bigquery e2e test ( #9555 )
2023-01-02 13:08:47 +05:30
Onkar Ravgan
dc157ee286
Fixed ingestion tags with new tags API ( #9552 )
...
* Fixed ingestion tags with new API
* fix cypress failures
* wait for tags API
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-12-31 07:33:02 -06:00
Ayush Shah
c08e805090
Revert Change ( #9534 )
...
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-12-28 14:05:18 +01:00
Chirag Madlani
bf6fc5f93a
prepare(release) next release ( #9479 )
...
* prepare(release) next release
* airflow typo
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-12-27 20:15:46 +05:30
Pere Miquel Brull
a4521fd664
Fix #6562 - Sources have their own package ( #9521 )
...
Fix #6562 - Sources have their own package (#9521 )
2022-12-27 15:00:22 +01:00
Mayur Singal
e7133d1e86
Fix Elasticsearch reindex through CLI ( #9528 )
...
* Fix Elasticsearch reindex through CLI
* Fix Elasticsearch reindex through CLI
2022-12-27 12:06:05 +01:00
Suresh Srinivas
758c976cba
Fixes #9259 Change Tags APIs to conform with rest of the APIs ( #9260 )
2022-12-26 12:32:17 -08:00
Nahuel
2c43ebba6f
Fix#9448: Add ES volumes ( #9506 )
...
* Add ES volumes
* Fix run_local_docker script
* Fix error run_local_docker script
* Update Es volumes in docker-compose files
2022-12-23 17:33:30 +01:00
Pere Miquel Brull
ec6ebb3694
Organise dynamic importers ( #9513 )
...
Organise dynamic importers (#9513 )
2022-12-23 12:17:06 +01:00
Mayur Singal
5be0634d86
Dagster YAML & Test connection Fix ( #9489 )
...
* Dagster YAML Fix
* Test connection fix
2022-12-22 11:00:27 +00:00
Sriharsha Chintalapani
b4e274c8d5
Add schemaText to topic Index ( #9482 )
...
* Add schemaText to topic Index
* Remove ChangeDescription from TopicIndex
2022-12-22 09:42:51 +01:00
Pere Miquel Brull
0e0779a926
Fix dict type hint for older versions ( #9480 )
2022-12-22 07:40:11 +00:00
Vj-L
f3855a1909
Added the default volumes for ElasticSearch ( #9452 )
...
* Added the default volumes for ElasticSearch
* Added docs for ES Volumes
* Updated as per the comments
* Updated as per the comments
* Added the docs for bare-metal upgrade
* Apply suggestions from code review
* Update docker/local-metadata/docker-compose.yml
* Pylint check addressed
Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-12-21 18:29:49 +00:00
Mayur Singal
e10fff89b4
Messaging Broker Minor Fix ( #9468 )
2022-12-21 17:41:26 +01:00
Sriharsha Chintalapani
b9242c1390
Fix topic indexes and migrations ( #9439 )
...
* Fix topic indexing
* Fix topic db migrations
* Fix topic db migrations
* Fix topic db migrations
* Fix test; add missing topic indexes on python side; fix ranking for tables using usage count
2022-12-21 10:33:12 +01:00
Teddy
9e01fe0636
feat(CLI): added migration logic cmd ( #9437 )
2022-12-21 07:55:18 +00:00
NiharDoshi99
9a3d599f30
Azure datalake minor changes ( #9407 )
2022-12-21 10:28:41 +05:30
NiharDoshi99
c72938085d
Fix:minor changes in atlas ( #9418 )
2022-12-20 20:44:57 +05:30
Teddy
cadfb68826
fix(profiler): bugs in logic to fetch metrics ( #9423 )
2022-12-20 14:23:44 +00:00
Onkar Ravgan
c1fdc59e84
Added DBT Workflow fixes ( #9419 )
...
* Added DBT fixes
* Addressed review comments
2022-12-20 19:06:01 +05:30
Mayur Singal
07aed10a7e
Snowflake E2E Tests ( #8781 )
2022-12-20 18:00:26 +05:30
Ayush Shah
2bf5eb9051
fix 7995: profileSample % and row number ( #9104 )
2022-12-20 14:55:11 +05:30
Pere Miquel Brull
6f66431587
Fix OM backup ( #9395 )
...
Fix OM backup (#9395 )
2022-12-20 08:23:31 +01:00
Mayur Singal
497eca70ab
Fix DBT HTTP source error ( #9397 )
2022-12-19 14:55:31 +01:00
Milan Bariya
a3495c67f0
Databricks improve code and doc update ( #9394 )
...
* Databricks improve code and doc update
* Change base on comments
* Fix pytest fail
2022-12-19 18:53:10 +05:30
Mayur Singal
425a50bc3a
DBT ingestion improvements ( #9383 )
...
* DBT ingestion improvements
* use fqn build instead of join
* fix linting
* fix linting
2022-12-19 06:37:54 +01:00
Milan Bariya
47a1c8015a
Fix: Databricks table type issue ( #9385 )
...
* Fix: Databricks table type issue
* Fix: Databricks table type issue
2022-12-18 15:31:13 +01:00
Mayur Singal
c06dbfd22e
Fix caching issue for BQ ( #9352 )
2022-12-17 09:16:16 -08:00
Milan Bariya
e1d347c714
Minor change in backup function ( #9370 )
...
* Minor change in backup function
* Minor change in backup function
2022-12-17 17:43:28 +05:30
Pere Miquel Brull
3b7ae73473
Airflow e2e integration test ( #9363 )
...
* Prep airflow operator integration tests
* Add integration test to Makefile
2022-12-16 19:52:12 -08:00
Teddy
b3160130f2
Fixe Ownership and Tier Processing ( #9356 )
...
* fix(DI): Owner perc. and tiers
* fix(DI): pylint
2022-12-16 19:51:54 -08:00
Mayur Singal
5164a55341
Fix Mysql e2e test ( #9364 )
2022-12-16 19:39:02 -08:00
Teddy
3856d63f3a
Fixes #5522 by implementing timeout for the profiler ( #9340 )
...
* feat(Profiler): Added timeout logic in the profiler workflow
* feat(Profiler): Added additional input in workflow UI
* fix(Profiler): timeout text blob in UI
* fix(Profiler): Return profile_results outside the context manager
* Fix(Profiler): Added localizer for timeout label and message
* Update ingestion/src/metadata/interfaces/sqalchemy/sqa_profiler_interface.py
Co-authored-by: Nahuel <nahuel@getcollate.io>
Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-12-16 17:01:12 +01:00
Pere Miquel Brull
7aed6d340b
Fix #9276 - Fix dump syntax for backup ( #9341 )
...
* Update docs
* Clean dump
* Clean dump
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-12-16 16:59:13 +01:00
Nahuel
a2b34dd0f4
Fix: Update Ingestion docker images and fix python libraries dependencies ( #9342 )
...
* Update Ingestion docker images and fix python libraries dependencies
* Install also apache-airlfow-providers-http
2022-12-16 14:46:25 +00:00
NiharDoshi99
3190ac2516
Fix: making token as optional field ( #9314 )
2022-12-16 14:17:47 +00:00
Milan Bariya
a15600e4e6
Fix: Metadata backup Azure blob issue ( #9328 )
...
* Fix: Metadata backup Azure blob issue
* Fix: Metadata backup Azure blob issue
* Fix: Metadata backup Azure blob issue
* Doc update for Backup
* Doc update for Backup
2022-12-16 18:59:00 +05:30
Ashish Gupta
7c181a5a89
Dbt worflow ( #9302 )
2022-12-16 17:00:28 +05:30
Ayush Shah
50d1538374
Fix failure in GCS datalake ingestions ( #9336 )
2022-12-16 08:37:39 +00:00
