1614 Commits

Author SHA1 Message Date
Onkar Ravgan
09fb69b68d
Fixed objectstore import (#11144) 2023-04-19 21:39:29 +05:30
Teddy
0f7d9699ad
Fix metrics filtering (#11149)
* fix: get column not filtering for metric types when profilerConfig with include columns is set

* fix: run python linting
2023-04-19 14:09:13 +00:00
Milan Bariya
66b25d2f30
Fix: Databricks usage issue (#11143) 2023-04-19 19:25:17 +05:30
Ayush Shah
ca861bc06e
Fixes #11137: Mssql Syntax Error + Arithemetic Error (#11138) 2023-04-19 15:08:12 +05:30
NiharDoshi99
1862ba2ba4
Changing behaviour for owners same as description for dashboards (#11118)
* changing behaviour for owners same as description

* fix typo
2023-04-19 12:31:56 +05:30
Pere Miquel Brull
a78a3b4734
Azure datalake metadata ingestion fixes (#11125)
* Add ADLS permissions

* Fix Azure DL ingestion

* Format

* enable decode for json

* fix gcs decode error

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-04-19 07:28:41 +02:00
Pere Miquel Brull
463f242d6b
Add S3 Storage docs & CI validation (#11120)
* Prep CI

* Update Athena

* Update docs for S3 Storage

* Add manifest information
2023-04-19 06:31:55 +02:00
Milan Bariya
7cbe48971d
fix: Redash lineage issue (#11098)
* fix: Redash lineage issue

* change based on comments

* change based on comments

* change based on comments
2023-04-18 21:22:17 +05:30
Teddy
97ff34967a
Fix histogram bin creation (#11105)
* fix: bin creation + pass full table name for mysql median computation

* fix: ran linting for python

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-04-18 14:49:21 +02:00
Nahuel
22ce62e13b
Fix: Add Redash E2E test (#11091) 2023-04-18 12:52:38 +05:30
Mayur Singal
857ddeab1e
Ingestion: Metabase Unit Tests (#11080) 2023-04-18 09:08:17 +05:30
Mayur Singal
e7013b481a
Improve Redshift Query (#11082) 2023-04-18 00:51:06 +05:30
Teddy
b04f7225f8
fix: column retrieval for SNOWFLAKE (#11090) 2023-04-17 14:36:58 +00:00
NiharDoshi99
7e4b63997b
Changes for no columns and added owner for table (#11086)
* changes for no columns and added owner for table

* added pydantic model for owners
2023-04-17 18:42:30 +05:30
Onkar Ravgan
b82abd5047
Fixed tableau url (#11071)
* Fixed tableau url

* review comments and tests

* changes to remove host-port addition from the UI for dashboard and chart urls

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2023-04-17 11:20:11 +02:00
Mayur Singal
199fe8753a
Fix Top Level Imports (#11075) 2023-04-14 17:18:38 +00:00
Pere Miquel Brull
ae984d1808
Handle impala auth mechanism (#11074) 2023-04-14 18:04:42 +02:00
Teddy
a7d98dddda
Fixes #9632 - Add Profiler Support for BQ Arrays of Structs (#11059) 2023-04-14 19:29:26 +05:30
Nahuel
ea70580aff
Fix#10584: Tableau E2E and docs (#11054) 2023-04-14 19:28:31 +05:30
Nahuel
c596ad2006
Doc: Update Kafka troubleshooting section (#11067) 2023-04-14 13:35:58 +02:00
Pere Miquel Brull
5152db488d
Add partition columns details (#11062) 2023-04-14 13:06:56 +02:00
Pere Miquel Brull
8f3cf712ff
Catch ConnectionError (#11065) 2023-04-14 13:06:47 +02:00
NiharDoshi99
6b8b9a179b
airflow support for higher versions (#11001) 2023-04-14 16:04:42 +05:30
Onkar Ravgan
bc6ce22a2b
Added oneof selection for tableau auth types (#11049)
* Added tableau oneof fields

* Fixed pytests

* fixed field in test

* Handle tableau auth converter

* Fixed java tests and imports

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-04-14 13:49:36 +05:30
Teddy
77b94f9ebb
fix: rename tests endpoint to dataQuality/<specificity> (#10970) 2023-04-14 00:14:49 -07:00
Pere Miquel Brull
4c9690b0c2
Add Looker Sample Data (#11047) 2023-04-13 18:05:06 +02:00
Milan Bariya
dda5963e34
Data insights with metadata command-line (#11045)
* Data insights with metadata command-line

* Python Checkstyle

* Python Checkstyle
2023-04-13 13:42:07 +00:00
Pere Miquel Brull
dcbb77f46c
Looker - Update GitHub creds for UI form (#11019)
* Update GitHub creds for UI form

* Add filters
2023-04-13 06:05:02 +02:00
Pere Miquel Brull
47cef52fa8
Handle container parents (#11026) 2023-04-12 18:36:04 +02:00
Nahuel
673573a512
Fix#10584: Add filters for data models (#11008)
* Fix#10584: Add filters for data models

* Minor UI error

* fix unit test

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2023-04-12 17:36:01 +05:30
Onkar Ravgan
dbd8da6a29
Added logic for cleaning up schemas (#10998)
* Added logic for deleting schemas

* added UI changes to allow soft deletion and restoration of database schemas

* fixed failing cypress test

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2023-04-12 16:50:44 +05:30
Onkar Ravgan
1b930fa6f7
Added enum for ssl and auth (#11016) 2023-04-12 10:57:31 +00:00
Mayur Singal
ae6683862f
Fix #10999: Add support for materialized views in oracle (#11005) 2023-04-12 16:21:27 +05:30
Ayush Shah
288358952b
Fixes 8264: add docs around adding taxonomy project id (#11018)
* add docs around taxonomy project id

* Add taxonomyProjectID to bigquery.yaml
2023-04-12 16:00:05 +05:30
Pere Miquel Brull
b5cb1d464a
Deprecate location and old storage service (#11004)
* Deprecate location and old storage service

* Format

* Fix test

* Refactor

* Clean location

* Rename object store to storage

* Rename object store to storage

* Rename object store to storage

* Format

* Format

* Refactor object store for storage

* Refactor object store for storage

* Rename object store to storage

* Fix test

* Fix test

* Format

* chore(ui): change Objectstore to  Storage

* Fixes

* Fix test

* Remove storage service from Glue cypress

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-04-12 11:44:46 +02:00
Schlameel
4c3f142a2c
Fixes #6340: Implement Median support for MySQL (#10962)
* ISSUE #6340: Implement Median support for MySQL
- Added code to existing function that previously returned None
- Important - Tested only external to OM.
- Performance tested - 1/7th the speed of other solutions. Not impacted by index.

* ISSUE #6340 - Implement median support for MySQL
Changed to remove setting user variable in expression per Teddy

* ISSUE #6340 - Implement median support for MySQL
Formatting
2023-04-12 08:07:36 +02:00
Ayush Shah
9d11029ec8
Fixes 10351: Fixes Metrics Computation, Samping, test suites and partioning (#10603)
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-04-11 20:58:31 +05:30
Mayur Singal
75aa42fe43
Use metabase models in client (#10995) 2023-04-11 18:28:35 +05:30
Pere Miquel Brull
09b283818d
Rel to #10927 - Looker DataModel (#10945)
* Organise calls

* Prepare skeleton

* Add looker model handling

* Parse files as sql

* Handle labels

* Linting

* Format

* Fix version

* Also check the API for explore lineage
2023-04-11 08:44:00 +02:00
Ayush Shah
02a52b3ddf
Fix table name with underscore (#10980) 2023-04-10 11:41:36 +00:00
Onkar Ravgan
3c32658d7a
Fixed dbt logging (#10977)
* Fixed dbt logging

* Fixed pytests

* Fixed pycheckstyle
2023-04-10 10:04:25 +00:00
Mayur Singal
9f2b10f6e2
Fix #10858: Fix dashbaord & pipeline url (#10957) 2023-04-10 09:48:49 +00:00
Mayur Singal
3d345f9b37
Fix #10273: Parse nested json for datalake (#10956) 2023-04-10 14:58:02 +05:30
Ayush Shah
6ff3557917
Issue 10975 - fix workflow break at none (#10976) 2023-04-10 09:05:38 +00:00
Teddy
203f17fa49
fix: add rollback logic to custom sql query to exit out of transaction (#10968) 2023-04-07 14:46:05 +02:00
Keith Sirmons
27984c25f3
Fixes 10959: MSSQL profile has error when averaging large integer column (#10960)
* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* removed print statements and ran make py_format

* updated to fix some pylint errors.
imported Dialects to remove string compare to "impala" engine

* moved huge comment into function docstring.
This comment shows us the sql to get quartiles in Impala

* added cast to decimal for column when running average in mean.py

* fixed lint error
2023-04-07 06:48:18 +02:00
Hemal Mamtora
0206af0303
[WIP] Fixes #10721: created client file for metabase and moved get request to client file (#10838)
* ingestion/metadata - created client file and moved get request to client file

* moved code to client.py, created models.py

* added, self to MetabaseClient constructor

* added status_code in metabase client functions and handled Optional datatype in metabase metadata functions

* moved api calls to client

* passing service_connection and metabase_session from constructor to MetabaseClient

* renamed client to metabaseClient to avoid rewriting DashboardServiceSource.client

* moved post request from connection.py to client.py

* run make py_format

* moved client code to init constructor

* added Dashboard Model for Metabase

* updated client type to MetabaseClient and used get_dashboards_list() function in custom_executor

* updated client type to MetabaseClient and used get_dashboards_list() function in custom_executor

* make py_format

* pylint issues resolved

* pylint and run make py_format

* calling init method in Metabase source

* removed init method
2023-04-06 19:26:52 +02:00
Keith Sirmons
42000053aa
Fixed Issue #10943: Impala query engine metadata ingestion and median function profiler (#10944)
* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* removed print statements and ran make py_format

* updated to fix some pylint errors.
imported Dialects to remove string compare to "impala" engine

* moved huge comment into function docstring.
This comment shows us the sql to get quartiles in Impala
2023-04-06 18:07:42 +02:00
Nahuel
62b88b0404
Fix#10584: Add lineage for Tableau data models (#10923)
* Refactor Tableau ingestion

* Add lineage from DB -> DataModel -> Dashboard

* Address PR comments

* Address PR comments
2023-04-06 07:28:41 +02:00
Pere Miquel Brull
c53c7b680d
Rel #10828 - Update container index and fix airflow ingestion (#10873)
* Update container index and fix airflow ingestion

* Fix lint

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-04-06 07:09:38 +02:00