984 Commits

Author SHA1 Message Date
Teddy
6f5d88ff63
tests: Added E2E tests for redshift (#11169) 2023-04-21 08:55:54 +02:00
Mayur Singal
7c1fe5202d
Fix Metabase E2E Tests (#11156) 2023-04-20 13:22:11 +00:00
Mayur Singal
dd754d586e
Metabase E2E Test & docs (#11126) 2023-04-20 00:50:23 +05:30
Teddy
0f7d9699ad
Fix metrics filtering (#11149)
* fix: get column not filtering for metric types when profilerConfig with include columns is set

* fix: run python linting
2023-04-19 14:09:13 +00:00
Onkar Ravgan
932ffa5fa0
Added Powerbi E2E and docs (#11130) 2023-04-19 11:49:20 +05:30
Milan Bariya
7cbe48971d
fix: Redash lineage issue (#11098)
* fix: Redash lineage issue

* change based on comments

* change based on comments

* change based on comments
2023-04-18 21:22:17 +05:30
Teddy
07b74ce8e8
fix: remove count of results processed (#11115) 2023-04-18 14:00:14 +00:00
Teddy
c415f04eb7
Issue 10805 profiler e2e mssql (#11096)
* feat: extracted builder logic into factory and cleaned up comments

* feat: added E3E logic for time based profiler partition + logic for testing detailed profiler results
2023-04-18 11:56:16 +02:00
Nahuel
22ce62e13b
Fix: Add Redash E2E test (#11091) 2023-04-18 12:52:38 +05:30
Mayur Singal
857ddeab1e
Ingestion: Metabase Unit Tests (#11080) 2023-04-18 09:08:17 +05:30
Onkar Ravgan
b82abd5047
Fixed tableau url (#11071)
* Fixed tableau url

* review comments and tests

* changes to remove host-port addition from the UI for dashboard and chart urls

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2023-04-17 11:20:11 +02:00
Teddy
a7d98dddda
Fixes #9632 - Add Profiler Support for BQ Arrays of Structs (#11059) 2023-04-14 19:29:26 +05:30
Nahuel
ea70580aff
Fix#10584: Tableau E2E and docs (#11054) 2023-04-14 19:28:31 +05:30
Pere Miquel Brull
5152db488d
Add partition columns details (#11062) 2023-04-14 13:06:56 +02:00
Onkar Ravgan
bc6ce22a2b
Added oneof selection for tableau auth types (#11049)
* Added tableau oneof fields

* Fixed pytests

* fixed field in test

* Handle tableau auth converter

* Fixed java tests and imports

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-04-14 13:49:36 +05:30
Teddy
77b94f9ebb
fix: rename tests endpoint to dataQuality/<specificity> (#10970) 2023-04-14 00:14:49 -07:00
Milan Bariya
42de096f24
Fix E2E for Mysql (#11040) 2023-04-13 09:04:01 +00:00
Pere Miquel Brull
47cef52fa8
Handle container parents (#11026) 2023-04-12 18:36:04 +02:00
Pere Miquel Brull
b5cb1d464a
Deprecate location and old storage service (#11004)
* Deprecate location and old storage service

* Format

* Fix test

* Refactor

* Clean location

* Rename object store to storage

* Rename object store to storage

* Rename object store to storage

* Format

* Format

* Refactor object store for storage

* Refactor object store for storage

* Rename object store to storage

* Fix test

* Fix test

* Format

* chore(ui): change Objectstore to  Storage

* Fixes

* Fix test

* Remove storage service from Glue cypress

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-04-12 11:44:46 +02:00
Pere Miquel Brull
b988f39152
Fix test usage resources (#11014) 2023-04-12 05:46:29 +00:00
Ayush Shah
9d11029ec8
Fixes 10351: Fixes Metrics Computation, Samping, test suites and partioning (#10603)
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-04-11 20:58:31 +05:30
Deepa Rao
147b7c551e
Fixes #10854 - Create/Update Chart Not working as Expected (#10991) 2023-04-11 06:28:01 -07:00
Pere Miquel Brull
09b283818d
Rel to #10927 - Looker DataModel (#10945)
* Organise calls

* Prepare skeleton

* Add looker model handling

* Parse files as sql

* Handle labels

* Linting

* Format

* Fix version

* Also check the API for explore lineage
2023-04-11 08:44:00 +02:00
Onkar Ravgan
3c32658d7a
Fixed dbt logging (#10977)
* Fixed dbt logging

* Fixed pytests

* Fixed pycheckstyle
2023-04-10 10:04:25 +00:00
Mayur Singal
9f2b10f6e2
Fix #10858: Fix dashbaord & pipeline url (#10957) 2023-04-10 09:48:49 +00:00
Mayur Singal
3d345f9b37
Fix #10273: Parse nested json for datalake (#10956) 2023-04-10 14:58:02 +05:30
Nahuel
62b88b0404
Fix#10584: Add lineage for Tableau data models (#10923)
* Refactor Tableau ingestion

* Add lineage from DB -> DataModel -> Dashboard

* Address PR comments

* Address PR comments
2023-04-06 07:28:41 +02:00
Teddy
9b4e9132ae
fixed #9656 - Add support for date type to column values to be between (#10890)
* fix: renamed  to  submodule

* fix: linting

* fix: columnValuesToBeBetween test for date column type
2023-04-04 17:16:44 +02:00
Suresh Srinivas
c8b640674b
10041 part2 - Refactor and cleanup APIs (#10900)
* Use @Tag annotation to group APIs in the swagger documentation.

* Hide internal APIs

* Change API path events/subscription to events/subscriptions

* Change API path from automations/workflow to automations/workflows

* Change API path v1/testCase to v1/testCases

* Change API path v1/testDefinition to v1/testDefinitions

* Change API path v1/testSuite to v1/testSuites

* Rename Kpi and kpi in the documentation to KPI

* Change API path v1/testConnectionDefinition to v1/testConnectionDefinitions

* Update API section in the API documentation

* Fix test failures

* Correctly capitalize Test Cases and Test Suites in API docs
2023-04-03 13:03:48 -07:00
Mayur Singal
752163ac71
Fix #10814: Improve parsing logic for union fields in topic (#10836) 2023-04-01 11:10:05 +05:30
Teddy
ecffd5ffc7
Fixes #10727 (& other minor improvements) (#10856)
* fix: logic for test suite config workflow

* fix: added caching for system metrics (snflk and bq)

* fix: linting

* fix: added tearDown logic for tests suite/case
2023-03-31 16:57:53 +02:00
Schlameel
6d24455738
Fixes 10343: Add methods to update Glossary and GlossaryTerm in Python SDK (#10810)
* ISSUE 10343: Python SDK Glossary and GlossaryTerms
- Added methods to glossary_mixin to PATCH Glossary and GlossaryTerm
- Created in patch_mixin_utils a super class for mixins that PATCH entities
- Moved common Patch enums from patch.py to patch_mixin_utils.py
- Updated imports and super classes for mixins that PATCH entities
- Added tests for Glossary and GlossaryTerm mixins

* ISSUE #10343: Python SDK extensions for Glossary and GlossaryTerms
- Fixed an import
- Fixed two method signatures

* Issue #10343 - Fixed formatting
2023-03-31 16:55:22 +02:00
Pere Miquel Brull
2e2c6a0cdf
Operators directory cleanup (#10834)
* Remove old connectors directory

* Add mssql test for url

* Format
2023-03-30 09:51:21 +02:00
Teddy
e1b193a719
Fixes #6795 - Implement profiler support for struct types (#10817)
* fix: raise more informative error message when service is not found

* fix: profiling for struct table

* fix: linting

* fix: added tests for struct and nestedt struct for get_columns
2023-03-29 10:06:34 +00:00
Onkar Ravgan
4fa98b2703
Updated count for mysql e2e filters (#10816) 2023-03-29 13:09:26 +05:30
Onkar Ravgan
5d6e18dc28
Fix 10642: Mark delete entities and tags toggle (#10695)
* Added mark delete logic

* Final test and optimization

* After merge fixes

* Added include tags for dash pipelines dbt

* added docs and fixed test

* Fixed py tests

* Added UI changes for following newly added fields:
- markDeletedDashboards
- markDeletedMlModels
- markDeletedPipelines
- markDeletedTopics
- includeTags

* Fixed failing unit tests

* updated json files of localization for other languages

* Improved localization changes

* added localization changes for other languages

* Updated mark deleted desc

* updated the ingestion fields descriptions in the ingestion form for UI

* automated localization changes for other languages

* updated descriptions for includeTags field for dbtPipeline and databaseServiceMetadataPipeline json

* fixed issue where includeTags field was being sent in the dbtConfigSource

* Added flow to input taxonomy while adding BigQuery service.

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2023-03-29 12:41:44 +05:30
NiharDoshi99
46afe69811
improvement in pii tagging (#10696)
* improvement in pii tagging

* fix conflict and changes as per comment

* Added confidence field

* changes as per comments

* Apply suggestions from code review

Co-authored-by: Teddy <teddy.crepineau@gmail.com>

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2023-03-28 19:37:48 +05:30
Teddy
fc9b64b52a
Fix histogram labels (#10797)
* fix: added logic to format bin labels for histogram metric

* fix: histogram labels

* fix: added types for arguments
2023-03-28 12:59:45 +02:00
Pere Miquel Brull
78d7dd8789
[WIP] - Test Connection - Prepare the new test connection ingestion+UI logic (#10660)
* Prepare the new test connection ingestion logic

* Update test assert

* Update Test Connection for SQA Sources

* Correct return type and method doc

* Handle decryption

* Non SQA Database Sources

* Add the run_automation script in ingestion-base

* Dashboard Test Connection Changes

* Pipeline, Messagin, MlModel & Metadata Sources

* ui: test connect flow-1

* Unmask connection parameters before sending to Ariflow

* ui: test connect flow-2

* Address review comments and pylint

* pytest fix

* ui: test connect flow-3 (refactoring and style fix)

* ui: test connect flow-4 (fix test connection status logic)

* sync local file

* ui: test connect flow-5 (fix lowercase issue and styling)

* ui: test connect flow-5 (show toast notifications)

* test: add unit test

* ui: test connect flow-5 (update service page test connection button)

* Databrick fix & pytest fix

* pylint

* Update test

* Fix merge

* S3 Test connection

* add style for mandatory step

* sync locales

* chore: add service name in workflow request

* Unmask using original service connection parameters

* Fix test connection unmasking

* Wrap inspector function to eliminate error outside test conn

* Fix linting

* fix:cy test

* Fix linting

* address comment

* refactor and fix connection type casing issue

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-28 06:29:13 +02:00
Pere Miquel Brull
cc3d2b448b
Clean Airflow lineage test (#10745) 2023-03-27 11:46:34 +02:00
Cristian Calugaru
3ffde9a293
Fixes #9064 - Ingestion for S3 bucket containers (#10515)
* Prep skeleton for object store connector

* First of of ingestion piepline for bucket containers.

* addressed PR review comments.

* swaped random with secrets due to security warning.

* linter issues.

* added prefix population for containers.

* more linting

* removed temp file.

* object store unit tests, PR comments and minor refactorings

* docs update and new TestConnection approach.

* linting

* removed isStructured field from containermeta schema

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-03-26 11:35:34 +02:00
Nahuel
ef759c7e88
Fixes#8038: Change how status is handled after running workflow (#10710)
* Change how status is handled after running workflow

* Reset changes in config files

* Add auxiliary Summary class

* Improve failures handling

* Pylint error

* Pylint error

* Show result in table

* Add test

* Fix setup.py

* Add comments
2023-03-24 17:59:06 +01:00
Mayur Singal
bbce9c5aa4
Fix #10489: Handle unknown data types & store raw data type (#10563) 2023-03-23 11:41:29 +05:30
Pere Miquel Brull
ebbd611825
Update counts after #10265 (#10693) 2023-03-22 06:32:18 +01:00
Pere Miquel Brull
e2a2bcc8da
Fix search by email index keyword (#10698)
* Fix search by email index keyword

* Fix search by email index keyword
2023-03-21 20:50:47 -07:00
Pere Miquel Brull
4dbe5e4f5c
Simplify Data Insight workflow builder (#10688) 2023-03-21 14:12:20 +01:00
Nahuel
dd225d1463
Fix: Clean Source and SourceStatus inheritance (#10687)
* Fix: clean source and sourcestatus inheritance

* Remove auth parameters from mysql.yaml

* Fix failing test

* Fix pylint checks

* Add missing docstrings
2023-03-21 12:57:48 +01:00
Schlameel
df855ad8c3
Issue #3809: Add python client for Roles and Policies (#10531)
* Issue #3809: Add python client for Roles and Policies
Includes Tests

* #3809: Add python client for Roles and Policies
- Moved constants to enums in client_utils.py
- Updated all patch methods to utilized new enums
- includes tests

* #3809: Add python client for Roles and Policies
- includes tests
- merged upstream updates and updated to use new enums
2023-03-20 08:42:01 +01:00
NiharDoshi99
4fe713a875
fix: chart url (#10605)
* fix: chart url

* fix: chart url

* changes as per comment

* added test and pydantic for chart url

* changed as per comment

* changed as per comment

* fix python test

* changes as per comment

* fix python test
2023-03-20 06:38:44 +01:00
NiharDoshi99
89301729a2
fix e2e test for mysql (#10650) 2023-03-18 00:35:01 +05:30