Mayur Singal
05dc42bdb8
Fix #11808 : Handle lineage for single db sources in superset ( #11933 )
2023-06-09 12:43:06 +05:30
Onkar Ravgan
caabe89f9c
Centralize tags ingestion logic ( #11880 )
2023-06-09 10:45:53 +05:30
Onkar Ravgan
66aa28a3ee
Added check for parent nodes ( #11911 )
2023-06-08 10:50:11 +05:30
Teddy
8dfa640a82
fix: check for empty description strings ( #11914 )
2023-06-07 12:49:29 +02:00
Mayur Singal
9e6e00caf9
Fix #9282 : Support Databricks Unity Catalog ( #11896 )
2023-06-07 11:50:31 +05:30
Sriharsha Chintalapani
2b2602b76b
Fix column resize to limit to 128 chars ( #11889 )
...
* Fix column resize to limit to 128 chars
* Fix pyformat issue
* replace column_name with col_name
---------
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-06-07 07:24:08 +02:00
Teddy
b0bfbad9da
fix: disable sys. metrics for snowflake ( #11900 )
2023-06-06 20:58:12 +00:00
Pere Miquel Brull
b68d063694
Update Test Connection Definition FQN ( #11861 )
...
* Update Test Connection Definition FQN
* chore: use object literal for string concatenation
* Fix test
* Update Test Connection Definition FQN
* Update Test Connection Definition FQN
* Update Test Connection Definition FQN
* Fix test
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-06 15:22:32 +02:00
Teddy
4b9f213dbf
Fixes Issue #11863 - Add Status to DQ ( #11893 )
...
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.
* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity
* feat: add status object to test case results
* feat: ran python linting
* feat: fixed update to
2023-06-06 10:09:16 +00:00
Teddy
721869428e
Revert "Fixe Issue #11863 - Add Status logic for test case results ( #11881 )" ( #11892 )
...
This reverts commit 06735fe8dbaac5b267c9a2cf744ca154f88a9247.
2023-06-06 09:56:12 +02:00
Teddy
06735fe8db
Fixe Issue #11863 - Add Status logic for test case results ( #11881 )
...
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.
* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity
* feat: add status object to test case results
* feat: ran python linting
2023-06-06 09:45:49 +02:00
Ayush Shah
236141d9df
Fixes 11825 - Add dataset description on schema level - bigquery ( #11878 )
2023-06-06 07:28:47 +00:00
Ayush Shah
65f370e4aa
Rename GCS to GCP ( #11812 )
2023-06-06 11:57:00 +05:30
Onkar Ravgan
e21f38c5fd
handled null upstreamcol values ( #11835 )
2023-06-02 16:55:28 +05:30
Teddy
d0cffdcd66
Fixes Issue #11438 - Implement threshold and startegy for custom SQL ( #11847 )
...
* feat: Add threshold and strategy logic on the custom SQL object test
* feat: ran python linting
* feat: added safety checks for custom sql query
* feat: ran python linting
2023-06-02 09:41:31 +02:00
Teddy
c98a15ca19
Fixes #11705 - Update ingestion and backend to match new DQ flow ( #11836 )
...
* feat: refactor ingestion flow logic
* feat: ran python linting
* feat: update tests to match new workflow
* feat: ran python linting
* feat: update sample data test suite name
* feat: Added backend logic to support logical and executable test suites
* feat: clean up java and json code
* feat: added sample data for logical and executable test suites
* feat: remove executable from CreateTestSuite
* feat: ran python and java linting
* feat: added README info for data quality structure
* skipping cypress to keep main green
* fixed typescript type issue
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-06-01 23:19:13 -07:00
Pere Miquel Brull
fdeea71671
Fix Looker explore git link & Add BitBucket reader ( #11837 )
...
* Add looker test connection step
* Add looker test connection step
* Update Credentials
* Fix explore link and add bitbucket reader
* Format
* Fix test
* Fix spline linting
* Fix import
2023-06-02 07:19:32 +02:00
Mayur Singal
b7a1066f5b
Fix #11781 : Fix atlas lineage issue ( #11832 )
2023-06-01 12:02:43 -07:00
Ayush Shah
8e57b165a6
Fix LineTooLong ( #11833 )
...
* Fix LineTooLong
* Add Timeout ( fix lint )
2023-06-01 09:40:18 +05:30
Pere Miquel Brull
11c07ee8ab
Fix #11516 - SAP Hana Connector ( #11777 )
...
* SAP Hana skeleton
* Add SAP Hana Connector
* Fix ingestion and docs
* Prep SAP Hana Profiler
* Linting
* Update index.md
* Revert: Update index.md
---------
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-31 16:00:31 +02:00
Mayur Singal
ae386d2e2b
Add Table Sample Data During Sample Data ingestion ( #11773 )
2023-05-31 12:23:45 +00:00
Mayur Singal
9e437fceec
Fix redshift view lineage parsing issue ( #11702 )
...
* Fix redshift view lineage parsing issue
* address review comments
2023-05-31 12:10:39 +00:00
Mayur Singal
b57bbf833f
Fix #11572 : Glue Support Partition Columns & Use Pydantic Models ( #11776 )
2023-05-31 12:03:34 +00:00
Pere Miquel Brull
e02fc231da
Improve logging for test connection workflow ( #11794 )
2023-05-30 10:36:10 +05:30
Chirag Madlani
7adc291364
fix(ui): circular deps for entityReference.json ( #11760 )
...
* fix(ui): circular deps for entityReference.json
* Fix circular Dependency python
* Cap Delta Spark version
---------
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-26 18:02:21 +05:30
Pere Miquel Brull
5bd1b09402
Add new Looker test connection steps ( #11749 )
2023-05-24 18:01:08 +02:00
Mayur Singal
c44bff8fd3
Fix ValueError: too many values to unpack for col lineage ( #11723 )
2023-05-24 11:30:44 +05:30
Sriharsha Chintalapani
6509a3670a
Fix #11664 : Refactor patch_mixin to use jsonpatch lib ( #11696 )
...
* Fix #11664 : Refactor patch_mixin to use jsonpatch lib
* Migrate to jsonpatch
* Fix nested cols
* Format
* Update patch_description
* Table constraints
* tag
* owner
* column tag
* column desc
* Format
* Format
* Fix log
* Update dbt patch
* Update column fqn
* Fix test
* Fix tests
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-05-23 15:47:11 +02:00
Onkar Ravgan
7589b7eeef
fixed kafka basicaauth bug ( #11729 )
2023-05-23 19:05:01 +05:30
Onkar Ravgan
f677408f20
Fixes 11603: Tableau Data Sources as Data Models ( #11688 )
...
* added initial changes
* Added lineage details
* fixed pylint
* Fixed name
* Fixed empty lists
2023-05-22 18:07:02 +02:00
Teddy
8c50d1af52
Fixes #4565 - Fetch Metrics from System tables ( #11645 )
...
* feat: fetch metrics from system tables
* feat: add permission doc for fetching metrics from system tables
* feat: fix E2E tests to reflect full table row count after table metric update
* feat: ran linting
* feat: fix doc string engine name + function typing
* feat: ran python linting
2023-05-22 09:04:18 +02:00
Teddy
ddbc7fe14d
Fixes #11570 - Add support for BQ Multi-project Profiler ( #11692 )
...
* fix: extracted profiler object from workflow and implemented factory to allow service base logic
* fix: ran python linting
* fix: renamed `base` to `base_profiler_source`
* fix: add logic to set correct database for BQ multi project ID connections
* fix: ran python linting
2023-05-20 14:22:53 -07:00
Mayur Singal
8ef98076ae
Fix view lineage not generated for oracle ( #11648 )
2023-05-19 10:47:55 -07:00
Pere Miquel Brull
0eb2201f94
Restructure NER Scanner internals ( #11690 )
...
* Simplify col name scanner
* Restructure NER Scanner internals
2023-05-19 18:21:01 +02:00
Mayur Singal
fafbfdaeab
Fix postgres view lineage parsing issue ( #11689 )
2023-05-19 17:47:50 +02:00
Ayush Shah
ad7258e7be
Fixes 10949: return Chunks for file formats & Centralize logic for different auth configs ( #11639 )
...
* Centralize Auth and File formats datalake
2023-05-19 18:54:28 +05:30
Pere Miquel Brull
d52d773707
Send encrypted automation workflow ( #11681 )
2023-05-19 15:04:42 +02:00
Mayur Singal
e9992a52a8
Fix #1604 : Add Spline Pipeline Connector ( #11562 )
...
* Fix #1604 : Add Spline Connector
* Add tests & grammer validation
* Spline UI Changes & Docs
* fix pipeline workflow doc
* chore: use common field for dbService name
* chore: use const for beta services
* chore: add service icon
* Update ingestion/src/metadata/ingestion/source/pipeline/spline/metadata.py
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-05-19 14:46:32 +05:30
Pere Miquel Brull
50ad38ea0f
Fix #11548 - Secrets Managers comms with OMeta ( #11602 )
...
* Remove secretsManagerCredentials from backend
* Remove secretsManagerCredentials from backend
* Add secrets manager loader
* Load SM in the ometa client
* Fix tests
* Fix tests
* Fix Lint
* Mock AWS region
---------
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-19 09:43:11 +02:00
Pere Miquel Brull
4626363fd8
Fix parsing for Storage ( #11663 )
2023-05-19 09:36:44 +02:00
Pere Miquel Brull
b60f198674
Fix GitHub reader headers and add docs ( #11675 )
...
* Fix header and add docs
* Format
2023-05-19 09:36:30 +02:00
Chirag Madlani
763b96b8fe
bump(ui): rjsf to v5 ( #11549 )
...
* bump(ui): rjsf to v5
It will fix validation and extra parameter issues
* address comments
fix code smells
fix cypress failure
* revert schema files
* bigquery fix
sonar cloud fix
* fix big query
* fix superset
* fix cypress
* fix cypress for big query
* fix failures
* fix java issue
* revert connectionBasicType.json
create custom SupersetConnection.json
* fix Tableau service issues #11637
2023-05-18 08:03:54 -07:00
Pere Miquel Brull
8795337f88
Clean NER Scanner imports ( #11653 )
2023-05-18 12:53:22 +02:00
Mayur Singal
e4997c3749
Fix #11571 : Support custom database name for glue ( #11631 )
2023-05-18 14:16:56 +05:30
Onkar Ravgan
e6a7ffc57c
Fixes: 11540 Added api_version and domain fields to Salesforce ( #11618 )
...
* Added sales new fields
* fixed image after merge
* Fix Failing Test
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-05-18 10:53:44 +05:30
Pere Miquel Brull
1b90badd0e
Restructure PII processor ( #11640 )
...
* Restructure PII processor
* Restructure PII processor
* Format
2023-05-17 15:58:17 +02:00
Mayur Singal
ea615eed9d
Fix elasticsearch error ( #11635 )
2023-05-17 15:48:20 +05:30
Onkar Ravgan
3d9d4416b7
Fixed incompatible column name for Postgres version 11.6 ( #11536 )
...
* postgres col name on version
* Added dependancy
* Added paranthesis validation
* review comments and tests
2023-05-15 11:48:03 +05:30
Onkar Ravgan
cff403a05a
Validate if tags are created before attaching them to CreateRequest ( #11554 )
...
* Added tags validation
* typo fixed
2023-05-11 16:04:55 +00:00
Pere Miquel Brull
6040078a7c
Validate context in Airflow ( #11551 )
2023-05-11 12:23:43 +02:00
