Mayur Singal
a316de67af
MINOR: Databricks LocationPath Update Fix ( #18815 )
2024-11-27 18:38:26 +05:30
Teddy
58699063db
MINOR -- Fix DQ Partition Issue ( #18641 )
...
* fix: renamed `random_sample` to `get_dataset` and change dunder method access for SQA Table object
* fix: removed handle_partition decorator
* fix: fixed DQ partition issue + moved to `tablesample` method
* style: ran python linting
* style: fix python format check issues
* feat: added postgres tablesample
* style: ran python linting
* fix: sampling delta
* fix: merge conflicts
* fix: resolved conflicts
* style: ran python linting
* fix: patch orm call in test case
* fix: mock build_table_orm call in tests
* fix: test case failures and errors
* fix: removed unused import
* fix: patch typo
* fix: trino table schema retrieval
* fix: remove tuple context manager for 3.8 test support
2024-11-27 08:50:54 +01:00
Imri Paran
c5171139c3
chore: added data diff to base requirements ( #18789 )
2024-11-26 17:28:22 +00:00
Imri Paran
2e9efe5b82
MINOR: fix: use importer as module ( #18793 )
...
* fix: use importer as module
to allow monkeypatching in collate
* Update i_validator_builder.py
2024-11-26 16:42:18 +01:00
Imri Paran
3e7808ba78
fixed either annotation ( #18778 )
...
* fixed either annotation
* added back Entity
* using `typing_extensions.Annotated` for compatibility with python3.8
* removed bound T for Either
2024-11-26 16:40:32 +01:00
Keshav Mohta
10a7f4ea6c
Fixes #15543 : Implemented ElasticSearch Index Template Ingestion ( #18686 )
2024-11-26 14:42:43 +05:30
Imri Paran
d1b8b9a966
handle import error in param_setter_factory.py ( #18745 )
2024-11-26 07:21:38 +01:00
Suman Maharana
fe5778361c
Minor: fix dbt ingest failed runs ( #18779 )
...
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-11-26 11:38:31 +05:30
Suman Maharana
f40138a56e
Minor: Fix Powerbi refresh token on expire ( #18774 )
...
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-11-25 23:19:30 +05:30
Imri Paran
d6470b7800
MINOR: fix(data-diff): get added columns ( #18694 )
...
* fix(data-diff): get added columns
- use both columns to calculate schema diff
* fix tests
2024-11-25 15:53:50 +01:00
Imri Paran
c164aff561
MINOR: add reportExplicitAny = false for basedpyright ( #18777 )
2024-11-25 14:38:04 +00:00
Mayur Singal
4b9e214078
MINOR: Fix StackTraceError validation error - Superset ( #18740 )
2024-11-25 15:01:59 +05:30
Imri Paran
ee7d043035
[GEN-2109] feat(mongo): added ssl support ( #18731 )
...
* feat(mongo): added ssl support
Added SSL support for MongoDB using the SSL manager.
Attached a video demo.
- [Example repository for setting up mongodb with SSL](https://github.com/sushi30/mongodb-docker-ssl-example )
- [MongoDB TLS documentation](https://www.mongodb.com/docs/manual/tutorial/configure-ssl/ )
* fixed test_doris.py
2024-11-22 08:54:13 -08:00
Suman Maharana
0a374e3872
Minor: Fix Athena location path ( #18750 )
2024-11-22 22:05:05 +05:30
Suman Maharana
b220bdb891
Fix: mstr removed dependency issues ( #18732 )
...
* Fix: mstr removed dependency issues
* fix session still active error
* py_format
* fix tests
* Addressed Comments
* Addressed Comments
* addressed comments
* Addressed comments
* Add constants
* Fix pytests
2024-11-22 21:19:21 +05:30
Keshav Mohta
10bd051039
Fix #18441 : kafka connect password as secret value ( #18742 )
2024-11-22 19:07:08 +05:30
Ayush Shah
17ffdf9850
fix: modify fqn to allow quotes with dots ( #18719 )
2024-11-22 09:33:50 +05:30
Pere Miquel Brull
61021be98a
TEST - Add autoClassification for e2e ( #18722 )
2024-11-21 15:07:04 +01:00
Pere Miquel Brull
2ea442030c
MINOR - Fix import ( #18721 )
2024-11-21 13:12:38 +01:00
Imri Paran
0169aad418
[GEN-970] feat(data-quality): support multiple runtime parameter types ( #18588 )
...
* feat(data-quality): support multiple runtime parameter types
- changed the runtime parameters setter factory to return sets
- add the runtime parameters based on the name of the runtime of the runtime parameter
**NOTE** requires changes on collate side
* empty set for default case
2024-11-21 08:07:33 +01:00
Pere Miquel Brull
6623ab17dc
MINOR - Fallback for testsuite imports ( #18710 )
2024-11-20 17:31:48 +01:00
IceS2
35ce7d7602
MINOR: Update Glossary Term tests ( #18698 )
...
* Update Glossary Term tests
* Remove unused code
* Fix test
2024-11-20 15:24:53 +01:00
Akash Verma
43bbee9075
fix #18278 "Pipeline ingested from Airflow are now not limited to 100" ( #18689 )
2024-11-19 18:47:55 +05:30
Pere Miquel Brull
c68a45e7d8
Create new Auto Classification Workflow ( #18610 )
2024-11-19 08:10:45 +01:00
Ayush Shah
6f1df37ba1
Fixes GEN-1260: Add Validators while creating table to escape special characters ( #18456 )
2024-11-18 15:02:57 +05:30
Sriharsha Chintalapani
88c8fb48f3
Add Edit glossary terms, Edit Tier , Edit Tags as separate permissions ( #18331 )
...
* Add EditGlossaryTerms Permission
* Fix #18330 : Add EDIT_GLOSSARY_TERM permission and enforce EDIT_TIER permisson
* add edit glossary term permission check in UI
* revert EDIT_GLOSSARY_TERMS operation
* Add EDIT_GLOSSARY_TERMS to common operations
* Add EDIT_TIER to common operations
* add default empty array for tags field, as patch calls can run into issues
* Fix tests
* Fix tests
* added glossary terms
* fix conflicts
* fix permission check for data model
* Add EditGlossaryTerms to DataConsumerPolicy
* Add EditGlossaryTerms,EditTier to DataConsumerPolicy
* fix tests
* Fix migrations for EditTier,EditGlossaryTerms
* add edit tier permission to data consumer
* Fix tests
* fix pytests
* missing test_dbt.py
---------
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2024-11-15 10:50:15 -08:00
Suman Maharana
a218bbf5cb
Minor: Fix Mysql cli Update table count ( #18582 )
2024-11-15 14:27:02 +05:30
mgorsk1
3d2dfeb583
feat: use native trino client authentication classes ( #16196 )
...
---------
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2024-11-15 12:54:42 +05:30
Imri Paran
bde6ee4125
MINOR: Data diff sample fix ( #18632 )
...
* fix(data-diff): sampling configuration
handle the sampling condition separately for the 2 tables allowing to apply sampling on columns with mismatching cases
* format
2024-11-15 08:22:13 +01:00
Mayur Singal
75d417d267
MINOR: Fix user search - exclude bots ( #18645 )
2024-11-14 21:25:21 +05:30
Akash Verma
fa30be0589
fix #17726 Databricks schema name with hyphen issue ( #18598 )
2024-11-14 20:02:20 +05:30
Keshav Mohta
968f892b48
Fix #16110 - Add support to ingest tags airflow ( #18626 )
2024-11-14 17:09:00 +05:30
harshsoni2024
cd3fcb5d22
MINOR: quicksight e2e fix ( #18629 )
2024-11-14 16:31:11 +05:30
Imri Paran
089fa785a8
build(setup-py): update pydantic version ( #18541 )
...
Update pydantic version to ">=2.7.0" in order to include `IncEx` that was introduced in 3d1355f168
2024-11-13 10:14:06 +01:00
Ayush Shah
6fa03ee66a
Fixes GEN-1994: Remove View Lineage from Metadata Ingestion flow ( #18558 )
2024-11-13 00:08:55 +05:30
Mayur Singal
f4fdafeb8a
MINOR: Athena & Tableau E2E fix ( #18596 )
2024-11-12 19:14:45 +05:30
Imri Paran
70c7880dfa
fixed bigquery system metrics e2e test ( #18601 )
2024-11-12 14:06:54 +01:00
Teddy
45d27a377d
GEN 1184 - Added Workflow Classification and Metric LevelConfig ( #18572 )
2024-11-11 15:59:42 +01:00
Imri Paran
a6d97b67a8
MINOR: fix system profile return types ( #18470 )
...
* fix(redshift-system): redshift return type
* fixed bigquery profiler
* fixed snowflake profiler
* job id action does not support matrix. using plain action summary.
* reverted gha change
2024-11-11 10:49:42 +01:00
Suman Maharana
fc79d60d83
Fixes: Added Sigma Column Level Lineage and Datamodels ( #18571 )
2024-11-11 14:42:57 +05:30
Imri Paran
cdaa5c10af
[GEN-1996] feat(data-quality): use sampling config in data diff ( #18532 )
...
* feat(data-quality): use sampling config in data diff
- get the table profiling config
- use hashing to sample deterministically the same ids from each table
- use dirty-equals to assert results of stochastic processes
* - reverted missing md5
- added missing database service type
* - use a custom substr sql function
* fixed nounce
* added failure for mssql with sampling because it requires a larger change in the data-diff library
* fixed unit tests
* updated range for sampling
2024-11-11 10:07:23 +01:00
Mayur Singal
efed932d97
Mask SQL Queries in Usage & Lineage Workflow ( #18565 )
2024-11-11 11:44:47 +05:30
Mayur Singal
b02c64931e
MINOR: Fix table not found error ( #18560 )
2024-11-09 20:33:32 +05:30
Suman Maharana
da039b197f
Add: Azure Data factory Connector ( #18543 )
...
* Added Azure Data factory Connector
* Added Lineage data factory
* removed not required files
* removed not required files
* Removed datafactory ui changes from oss
* resolve merge conflicts
* resolve merge conflicts
* added python requirements
2024-11-08 07:38:45 +01:00
Imri Paran
b92b950060
Fix 18434: feat(statistics-profiler): use statistics tables to profile trino tables ( #18433 )
...
* feat(statistics-profiler): use statistics tables to profile trino tables
- implemented the collaborative root class
- added the "useStatistics" profiler parameter
- added the "supportsStatistics" database connection property
- implemented the ProfilerWithStatistics and StoredStatisticsSource to add this functionality to specific profilers
- implemented TrinoStoredStatisticsSource for specific trino statistics logic
* added ABC to terminal classes in collaborative root
* fixed docstring for TestSuiteInterface
* reverted unintended changes
* typo
2024-11-07 18:37:31 +01:00
Imri Paran
729a06b5f0
fix: use enum.Enum instead of sqlalchemy enum ( #18464 )
2024-11-07 11:42:03 +01:00
Mayur Singal
8d40d8ea77
MINOR: Fix Materialized View Lineage ( #18539 )
2024-11-07 09:21:54 +01:00
Mayur Singal
66cf003cc3
MINOR: Fix pytest 3.11 taking 2hr ( #18533 )
2024-11-06 19:28:48 +05:30
Mayur Singal
f813ab730e
MINOR: Airflow dependency Fix ( #18530 )
2024-11-06 15:51:43 +05:30
Teddy
d579008c99
GEN 1683 - Add Column Value to be At Expected Location Test ( #18524 )
...
* feat: added column value to be in expected location test
* fix: renamed value -> values
* doc: added 1.6 documentatio entry
* style: ran python linting
* fix: move data packaging to pyproject.yaml
* fix: add init file back for data package
* fix: failing test case
2024-11-06 11:17:13 +01:00
