Ayush Shah
7c7a7c35a9
Check Test Connection Oracle DBA Access ( #15764 )
2024-04-01 18:18:21 +05:30
Onkar Ravgan
2e640f6ef2
fixed dbt exception condition ( #15762 )
2024-04-01 15:21:06 +05:30
Onkar Ravgan
2931798be9
fixed superset class init ( #15761 )
2024-04-01 15:10:08 +05:30
Imri Paran
5b963a5d9e
MINOR: remove metadata/__init__.py ( #15750 )
2024-03-29 15:15:52 +05:30
harshsoni2024
884029f031
Fix #14089 : Add QlikCloud connector support ( #15610 )
2024-03-28 20:15:22 +05:30
Pere Miquel Brull
890820ed92
MINOR - App routes & datamodel ( #15722 )
...
* MINOR - App routes & datamodel
* fix future annotations
* fix future annotations
2024-03-27 19:12:24 +01:00
Mayur Singal
bf0ec44f4b
Fix #15678 : Accomodate Metabase API changes ( #15692 )
2024-03-27 15:19:20 +05:30
Mayur Singal
6acb47a87a
MINOR: Fix unity catalog test connection ( #15712 )
2024-03-27 15:14:26 +05:30
IceS2
ce3f124a33
MINOR: Add Multithread Documentation ( #15706 )
...
* Add Multithread Documentation
* Add general considerations
2024-03-27 08:30:16 +01:00
Mayur Singal
0532bb1226
MINOR: Fix external table lineage processing ( #15713 )
2024-03-27 12:37:20 +05:30
Mayur Singal
8073a80989
Fix #14285 : Add column lineage support for tableau datamodels ( #15646 )
2024-03-27 11:03:40 +05:30
Pere Miquel Brull
9d7bfa363e
MINOR - Clean metadata CLI ( #15631 )
...
* Docs
* MINOR - Clean metadata CLI
* remove tests
2024-03-26 16:36:47 +01:00
Ayush Shah
6039fe9462
Fix TypeError Missing arg ( #15698 )
2024-03-26 18:32:32 +05:30
Mayur Singal
6b90c245d4
MINOR: Add support for json schema parsing for datalake & s3 ( #15615 )
2024-03-26 10:03:21 +05:30
IceS2
e7c9d6aa7f
FIXES 15215: Implement initial Multithreading approach for the Metadata Ingestion on Databases ( #15130 )
...
* Implement Initial MultiThread suggestion
* Update all the ingestion sources to use the new ContextManager
* Fix missing wraps on decorator
* Fix Unittests
* Fix linters
* Fix linters
* Fix BigQuery UnitTests
* Add UnitTests to the newly created code
* Fix unittest
* change the threads from table to schemas
* Update README.md
* Small change suggested by Sonar
* Slight change to test a different way to multithread over tables
* Debug changes
* More multithread tests
* Remove uneeded wait time
* Testing
* refactor code based on removal of time.sleep
* Fix wrong paste
* Improve ExecutionTimeContextManager
* Fix missing .get() and unit tests
* Fix conflicting changes
* Update Multithread logic with the incremental extraction
* Fix linters
* Fix unittest
* Remove commented code
* Fix Unittests
* Fix checkstyle
* Change default to threads = 1
2024-03-25 18:20:40 +01:00
Ayush Shah
00677a1e1b
Fix External Account Json Schema Issue ( #15671 )
2024-03-23 16:47:55 +05:30
Pere Miquel Brull
a79e79ef3d
#15662 - List All test cases from a table in DQ ( #15665 )
...
* #15662 - List All test cases from a table in DQ
* #15662 - List All test cases from a table in DQ
* #15662 - List All test cases from a table in DQ
2024-03-22 11:30:02 +01:00
Ayush Shah
8b880bbf91
Fixes 14370: Add Azure Client, support Default Creds ( #15554 )
...
* Add Azure Client, support Default Creds
2024-03-22 14:28:42 +05:30
Mayur Singal
ad28af4f4f
MINOR: Fix sample data upload - binary data error ( #15659 )
2024-03-22 12:13:26 +05:30
Mayur Singal
2208662886
MINOR: Move external table lineage to post processing ( #15633 )
2024-03-22 11:46:14 +05:30
Pere Miquel Brull
b778bc7968
#14943 - Check tags before PII processor ( #15622 )
2024-03-21 14:15:28 +05:30
Imri Paran
7eeb0e45d2
1. add profiler support for GEOMETRY type in redshift. ( #15628 )
...
2. Add GEOMETRY to values not to compute.
2024-03-20 13:42:46 +01:00
Ayush Shah
1bb7d893ac
Fix 15419: Improve fetching Oracle Queries for SP ( #15621 )
2024-03-20 15:58:06 +05:30
Ayush Shah
e06e5c1bdd
Fixes 15544: Histogram not working for more than 15 units ( #15617 )
2024-03-20 11:35:52 +05:30
Onkar Ravgan
2dd912ab8a
fixed dagster tasks status ( #15605 )
2024-03-19 18:03:58 +05:30
Mayur Singal
ef61bdc3a8
MINOR: Looker fix bitbucket protocol ( #15604 )
2024-03-19 10:06:24 +01:00
Matias Puerta
7036a7bb25
Fix typo in Bitbucket URL ( #15602 )
2024-03-18 16:24:11 -07:00
Mayur Singal
b5fb57f7c6
Fix #15118 : Handle exception while processing usage comparisons ( #15597 )
2024-03-18 17:26:32 +05:30
Mayur Singal
4696e14f6c
MINOR: Add support for databricks external table lineage ( #15585 )
2024-03-18 16:23:53 +05:30
Pere Miquel Brull
0eb18ad891
MINOR - Clean MSSQL lineage & usage ( #15571 )
2024-03-18 10:55:29 +01:00
Mayur Singal
104b41cfd2
MINOR: Fix Looker clone repo failure for bitbucket ( #15590 )
...
* MINOR: Fix Looker clone repo failure for bitbucket
* pyformat
2024-03-16 15:44:14 +01:00
Trs
4db9b775ea
#14169 : Support external_account type for GCP Auth ( #14166 )
2024-03-16 19:59:02 +05:30
IceS2
51e3d7a466
FIXES 15215: First draft implementation on extracting metadata incrementally. Done for Snowflake, BigQuery and Redshift ( #15201 )
...
* Initial incremental implementation for snowflake
* Initial unit test refactor for snowflake
* Fix linter complaints
* Propagate change on abstract create method
* Add missing argument to create
* Polish Snowflake incremental extraction
* Fix linters and make enabled required
* Initial proposal for incremental bigquery extraction
* BigQuery incremental tests
* Remove debugging override
* Fix linters
* Remove unused query
* Initial Redshift Incremental Extraction
* Add Incremental Extraction documentation
* Move the default to False
* Improve code based on sonarcloud input
* Apply suggestions
* Fix wrong path
* Change timestamp to be time aware as per sonar
* Move documentation to 1.4
* Move documentation to 1.4
* Fix linters
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-03-15 14:00:49 +01:00
Onkar Ravgan
46954dc848
Fix #15563 : Fixed incorrect col ordering after patch request from ingestion ( #15577 )
...
* fixed patch col order
* Added excp handling
* changed logs to warning
* rmv excp
2024-03-15 13:08:33 +05:30
Mayur Singal
88ab7475e7
MINOR: Restructure dbServiceName field in dashboard and pipeline ( #15548 )
2024-03-15 12:42:47 +05:30
Mayur Singal
ed41f25f18
MINOR: Fix multiline insert query stored procedure lineage ( #15578 )
2024-03-15 12:40:59 +05:30
Mayur Singal
b643206bba
Fix #11905 : Automated lineage between external table and container snowflake ( #15537 )
2024-03-15 00:52:41 +05:30
Pere Miquel Brull
fd403bae9a
MINOR - Review query performance ( #15553 )
...
* MINOR - Review query performance
* MINOR - Review query performance
* MINOR - Review query performance
* MINOR - Review query performance
2024-03-14 06:37:38 +01:00
Mayur Singal
658526e02c
MINOR: Skip source hash generation for service ( #15516 )
2024-03-13 22:13:19 +05:30
Sriharsha Chintalapani
d0efaac877
Fix #11868 : Duplicated queries cannot be created ( #15519 )
...
* Fix #11868 : Duplicate query should throw an error of entityExists
* Fix #11868 : Duplicate query should throw an error of entityExists
* fix test
* fix test
* Fix uniquee constraint for checksum in Postgres
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-03-13 13:02:26 +01:00
Teddy
93a4453a6f
[MINOR] GX logging hierarchy ( #15542 )
...
* fix: GX module logging hierarchy
* style: ran python linting
2024-03-13 07:37:57 +00:00
Mayur Singal
80123b3c0a
Fix #15533 : Fix name & display name for kafka json schema parser ( #15534 )
2024-03-12 23:02:41 +05:30
Sriharsha Chintalapani
8af194193a
Add stack trace while throwing an error to debug ( #15522 )
2024-03-12 10:07:46 -07:00
Kent Chenery
34b727f6c9
Fixes ISSUE-13473: Ensure MSSQL columns query filters to tables and views ( #15530 )
...
* Update queries.py and utils.py to ingest table and view descriptions
* Ensure MSSQL columns query filters to tables and views
* Fix linters
---------
Co-authored-by: Pablo Takara <pjt1991@gmail.com>
2024-03-12 13:56:30 +01:00
Mayur Singal
8a2ee00fe3
Fix #15432 : make sample data external storage path configurable ( #15478 )
2024-03-12 15:02:28 +05:30
mgorsk1
98850ab5cc
feat: OpenLineage integration ( #15317 )
...
* 🎉 Init OpenLineage connector
Co-authored-by: dechoma <dominik.choma@gmail.com>
* MLH - make linter happy
* review fixes
* 🐛 Fix path for ol event in tests
* 🐛 Fix path for ol event in tests
* Update ingestion/setup.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/models.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* review fixes 2
* linter
* review
* review
* make linter happy
* fix test_yield_pipeline_lineage_details test
* make linter happy
* fix tests
* fix tests 2
---------
Co-authored-by: dechoma <dominik.choma@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-03-12 08:39:25 +01:00
Teddy
36327c2ee9
FIX #15436 - Add disconnect method for databricks client ( #15514 )
...
* fix: implement is_disconnect for databricks client
* style: ran python linting
2024-03-12 07:42:04 +01:00
Imri Paran
aade838020
Fixes #15388 : Use native backup tools ( #15393 )
...
* feat: use native backup tools
1. added mysqldump 8.3 to the ingestion container.
2. documented how to use native tools to back up and restore.
3. added deprecated message on the cli backup and restore.
* added deprecation notice for 1.3 backup
* removed 1.3.x deprecation notice
* added another backup page in 1.3 introducing SQL dump tools
* added --set-gtid-purged=OFF to the mysql dump process
2024-03-12 06:23:05 +01:00
IceS2
7805a0b609
MINOR: Fix athena e2e tests ( #15486 )
...
* Comment side effects
* Update assert to match clauses better
* Improve input
* Improve input
* Update assert to match clauses better
* Fix Athena E2E Values
* Uncomment needed steps
* Fix linters
2024-03-08 09:31:06 +01:00
Imri Paran
f4932ee420
added log levels to all example ingestion configs config ( #15488 )
...
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-03-07 16:54:10 +00:00
