Ayush Shah
9c6d202555
Add Sample data, modify regex pattern ( #14467 )
2024-01-11 14:23:33 +05:30
Mayur Singal
3b61938c6d
Minor: Fix Mysql E2E Count ( #14631 )
2024-01-09 11:28:43 +05:30
Pere Miquel Brull
24643a397a
#14492 - Fix Snowflake SP parsing with empty signature ( #14623 )
2024-01-08 11:16:35 -08:00
Mayur Singal
a789fc86d6
Fix #13053 : Remove Connection URI config MongoDB ( #14584 )
...
* Fix #13053 : Remove Connection URI cofig MongoDB
* pyformat & test fixes
2024-01-05 10:51:12 -08:00
Pere Miquel Brull
f4bbca3f72
MINOR - Clean topology & add tests ( #14527 )
...
* Clean topo
* Format
* Add tests
* Fix tests
* Merge main
2023-12-29 17:00:59 +01:00
Pere Miquel Brull
3613d0b18a
MINOR - Airflow Lineage Backend to verify the fingerprint ( #14530 )
...
* hash
* MINOR - Airflow Lineage Backend to verify the fingerprint
2023-12-29 16:36:28 +01:00
Pere Miquel Brull
b84ce33b80
#11799 - Fix Airfow ownership & add pipeline tasks ( #14510 )
...
* Fix airflow owner and add tasks
* Add pipeline tasks ownership
* MINOR - Fix py CI
* Add pipeline tasks ownership
* Add pipeline tasks ownership
* MINOR - Fix py CI
* MINOR - Fix py CI
* Add pipeline tasks ownership
* patch team
* patch team
* Format
2023-12-28 10:25:00 -08:00
Teddy
61ef55290e
MINOR - generic profiler optimization for sampling and BQ ( #14507 )
...
* fix: limit sampling to specific column
* fix: handle bigquery struct columns
* fix: default partition to 1 DAY for BQ
* fix: default to __TABLES__ for BQ table metrics
* style: ran python linting
* style: fix linting
* fix: python style
* fix: set partition to DAY if not HOUR
2023-12-27 19:13:44 +01:00
Teddy
3dc642989c
Fixes #7729 - Add logic to compute passed/failed rows ( #14472 )
...
* feat: add test case resolution task workflow
* chore: add migration for test case resolution feature
* fix: removed required field for object compatibiity in older migrations
* fix: minor testCaseResolution status logic
* chore: revert migration for test case incident
* chore: update migration file
* style: renamed variables
* feat: added logic to compute failed/passed rows
* feat: add support for row level computation in schema
* chore: add test definition migration
* feat: add logic to explicitly compute row level failure
* chore: clean up code
* style: fix java
* style: fix pyton format
* fix: unhidde API for incident manager
* style: fix java styling
2023-12-27 13:38:51 +01:00
Pere Miquel Brull
1339d6518f
MINOR - Fix py CI ( #14488 )
2023-12-22 16:27:21 +01:00
Pere Miquel Brull
b786064bc2
#11857 - Store workflow status in the Ingestion Pipeline Status ( #14462 )
...
* Register StackTraceError in spec
* Register StackTraceError in spec
* Register StackTraceError in spec
* Add todos
* Update status
* docs
* format
* Fix tests
* Fix tests
* Fix tests
* Ignore generated
* Fix tests
* Fix tests
* Tests
* Try constants
* Try constants
* Print
* Print
* Print
* order
* Fix service name
* fix ui error
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-22 15:43:50 +01:00
Pere Miquel Brull
7e8e4a7e68
#14320 - FIx Airflow Callback datetime conversion ( #14487 )
...
* #14320 - don't cast twice
* #14320 - Fix Airflow Callback datetime conversion
* import
2023-12-22 15:43:41 +01:00
Onkar Ravgan
79444f4a24
Fix 13823: Validate and Parse gracefully IngestionPipeline ( #14461 )
...
* fixed ingestion pipeline parsing
* Added validation for automation workflow
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-12-22 09:56:39 +05:30
Ayush Shah
ebc0a551e5
Fixes 12947: Add Support For DQ and Profiler in Databricks Unity Catalog ( #14424 )
2023-12-20 21:18:05 +05:30
Pere Miquel Brull
e16ad3c5dc
MINOR - Clean ingestion ES code and auth providers ( #14358 )
...
* MINOR - Clean ingestion ES code and auth providers
* clean parser
* Clean security config for the client
* Clean security config for the client
* Improve class conversion exceptions
* Fix tests
* Clean up java client
* Clean up java client
* clean parser
* Fix test
* Fix test
* fix NO_AUTH error
* Fix test
* Format
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-19 12:08:48 +01:00
Pere Miquel Brull
a83a5ba3a3
MINOR - Skip delta tests for 3.11 ( #14398 )
...
* MINOR - Bump delta for 3.11
* Update flags
* MINOR - Bump delta for 3.11
* Update tests regex
* Update version
* Deprecations
* Format
* Version
* Try delta spark
* Skip delta tests for 3.11
* Update ingestion/tests/unit/topology/pipeline/test_airflow.py
2023-12-18 17:01:57 +01:00
Onkar Ravgan
ef48b7eae7
MINOR: Added custom properties methods in python sdk ( #14402 )
...
* Added custom properties methods in python sdk
* fixed tests
2023-12-18 12:57:14 +05:30
Pere Miquel Brull
d8984d267e
#14340 & #13849 - Clean secret ID and improve encrypt/decrypt exception management ( #14356 )
...
* Fix supported characters in SM
* Update SM
* Fixes
* Fixes
* Improve class conversion exceptions
* Comments
* Rename noop to db secrets manager providee
* Update sm
* Fix
* db SM
* db SM
* Fix test
* UI
* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts
* update default
2023-12-18 06:43:20 +01:00
Mayur Singal
f53fcf9168
Minor: Fix e2e count mismatch ( #14352 )
...
* Minor: Fix e2e count mismatch
* fix dbt test
* fix pyformat
* use > instead of ==
2023-12-12 22:32:13 +05:30
Onkar Ravgan
e4bfd12811
fixed e2e test after patch and collate demo ( #14321 )
2023-12-08 19:55:49 +05:30
Lucas Garcia
fe06b5cbb2
#14235 : adding dialect based on connection type to LineageParser ( #14249 )
...
* Fix #14235 : adding dialect based on connection type to LineageParser
* Fix: formating changes
* Update ingestion/src/metadata/ingestion/source/dashboard/metabase/metadata.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* style: fix indentation errors
* Fix pytest
---------
Co-authored-by: LucasGarcia07 <lucas.junqueira@hurb.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-12-08 19:49:59 +05:30
NiharDoshi99
8d925c46a5
#13696 : add support for dot in schema name to fetch tables ( #14246 )
2023-12-08 12:04:28 +05:30
Teddy
3bbf55fcda
FIXES #14049 - Split test case resolution status from test case result ( #14204 )
...
* refactor: entityFQN as ListFilter condition
* feat: implement resolution entity timeseries
* fix: rename to testCaseResolutionStatus
* ref: extracted ES query builder into private method
* ref: extract OS query builder in its own method
* ref: remove ingestion logic for test case resolution
* fix: reorganize json schemas to fix circular import in Python
* ref: object names in typescript code
* feat: added indexing of test case resolution
* feat: added test case resolution sample data
* fix: test case resolution api logic
* fix: audit logger for entityTimeSeriesInterface
* fix: DDL generation
* style: python linting
* fix: skip UI test case resolution tests
* fix: remove extension field
* fix: renamed testCaseFailureStatus to testCaseResolutionStatus
* fix: remove reviewer
* fix: rename sequenceId to stateId
* fix: re adjust search weights
* fix: removed InReview status
* style: ran python linting
2023-12-04 23:18:01 -08:00
Teddy
f571f67eda
fix: DI skip on failure when listing entities ( #14226 )
2023-12-04 16:59:09 +00:00
Mayur Singal
389ae79d3c
#14115 : Separate Unity Catalog From Databricks ( #14138 )
2023-12-04 11:22:46 +05:30
Pere Miquel Brull
7fcdf08ca4
#11626 & #14131 - Lineage with other Entities & attr-based xlets ( #14191 )
...
* Add OMEntity model
* Test OMEntity
* Update repr
* Fix __str__
* Add entity ref map
* Test serializer for backend
* Fix tests
* Fix serializer
* Test runner
* Add runner tests
* Update docs
* Format
2023-12-01 06:29:44 +01:00
Onkar Ravgan
ebb2317cf0
Fix 14040: Part 1 Remove get_by_name calls from topology ( #14098 )
...
* Changed for database
* Added changes for dashboard_service
* Changed for messaging service
* Changed for mlmodel service
* Changed for pipeline service
* Changed for search service
* Changed for objectstore service
* fixed wrong import
* fixed lint
* fixes
* fixed pytests
* fixed domo db pytest
* Fixed review comments
2023-11-27 16:15:47 +05:30
Onkar Ravgan
335aa86fcc
fixed athena e2e ( #14041 )
2023-11-20 21:20:32 +05:30
Onkar Ravgan
b781c66206
Fixed tableau e2e counts due to new sample data in the instance ( #14039 )
...
* fixed tableau e2e new
* rmv hive test
2023-11-20 19:54:06 +05:30
Mayur Singal
9f14ef7fab
Fix #13954 : Fix ParseException for older version of databricks ( #14015 )
2023-11-20 13:14:40 +05:30
Teddy
c7ac28f2c2
Fixes #11357 - Implement profiler custom metric processing ( #14021 )
...
* feat: add backend support for custom metrics
* feat: fix python test
* feat: support custom metrics computation
* feat: updated tests for custom metrics
* feat: added dl support for min max of datetime
* feat: added is safe query check for query sampler
* feat: added support for custom metric computation in dl
* feat: added explicit addProper for pydantic model import fo Extra
* feat: added custom metric to returned obj
* feat: wrapped trino import in __init__
* feat: fix python linting
* feat: fix typing in 3.8
2023-11-17 17:51:39 +01:00
Teddy
f3da919329
Feat: Backend Support for Custom Metrics ( #13965 )
...
* feat: add backend support for custom metrics
* feat: fix python test
2023-11-17 19:16:35 +05:30
Suresh Srinivas
5f34bd02d3
Fixes #13595 Consolidate changes by a user in a single session to a single change ( #13617 )
...
* getChangeDescription to use entity and update type to decide previous version
* Fixes #13595 - Consolidate changes by a user in a single session to a single change
2023-11-16 18:14:27 -08:00
Mohit Yadav
10d8ec84fe
Logs added for Search Indexing and Stats issue fixed ( #13956 )
...
* Logs added for Search Indexing and Stats issue fixed
* Fix uninstall error
* Add error handling
* fix lint
* Push Job Level Exception on top
* disable flaky tests
* Fix Logs not visible in Search
---------
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-11-13 23:39:56 +05:30
Pere Miquel Brull
c742835766
Auto Tagger Application - Preparing the Ingestion Framework ( #13862 )
...
* Prepare the skeleton for generic app registration
* Prepare the skeleton for generic app registration
* Handle app runner
* Prepare the skeleton for generic app registration
* Prepare the skeleton for generic app registration
* Allow deployment
* Fix PII APP
* Fix lint
* Fix PII APP
* Fix PII APP
* Prepare config-based external apps
* Prepare config-based external apps
* Fix lint
* Prepare config-based external apps
* Fix DI errors
* Amend comments
2023-11-13 08:58:38 +01:00
Mayur Singal
367bac9064
Fix #13787 : Add support for ES data types ( #13916 )
...
* Fix #13787 : Add support for ES data types
* fixed tests
---------
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-11-10 20:14:42 +05:30
Pere Miquel Brull
0eacc829a4
Fix #13794 - Add domain support to the Python SDK ( #13931 )
2023-11-10 11:00:06 +01:00
Pere Miquel Brull
7c06116b53
Add deprecation warnings ( #13927 )
2023-11-10 15:17:07 +05:30
Pere Miquel Brull
8891a9a410
Fix #13906 - Fix add_mlmodel_lineage description field ( #13920 )
2023-11-10 15:16:09 +05:30
Pere Miquel Brull
b250cd8808
Fix #13699 - Add separator for Storage Container manifest ( #13924 )
...
* Fix #13699 - Add separator for Storage Container manifest
* Fix #13906 - Fix add_mlmodel_lineage description field
* Add separator
* Add separator
2023-11-10 10:44:47 +01:00
Mayur Singal
4b625f7ba5
Move pandas top level import ( #13926 )
2023-11-10 14:15:14 +05:30
Mayur Singal
a8145a82fa
Fix #13603 : Configurable Sample Data Rows for Profiler ( #13807 )
...
* Fix #13603 : Configurable Sample Data Rows
* Fix #13603 : Configurable Sample Data Rows for Profiler
* fix table config
* support configurable overwriting of sample data
* add support for schema and database profiler configuration
* chore(ui): put sampleDataStorageConfig under advanced config
* fix tests
* py format
* chore(ui): add sampleDataCount in table profiler config
* fix tests
* pylint & tests
* feat(ui): add profiler settings tab in database and database schema page
* chore(ui): show different inputs for profile sample type
* schema changes to make default storange config null
* add unit test
* schema changes to fix api
* update profiler setting schema
* move profiler settings to manage button
* sync locals
* fix(ui): unit tests
* fix tests
* py format
* fix lint
* minor improvements
* chore(ui): update profiler settings schema
* resolve review comments
* pytest
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-11-09 18:49:42 +05:30
Suresh Srinivas
a89e317a2b
Fixes #13863 - Show inherited relationships of an entity ( #13864 )
...
* Fixes #13863 - Show inherited relationships of an entity
* Test failure fixes
* Commenting out invalid python test
2023-11-07 09:11:06 -08:00
Onkar Ravgan
c7834e74cc
fixed avro recursive record ( #13856 )
2023-11-06 16:27:06 +05:30
Teddy
d025e217d6
fix: catch not Either type in workflow and return explicit error message ( #13796 )
2023-11-02 13:02:26 +01:00
Ayush Shah
0a04ce85bb
Add Multilingual Support in EntityLink ( #13826 )
2023-11-02 14:35:22 +05:30
Teddy
10904049e4
fix: handle lower and upper case name ( #13778 )
2023-10-31 09:51:13 +01:00
Teddy
452a33b1a0
Fixes Druid Profiler failures ( #13700 )
...
* fix: updated playwrigth test structure
* fix: druid profiler queries
* fix: python linting
* fix: python linting
* fix: do not compute random sample if profile sample is 100
* fix: updated workflow to test on push
* fix: move connector config to category folder
* fix: updated imports
* fix: added pytest-dependency package
* fix: updated readme.md
* fix: python linting
* fix: updated profile doc for Druid sampling
* fix: empty commit for CI
* fix: added workflow constrain back
* fix: sonar code smell
* fix: added secrets to container
* Update openmetadata-docs/content/v1.2.x-SNAPSHOT/connectors/ingestion/workflows/profiler/index.md
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* Update openmetadata-docs/content/v1.2.x-SNAPSHOT/connectors/ingestion/workflows/profiler/index.md
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* Update ingestion/tests/e2e/entity/database/test_redshift.py
* fix: ran pylint
* fix: updated redshift env var.
* fix: import linting
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-10-25 20:47:51 +02:00
Ayush Shah
bfb361dc85
Fix Bigquery lineage Pytests ( #13695 )
2023-10-25 11:15:41 +05:30
Ayush Shah
57cb72c26f
Fix Checkstyle ( #13683 )
2023-10-23 15:51:40 +05:30
