733 Commits

Author SHA1 Message Date
Teddy
84469e643c
FIX #13475 - Add Parameter Validation for Min/Max tests (#15849)
* feat: add validation field to test case definition parameters

* feat: added parameter validation logic

* fix: set ES testcontainer startup attemp to 3

* style: ran java linting

* fix: added waitForEsAsyncOp in post_trigger_app_200 test
2024-04-15 07:52:43 -07:00
Karan Hotchandani
bd3838868c
disable install for preview apps (#15884)
* disable install for preview apps

* move screenshot path to base class

* MINOR - Add Marketplace preview

* format

* add unit tests

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-04-15 17:32:01 +05:30
sonika-shah
c010939072
Added payload size check logic for Elasticsearch (#15540)
* Added payload size check logic for Elasticsearch

* Added payload size check logic for Elasticsearch - made review changes

* Added payload size check logic for Elasticsearch - added env in docker files

* Optimize payload logic, fix search application error stats issue
2024-04-15 16:25:06 +05:30
Ayush Shah
0c3e580592
MINOR: Fix Entity Link Error (#15864) 2024-04-11 18:41:59 +05:30
Imri Paran
b2ce491ff1
MINOR: Add failed rows sample to test case (#15682)
* add failed sample data

* format

* fixed masking pii data in test failed rows sample

* format

* failedRowsSamples -> failedRowsSample

* failedRowsSamples -> failedRowsSample

* fixed tests

* format

* wip

* added computePassedFailedRowCount to python client

* comment for loggerLevel

* format

* fixed tests

* tests for putting / deleting failed samples

* format

* format

* added test case for pii test

* changed method name to deleteTestCaseFailedRowsSample

* added getComputePassedFailedRowCount
2024-04-10 17:00:00 +02:00
Pere Miquel Brull
67dee85abd
MINOR - Add Application type for logs (#15802)
* MINOR - Add Application type for logs

* MINOR - Add Application type for logs

* Add application_task to DataInsightLatestRun interface and update LogsViewer component

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2024-04-05 07:09:59 +02:00
Teddy
205850be79
[MINOR] fix antlr parser definition for entity link (#15758)
* fix: update antlr regex for entity fqn

* fix: update antlr rule to allow single character

* style: ran python linting

* fix: updated antlr token for NAME_OR_FQN
2024-04-03 08:34:43 +00:00
Teddy
4b13403931
Fix #15594 - Implement domain support for test cases (#15736)
* feat: add list from search service logic

* fix: added test and enable ES testing

* style: ran java linting

* fix: ES container startup timeout

* style: ran java linting

* fix: removed array definition as attributes of filter class

* style: ran java linting

* fix: update list from search

* feat: added q param to list from search endpoint

* style: ran java linting

* fix: enable ES test for list from search

* fix: added include/exclude from source

* style: ran java linting

* feat: add domain support for test case

* fix: added search service check for domain inheritence

* feat: added domain query param to list from search

* style: ran java linting
2024-04-02 22:45:48 +02:00
harshsoni2024
feb33a0cc2
Fix #12964: Qlik Sense & Qlik Cloud filter draft dashboards (#15726)
* Fix #12964: filter draft dashboards from config

* Fix #12964: add unit test for qlik_sense

* Fix #12964: added UI and doc code

* Fix #12964: move includedraftdashboard flag from source_connection to source_config

* Fix #12964: filter draft dashboards in qlikcloud

* Fix #12964: add unit test for qlik cloud

* Fix #12964: remove unnecessary comments, code clean

* Fix #12964: pylint changes
2024-04-02 14:30:33 +02:00
Imri Paran
740480fe28
MINOR: change systemApp to system (#15757) 2024-04-01 18:50:43 +05:30
Suman Maharana
bb2684dd62
Fix #11351 Added Kafka Connect connector JSON schemas (#15699) 2024-04-01 18:45:39 +05:30
Mohit Yadav
c061522bbc
Fixes #15483 : Change Generic to Webhook (#15744)
* Fixes #15483 : Change Generic to Webhook

* Typo
2024-04-01 12:36:09 +05:30
Pere Miquel Brull
fc2c909f6c
MINOR - MetaPilot updated conf (#15745) 2024-03-28 17:18:41 +01:00
Teddy
dafbcbd726
fix: update ingestion profiler config (#15753) 2024-03-28 16:06:08 +00:00
harshsoni2024
884029f031
Fix #14089: Add QlikCloud connector support (#15610) 2024-03-28 20:15:22 +05:30
Pere Miquel Brull
890820ed92
MINOR - App routes & datamodel (#15722)
* MINOR - App routes & datamodel

* fix future annotations

* fix future annotations
2024-03-27 19:12:24 +01:00
Teddy
9b7f402a16
FIX #14941 - Implement test cases filters (#15616)
* feat: add list from search service logic

* fix: added test and enable ES testing

* style: ran java linting

* fix: ES container startup timeout

* style: ran java linting

* fix: removed array definition as attributes of filter class

* style: ran java linting

* fix: update list from search

* feat: added q param to list from search endpoint

* style: ran java linting

* fix: enable ES test for list from search

* fix: added include/exclude from source

* style: ran java linting

* fix: test case

* fix: exception type
2024-03-27 07:25:11 +01:00
Sriharsha Chintalapani
7a3a271f26
Add entityReferences and entityReferenceList as custom properties (#15448)
* Add entityReferences and entityReferenceList as custom properties

* ui: allow to add custom properties for all available property types.

* make entity type as an array

* Fix label typo and import order in AddCustomProperty component

* Remove unnecessary property types with format

* Refactor CustomPropertyTable.tsx to handle array config and enum config

* Add entity reference options and update custom property form

* Add EntityReference and EntityReferenceList as a custom property

* add validation for date format

* Add date and dateTime input support to PropertyValue component

* Fix import order and add search functionality to type dropdown

* Fix custom property cypress tests

* add input for number, email, timestamp, timeInterval, duration, time, sqlQuery,

* add input support for entityReference and entityReferenceList

* Add placeholders for email, timestamp, start time, end time, and duration inputs

* Refactor PropertyValue component to use destructuring for timeInterval object

* Add minWidth style to PropertyValue component and include SQL query editor

* Add entityReference and entityReferenceList as a generic types and not reference to the existing types

* Remove services from entity reference types list

* handle property values for different property type

* Update ExtensionTable to handle object values in CustomPropertyTable

* Add entity reference list rendering and styling

* Fix file paths in complexTypes.json

* Add regex constant for UNIX timestamp in milliseconds and update language files

* Refactor custom property configuration and add new options

* Fix import order and update property value handling

* add unit tests for different input types for different property type

* add unit test for different property type values

* fix cypress tests

* add cypress test for CRUD for different types

* add cypress test for entity ref and entity ref list property type

* add cypress tests for all the 11 entities which support custom properties

* fix cypress test for createCustomPropertyForEntity

* Add new interfaces and types for CustomPropertyTable

* add cypress test for custom property to create via APIs for entities

* add cypress tests for string,integer, markdown, number, duration, email

* add cypress test for enum property for all entities

* add cypress test for sqlQuery, timestamp and timeInterval property

* fix cypress test for sqlQuery input

* fix flaky cypress test for sqlQuery input

* address comments

* Refactor import statements and update ENTITY_PATH enum

* Update import statement in CustomProperty.ts

* change TimeInterval to Time Interval

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2024-03-26 11:02:44 +05:30
Onkar Ravgan
891e0bf893
Updated alation json schemas (#15674) 2024-03-26 10:50:43 +05:30
IceS2
e7c9d6aa7f
FIXES 15215: Implement initial Multithreading approach for the Metadata Ingestion on Databases (#15130)
* Implement Initial MultiThread suggestion

* Update all the ingestion sources to use the new ContextManager

* Fix missing wraps on decorator

* Fix Unittests

* Fix linters

* Fix linters

* Fix BigQuery UnitTests

* Add UnitTests to the newly created code

* Fix unittest

* change the threads from table to schemas

* Update README.md

* Small change suggested by Sonar

* Slight change to test a different way to multithread over tables

* Debug changes

* More multithread tests

* Remove uneeded wait time

* Testing

* refactor code based on removal of time.sleep

* Fix wrong paste

* Improve ExecutionTimeContextManager

* Fix missing .get() and unit tests

* Fix conflicting changes

* Update Multithread logic with the incremental extraction

* Fix linters

* Fix unittest

* Remove commented code

* Fix Unittests

* Fix checkstyle

* Change default to threads = 1
2024-03-25 18:20:40 +01:00
Pere Miquel Brull
4db6dce2f9
MINOR - Add /status endpoint (#15394)
* MINOR - Add `/validate` endpoint

* rename endpoint

* fix test

* add new migration

* Status

* Simplify migration client

* Simplify migration client

* Simplify migration client
2024-03-25 07:05:13 +01:00
Pere Miquel Brull
9329b5a488
#12678 - Add SAP Hana Backend Converter (#15629)
* #12678 - Add SAP Hana Backend Converter

* Format

---------

Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2024-03-23 19:36:56 +01:00
Ayush Shah
00677a1e1b
Fix External Account Json Schema Issue (#15671) 2024-03-23 16:47:55 +05:30
Ayush Shah
8b880bbf91
Fixes 14370: Add Azure Client, support Default Creds (#15554)
* Add Azure Client, support Default Creds
2024-03-22 14:28:42 +05:30
Mohit Yadav
f03ae2d6ab
[Fix] Application Fixes, Updated AppRunType and Removed Server App Inits (#15649)
* - Fix App Run Type
- Remove Init on App Start
- Migrate stats setup for SearchIndexing to execution

* fix the AppLogsViewer unit test failing due to appRunType type changes

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2024-03-22 08:48:23 +05:30
Pere Miquel Brull
3fb4dbb55a
MINOR - Application Preview (#15609) 2024-03-21 07:56:14 +01:00
Sriharsha Chintalapani
8377128c93
Application cannot be installed with None as schedule type (#15524)
* Application cannot be installed with None as schedule type

* MINOR: Add / Fix GCS and ADLS - docs, bugs (#15502)

Add GCS and ADLS docs

* Add stack trace while throwing an error to debug (#15522)

* Fix #15533: Fix name & display name for kafka json schema parser (#15534)

* MINOR: postgres add ssl options in yaml (#15538)

* Small change to fix the Glossary TErm Tasks not fetching Owner automatically (#15535)

* remove the DataInsightAlert spec as the alerts test are covered in ObservabilityAlerts spec (#15531)

* MINOR: fix help dropdown item font and icon sizes (#15511)

* fix help dropdown item font and icon sizes

* added unit test for navbarUtils

* fix sonar failure

* changes as per comments

* minor fixes

* Minor: fixed DQ edit test case issue and searchIndexDetails typescript issue (#15528)

* Minor: fixed DQ edit test case issue and searchIndexDetails typescript issue

* fixed failing unit test

* fixed unit test for Data quality test

* reverting e2eLabeler changes

* chore(ui): separate routes as per categories (#15512)

* chore(ui): seprate routes as per categories

* add tests

* fix test

* domain path fix

* [MINOR] GX logging hierarchy (#15542)

* fix: GX module logging hierarchy

* style: ran python linting

* - Use Entity Type (#15546)

* removed docs 1.4.x (#15550)

* Suggestions Alert new design (#15532)

* update ux of suggestions alert

* locales

* minor changes

* fix descriptions

* minor css fixes

* Fix #11868: Duplicated queries cannot be created (#15519)

* Fix #11868: Duplicate query should throw an error of entityExists

* Fix #11868: Duplicate query should throw an error of entityExists

* fix test

* fix test

* Fix uniquee constraint for checksum in Postgres

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

* added OpenMetadataApplication.getDao (#15549)

* Add the entityUrl to the header for ThreadMessages (#15552)

* feat: AppResource (#15555)

forbid modification of system app

* MINOR: Skip source hash generation for service (#15516)

* sync the documentation roadmap page with the getcollate roadmap (#15551)

* Application cannot be installed with None as schedule type

* None type should not register in scheduler

* scheduleTimeline fix

* fix payload of schedule type none and change it to scheduleTimeline

* fix unit test

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: IceS2 <pjt1991@gmail.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Imri Paran <imri.paran@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2024-03-19 18:23:25 +05:30
Mohit Yadav
91184ecf5d
Using Backend for Oidc Flow (#15557)
* - Initial Draft of Using Backend for Oidc Flow

* - Review Comments

* - Add Session Handler

* - Add Nonce Validation

* Typo

* support backend auth flow from client side

* Add Refresh Logic

* Remove oidc Config from /auth

* Auth Logout Handling

* Add Refresh as separate endpoint

* return empty on no refresh token

* support refresh token and logout

* fix route issues

* fix path issues

* Redirect to Logout on know session

* fix store issues

* exclude logout api from domain filter

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-03-19 17:15:30 +05:30
Pere Miquel Brull
060a9d7f68
MINOR - Prepare AI chat feed (#15510)
* MINOR - Prepare AI chat feed

* Rename ai to chatbot

* Rename ai to chatbot
2024-03-18 10:29:52 +01:00
Trs
4db9b775ea
#14169: Support external_account type for GCP Auth (#14166) 2024-03-16 19:59:02 +05:30
IceS2
51e3d7a466
FIXES 15215: First draft implementation on extracting metadata incrementally. Done for Snowflake, BigQuery and Redshift (#15201)
* Initial incremental implementation for snowflake

* Initial unit test refactor for snowflake

* Fix linter complaints

* Propagate change on abstract create method

* Add missing argument to create

* Polish Snowflake incremental extraction

* Fix linters and make enabled required

* Initial proposal for incremental bigquery extraction

* BigQuery incremental tests

* Remove debugging override

* Fix linters

* Remove unused query

* Initial Redshift Incremental Extraction

* Add Incremental Extraction documentation

* Move the default to False

* Improve code based on sonarcloud input

* Apply suggestions

* Fix wrong path

* Change timestamp to be time aware as per sonar

* Move documentation to 1.4

* Move documentation to 1.4

* Fix linters

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-03-15 14:00:49 +01:00
Mayur Singal
88ab7475e7
MINOR: Restructure dbServiceName field in dashboard and pipeline (#15548) 2024-03-15 12:42:47 +05:30
Mayur Singal
b643206bba
Fix #11905: Automated lineage between external table and container snowflake (#15537) 2024-03-15 00:52:41 +05:30
Mayur Singal
8a2ee00fe3
Fix #15432: make sample data external storage path configurable (#15478) 2024-03-12 15:02:28 +05:30
mgorsk1
98850ab5cc
feat: OpenLineage integration (#15317)
* 🎉 Init OpenLineage connector

Co-authored-by: dechoma <dominik.choma@gmail.com>

* MLH - make linter happy

* review fixes

* 🐛 Fix path for ol event in tests

* 🐛 Fix path for ol event in tests

* Update ingestion/setup.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/models.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* review fixes 2

* linter

* review

* review

* make linter happy

* fix test_yield_pipeline_lineage_details test

* make linter happy

* fix tests

* fix tests 2

---------

Co-authored-by: dechoma <dominik.choma@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-03-12 08:39:25 +01:00
Imri Paran
4bb5550a46
MINOR: add the systemApp property (#15473)
* feat: add "systemApp" property

* use BAD_REQUEST for System Apps uninstall

* use IllegalArgumentException
2024-03-11 15:39:20 +01:00
Mohit Yadav
9491e04088
- Fix Poll Interval (#15453)
* - Fix Poll Interval

* make poll interval as 1

* - Add random tests
- Disable Tests for Subscription for Query

* - Checkstyle Fix

* - Remove ondemand jobs as well
- On update remove already init instance

* - use alias

* - typo

* - Search Index Fix
2024-03-06 02:21:28 +05:30
Aniket Katkar
110e7a4f4b
Fix #15438: Update filters and destination for 'Conversation' source in notification alert (#15443)
* Add 'Mentions' enum for subscriptionCategory

* fix the user list selection field not showing for 'Mentioned Users' filter

* update the cypress for new changes

* fix teams not showing in the mention suggestions
2024-03-04 21:40:31 +05:30
Sriharsha Chintalapani
cecbf80a2d
Add Custom Propety Config to store format, enum values, entity types (#15302)
* Add Custom Propety Config to store format, enum values, entity types

* Fix import statements and remove unused code

* Add Custom Propety Config to store format, enum values, entity types

* Add support for enum field type in custom properties

* update name in customPropertyConfigTypeValueField

* add custom property config column in custom property table

* Update padding-left in block-editor.less

* Add enum value translation for multiple languages

* update placeholder of config

* fixed python sdk

* add enum type in property value

* add unit tests

* Add Custom Propety Config to store format, enum values, entity types

* update ui to handle the enum config and validation

* Fix enum value handling in EditCustomPropertyModal and PropertyValue

* Update CustomProperty.md with enum values and multi-select option

* add cypress test

* add cypress for multiselect enum value

* Add tests for enum props

* add cypress for editing the enum property

* Add validations to enum

* Fix dependency issue

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2024-02-29 14:36:24 +05:30
Onkar Ravgan
1fc2c7f974
MINOR: Part 1 of #15090: dbt JSON Schema & Parsing Improvements (#15297) 2024-02-29 10:41:21 +05:30
Teddy
056e6368d0
Issue #14765 - Preparatory Work (#15312)
* refactor!: change partition metadata structure for table entities

* refactor!: updated json schema for TypeScript code gen

* chore: migration of partition for table entities

* style: python & java linting

* updated ui side change for table partitioned key

* miner fix

* addressing comments

* fixed ci error

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-02-28 07:11:00 +01:00
Aniket Katkar
0a38334c98
Minor: Add edit alert form item improvements (#15346)
* add "Assignees" option in subscription category enum

* add Assignee option for task alert source destination options

* change alert form section names from "Trigger" to "Source" and "Actions" to "Trigger"

* localization changes for other languages

* rename the trigger item components

* rename observability alert form actions item component to a proper name

* only show "Owners" and "Assignees" options for "Task" source

* change all alert trigger references to source

* change all alert action item references to trigger

* fix failing unit test

* add cypress coverage for task source

* convert alert specs to .ts files

* update the changed key value for other languages
2024-02-28 11:13:13 +05:30
Ashish Gupta
1efbeb1204
MINOR: made ui enhancement in setting page (#15269)
* made ui enhancement in setting page

* ui fixes

* fix ui render in application and other setting page

* fix unit test and user profile page issues

* added loader in team selectable in user profile page

* changes as per comments

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-02-23 14:18:01 +05:30
Imri Paran
18c22c4178
Fixes #10013: Implement first stage of NoSQL profiler (#15189)
* feat(nosql-profiler): row count

1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.

* added profiler support for mongodb connection

* doc

* use int_admin_ometa in test setup

* - fixed linting issue in gx
- removed unused inheritance

* moved the nosql function into the metric class

* formatting

* validate_compose: raise exception for bad status code.

* fixed import

* format
2024-02-22 11:46:19 +01:00
Pere Miquel Brull
62c0cc7563
#13985 - Azure KV Secrets Manager (#15192)
* #13985 - Azure KV Secrets Manager

* Format

* #13985 - Azure KV Secrets Manager

* #13985 - Azure KV Secrets Manager

* Simplify credentials loading

* Simplify credentials loading

* Simplify credentials loading
2024-02-20 07:18:35 +01:00
Mayur Singal
64e3cedeea
MINOR: Improve Kafka Test Connection (#15207) 2024-02-19 10:11:52 +05:30
Shin-ichi Hashiba
2120b38672
fix-typo: description of default batchsize (#15206) 2024-02-16 08:10:44 +00:00
Teddy
e72c80f264
MINOR - fix test definition deletion (#15203)
* fix: added provider for testDefinition entities

* fix: changed testDefinition <-> testCase relationship + allow recursive deletion of testDefinition

* fix: added migration steps

* style: ran java linting
2024-02-15 18:22:26 +00:00
Supan Shah
4915d6884b
#14881: PinotDB Docs (#14882)
* ISSUE-14881: PinotDB Test Connection Success

* Update ingestion/src/metadata/ingestion/source/database/pinotdb/connection.py

* Update ingestion/src/metadata/ingestion/source/database/pinotdb/connection.py

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-02-13 09:29:23 +01:00
Imri Paran
aeb5fbe303
fixes #12591: add BigTable (#15122)
* feat(connector): add BigTable

* bigtable work

1. docstrings
2. tests
3. created a Row BaseModel
4. implemented a ClassConverter

* docs moved to separate PR

* format files

* minor cosmetic

- removed TODO
- changed headers' year to 2024 for new files
- fixed typos

* format

* formatting and comments

1. added missing docstrings.
2. abstracted the _find_instance method.
3. aliased the IDs used in the BigTable connection

* added comment regarding private key

* added comments regarding column families

* enclose get_schema_name_list in `try/except/else`

* format

* streamlined get_schema_name_list to include all logic in the try block
2024-02-13 08:28:01 +01:00