394 Commits

Author SHA1 Message Date
Sachin Chaurasiya
457f3d919a
GEN-1322: API Entity - Remove Beta (#17967)
* GEN-1322: API Entity - Remove Beta

* minor: add doc for the metadata pipeline

* api service refactor

* api service refactor backend changes

* add apiconnection in test service connection

* pytest fix

* fix java file formatting

* Fix casing of REST in ApiServiceRest.spec.ts

* Refactor REST to Rest in API classes

* minor change

* minor change

* minor change

* fix cashing for API to Api

* add playwright test for api service ingestion

* fix: playwright test

---------

Co-authored-by: harshsoni2024 <harshsoni2024@gmail.com>
2024-10-08 14:39:55 +05:30
Ayush Shah
23c6aa3ca5
fix: add children to array of struct (#17850) 2024-09-16 18:02:17 +05:30
Pere Miquel Brull
a098c20c7c
MINOR - LKML sample data (#17359) 2024-08-10 18:01:00 +02:00
IceS2
c522f14178
MINOR: Refactor output_handlers to a WorkflowOutputHandler class (#17149)
* Refactor output_handlers to a WorkflowOutputHandler class

* Add old methods as deprecated to avoid breaking changes

* Extract WorkflowInitErrorHandler from workflow_output_handler

* Fix static checks

* Fix tests

* Fix tests

* Update code based on comments from PR

* Update comment
2024-07-29 09:20:34 +02:00
Teddy
c336d86a44
chore: add "learning" sample data for dynamic assertion (#17155) 2024-07-25 08:24:15 +02:00
harshsoni2024
d5af2ba6b6
sample data for open metadata user & table apis (#16981) 2024-07-12 10:32:21 +05:30
harshsoni2024
0caa8a8018
API service sample data (#16971)
* add sample data for api service

* pylint fix

* correct service connection json

* fix sample data

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-07-09 22:06:47 -07:00
Teddy
c361305902
MINOR - fix sample data ingestion (#16949)
* fix: sample data ingestion

* style: ran python linting
2024-07-09 10:18:55 +02:00
Imri Paran
2f0a8efceb
MINOR: added table diff sample results (#16844)
* chore(sample-data): table diff

added table diff results sample

* format
2024-07-01 15:28:14 +00:00
Teddy
38fe061227
MINOR -- Add Test Definition Dimension (#16769)
* feat: added test defintion dimension + sample data for bounds

* chore: added migration for definition dimension

* style: ran python linting

* fix: rename dimension to dataQualityDimension

* fix: test definition dimension key
2024-06-24 15:01:12 +00:00
Pere Miquel Brull
d8e2187980
#15243 - Pydantic V2 & Airflow 2.9 (#16480)
* pydantic v2

* pydanticv2

* fix parser

* fix annotated

* fix model dumping

* mysql ingestion

* clean root models

* clean root models

* bump airflow

* bump airflow

* bump airflow

* optionals

* optionals

* optionals

* jdk

* airflow migrate

* fab provider

* fab provider

* fab provider

* some more fixes

* fixing tests and imports

* model_dump and model_validate

* model_dump and model_validate

* model_dump and model_validate

* union

* pylint

* pylint

* integration tests

* fix CostAnalysisReportData

* integration tests

* tests

* missing defaults

* missing defaults
2024-06-05 21:18:37 +02:00
Pere Miquel Brull
f4369ded8d
MINOR - Allow dict-based xlets in Airflow (#16262) 2024-05-15 15:08:09 +02:00
Suman Maharana
488078da8a
Add DDL query ingest (#15860) 2024-05-06 18:03:50 +05:30
Imri Paran
4ac5912d4c
MINOR: added TestCase inspection query to backend and sample data (#16003)
* added TestCase inspection query to backend and sample data

* format

* format
2024-04-26 11:49:08 +02:00
IceS2
fd51df25fa
Removed stale examples. Update dependencies (#15951) 2024-04-18 17:41:09 +02:00
Imri Paran
706d1ab97e
fixed ingestion of sample data for failed sample rows (#15879) 2024-04-15 07:59:27 +02:00
Imri Paran
b2ce491ff1
MINOR: Add failed rows sample to test case (#15682)
* add failed sample data

* format

* fixed masking pii data in test failed rows sample

* format

* failedRowsSamples -> failedRowsSample

* failedRowsSamples -> failedRowsSample

* fixed tests

* format

* wip

* added computePassedFailedRowCount to python client

* comment for loggerLevel

* format

* fixed tests

* tests for putting / deleting failed samples

* format

* format

* added test case for pii test

* changed method name to deleteTestCaseFailedRowsSample

* added getComputePassedFailedRowCount
2024-04-10 17:00:00 +02:00
Ayush Shah
0d0d820305
MINOR: Add Static Sample Data, Modify Lineage (#14752) 2024-01-17 20:07:49 +05:30
Mayur Singal
07ea5e97c8
Minor: Postgres cypress move to query log file usage (#14750) 2024-01-17 14:26:57 +05:30
Ayush Shah
9c6d202555
Add Sample data, modify regex pattern (#14467) 2024-01-11 14:23:33 +05:30
Pere Miquel Brull
0255171218
MINOR - Create Test Case Resolution ts entry & delete resolution when… (#14541)
* MINOR - Create Test Case Resolution ts entry & delete resolution when Test Case is deleted
2024-01-05 09:15:49 +01:00
Pere Miquel Brull
e16ad3c5dc
MINOR - Clean ingestion ES code and auth providers (#14358)
* MINOR - Clean ingestion ES code and auth providers

* clean parser

* Clean security config for the client

* Clean security config for the client

* Improve class conversion exceptions

* Fix tests

* Clean up java client

* Clean up java client

* clean parser

* Fix test

* Fix test

* fix NO_AUTH error

* Fix test

* Format

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-19 12:08:48 +01:00
Teddy
3bbf55fcda
FIXES #14049 - Split test case resolution status from test case result (#14204)
* refactor: entityFQN as ListFilter condition

* feat: implement resolution entity timeseries

* fix: rename to testCaseResolutionStatus

* ref: extracted ES query builder into private method

* ref: extract OS query builder in its own method

* ref: remove ingestion logic for test case resolution

* fix: reorganize json schemas to fix circular import in Python

* ref: object names in typescript code

* feat: added indexing of test case resolution

* feat: added test case resolution sample data

* fix: test case resolution api logic

* fix: audit logger for entityTimeSeriesInterface

* fix: DDL generation

* style: python linting

* fix: skip UI test case resolution tests

* fix: remove extension field

* fix: renamed testCaseFailureStatus to testCaseResolutionStatus

* fix: remove reviewer

* fix: rename sequenceId to stateId

* fix: re adjust search weights

* fix: removed InReview status

* style: ran python linting
2023-12-04 23:18:01 -08:00
Ashish Gupta
42463ff40b
#14134: supported retention period in table entity (#14163)
* supported retention period in table entity

* Add retention period updates

* supported unit test

* added curd operation for retention center

* minor changes

* fix modal issues and added validation

* added unit test for retention period

* fix code smell

* fix sonarcloud

* minor chnages

* Fix java code styling

* added hours in retention period

* changes as per comments

* fix sonar

* remove localization keys

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2023-12-05 10:42:37 +05:30
Pere Miquel Brull
7fcdf08ca4
#11626 & #14131 - Lineage with other Entities & attr-based xlets (#14191)
* Add OMEntity model

* Test OMEntity

* Update repr

* Fix __str__

* Add entity ref map

* Test serializer for backend

* Fix tests

* Fix serializer

* Test runner

* Add runner tests

* Update docs

* Format
2023-12-01 06:29:44 +01:00
Teddy
8ff70e31fe
MINOR: fixes ingestion of sample data for custom metrics (#14170)
* fix: updated sample data ingestoon for custom metrics

* style: fix python linting

* style: fix java linting
2023-11-29 17:28:19 +01:00
Teddy
131eea32f8
Add Custom metrics sample data (#14069)
* feat: add sample data for custom metrics

* feat: update project.scripts to `metadata`
2023-11-22 11:57:19 +01:00
Onkar Ravgan
9d58b56a1c
Added stored procedures sample data (#13838)
* Added stored proc sample data

* Added sp lineage
2023-11-03 14:05:02 +01:00
07Himank
6ffe79f793
fixed ES Indexing for very large S3 Storage Service buckets fails (#13507) 2023-10-13 10:22:53 +05:30
Teddy
e57849b732
Fixes #12298 - Update report data type to camel case (#13505)
* fix: updated DI to camelCase

* fix: ran linting

* fix: added migration

* fix: remove extra parenthesis in migration file

* fix: psql migration query

* fix: OS compose host

* fix: removed commented code block
2023-10-11 08:14:21 +02:00
Teddy
eefce68015
fix: updated DI cost analysis aggregated report (#13498) 2023-10-10 07:04:40 +02:00
Pere Miquel Brull
f6a87ee02a
Fix #12082 - Bump PyAthena version (#13464) 2023-10-09 20:47:19 +02:00
Onkar Ravgan
1e48d2ecff
Added sd changes (#13446)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-05 12:24:32 +02:00
Teddy
9ef3ff7a58
Cost analysis agg (#13408)
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: added unusedReport to DI

* feat: added aggregated unused report

* feat: ran linting

* feat: reverted compose file changes

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-03 09:27:18 +02:00
07Himank
74e29a9f16
Generic search changes. (#13326)
* working on new search changes

* working on new search changes

* working

* working

* owner propogation done

* working on propogation

* done

* change in storageservice index

* Merge conflict fix

* Draft changes

* working on making updates generic

* added code to opensearchClientImp

* renamed suppportsSearchIndex to supportsSearch

* checkstyle

* added generic code for deleted as well

* fix tests

* fix all tests

* addressing comments

* fixed test case failure

* Fix lifecycle validation error name typo

* fix realted domain propogation

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-09-27 10:48:33 -07:00
Teddy
e9ef7b5e81
Issue-12857: Remove ES Dependency from DI Workflow (#13303)
* feat: move elasticsearch indexing to backend + introduced EntityTimeSeries interface for timeseries type object

* feat: make reportData.json inherit from EntityTimeSeriesInterface

* feat: updated type to Object

* feat: deleted elasticsearch dependencies

* feat: removed elasticsearch indexing from workflow

* feat: added data insight sample data

* feat: cleaned up tests
2023-09-21 16:17:47 -07:00
Onkar Ravgan
8acebbb892
Update ingestion logic to use PATCH API for lifeCycle info (#13283) 2023-09-21 16:40:09 +05:30
Sriharsha Chintalapani
e0ecf49585
Fix #11970: Search by FQN; Refactor Search Indexing, Add API to searc… (#13271)
* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Add wildcard support

* Fix GlossaryTerm Patch
2023-09-20 14:40:10 +05:30
Sriharsha Chintalapani
c2ed4f422f
Fix LifeCycle inconsistencies in Schema, make it common entity field (#13252)
* Fix LifeCycle inconsistencies in Schema; Add DELETE api

* set autocommit to true for non transactional

* make lifecycle common field for entities

* Add LifeCycle as common entity field

* Fix python life cycle code

* Fix search indexes

* remove unnecessary constant

* Add test back to entity resource test

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Add missing schema

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-19 14:03:57 +02:00
Onkar Ravgan
1e4d48a034
Added Life Cycle sample data and changed datetime to timestamp (#13141) 2023-09-13 10:59:19 +05:30
Mayur Singal
4e633877b3
Fix ElasticSearch Test Connection & Deploy (#13061) 2023-09-08 12:40:48 +05:30
Pere Miquel Brull
a9b13de893
Move workflows back into src (#13100)
* Move workflows back into src

* Fix test
2023-09-06 11:30:33 +02:00
Onkar Ravgan
a3ca8b6e66
Added project filter pattern to dashboard entity (#12925) 2023-09-05 11:01:28 +05:30
gauthk6
ba2201f4ea
draft: implementation of lightdash connector. (#12957)
* Fixes a bug while patching the description of a TestCase

* Update docker-compose.yml

* Update docker-compose.yml

* Ran pre-commit checks and linter

* Added some clarifying points and fixed some grammatical errors in the documentation for installation instructions.

* revert changes made to docs

* implementation of lightdash connector.

* ui: add icon for lightdash service

* chore: update lightdash image

* chore: update the icon

* Cleaned up code (took out debug statements, etc.). Still TODO: yield_dashboard_lineage_details() not being called just yet.

* fix: ran linting

* added null checks

* Delete openmetadata-server.md

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2023-08-31 17:28:07 +02:00
Pere Miquel Brull
6c0e9f5061
Part of #7272 - Centralize Workflows, Status, and Exception Management (#13029)
* Prep changes

* Prep changes

* prep changes

* Update imports

* Format

* Prep delete

* Prep delete

* Fix sink

* Prep test

* Commit

* passing either

* passing either

* Prep Either

* Metadata source with Either

* Update status

* Merge remote-tracking branch 'upstream/main' into issue-7272

* Format

* Linting

* Linting

* Linting

* Linting

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Comments
2023-08-30 15:49:42 +02:00
vanshika18
3cc15e6d0b
Fixed #8418 added new connector couchbase (#12836) 2023-08-29 11:16:32 +05:30
Sriharsha Chintalapani
19b5c946a7
Fix #12167: Support for Stored Procedures as another entity under Database Schema (#12999)
* Add Stored Procedure Entity

* Stored Procedure repository

* Stored Procedure repository

* Fix #12998: Support for Stored Procedures as another entity under Database Schema

* Fix #12998: Support for Stored Procedures as another entity under Database Schema
2023-08-25 08:14:30 +02:00
vanshika18
52153dba98
fixed #10691 support multiDatabase, usage & lineage support for Azure SQL (#12982) 2023-08-24 17:47:41 +05:30
Pere Miquel Brull
191754ff20
Set Test Suite FQN (#12890)
* Set Test Suite FQN

* quoteName

* quoteFqn is false for TestSuites

* Fix tests
2023-08-18 18:56:26 +05:30
Mayur Singal
2f7a2193e0
Fix #12302: Add ElasticSearch Ingestion Source (#12892) 2023-08-18 15:10:31 +05:30