68 Commits

Author SHA1 Message Date
Suresh Srinivas
ac157a8f0b
Fixes #9618 Allow apostrophe in entity names (#9642)
* Fixes #9618 Allow apostrophe in entity names

* Add support for space in entity names (with exception) and column names

* Add regex checks to entity names, table column names to allow only required characters

* fix: allowed characters in entityName

* Add all the allowed special characters in the entity name in tests

* Allow '&', '/', ':' only for the table

* Override basic entityName by defining entity specific entityName for Python code gen

* fix: dagster migration + json schema path

* fix: python style check

* Fix test failure

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-01-17 11:34:29 -08:00
Teddy
903209bc70
Fixes #9656 -- Add support for datetime type for col between (#9671)
* feat (profiler): Added support for datetime type for sqa profiler

* feat (profiler): migration file

* fix: added comment for ms timestamp logic
2023-01-11 16:51:36 +01:00
Onkar Ravgan
e9ac8ecb85
Fix 7972: Added dbt tag classification name (#9575)
* Added dbt tag classification name

* resolved merge conflicts

* handle ui side change for dbt schema change

* Fixed migration scripts

* fixed failing cypress for dbt

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-01-06 11:08:21 +05:30
Mayur Singal
0a34e18e26
Fix #9562: Add duration in usage ingestion (#9564) 2023-01-05 06:33:07 +00:00
Pere Miquel Brull
329d7ef69e
Fix bigquery migration and deprecate sample data type (#9533) 2022-12-27 19:09:17 +01:00
Suresh Srinivas
758c976cba
Fixes #9259 Change Tags APIs to conform with rest of the APIs (#9260) 2022-12-26 12:32:17 -08:00
Sriharsha Chintalapani
b9242c1390
Fix topic indexes and migrations (#9439)
* Fix topic indexing

* Fix topic db migrations

* Fix topic db migrations

* Fix topic db migrations

* Fix test; add missing topic indexes on python side; fix ranking for tables using usage count
2022-12-21 10:33:12 +01:00
Mohit Yadav
08f6536558
Alert Fixes : removed redundant files, updated Triggwe (#9346)
* Removed Redundant Code + Removed Unnecessary Triggers, Activity Feed Filterin

* all should have intersecting filters

* updated by should have entities index
2022-12-16 21:59:09 +05:30
NiharDoshi99
3190ac2516
Fix: making token as optional field (#9314) 2022-12-16 14:17:47 +00:00
Ashish Gupta
7c181a5a89
Dbt worflow (#9302) 2022-12-16 17:00:28 +05:30
Nahuel
819001182f
Fix#9251: DB2 connection config and ingestion update (#9322)
* DB2 connection config and ingestion update

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v007__create_db_connection_info.sql

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-12-16 07:43:18 +01:00
Mohit Yadav
bfdc32c8c1
Alerts (#9267)
* Schema for Alerts #9248

* 1. Api for alerts
2. Api for alertAction

API for Alert and AlertAction #9266

* missing file

* fix ui failures

* Added Api to give function parameter type and additional Context for UI

* Added default Triggers in Alerts

* checkstyle fix

* docs update

* function names updated

* added timestamp for alert status update

* receiver should be unique in json schema and alerts publishers

* updated bootstrap filters

* updatated review comments

* added functions to match eventType for entities

* added migrations for alerts

* added alert config in alertACtionConfig

* Fix issue in sending mails

* removed commented part

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-12-15 00:14:10 +05:30
Teddy
387cd570f7
Fixes 9192 - Add entity link in response (#9239)
* staging commit

* updated terms to french

* fix(dataInsight): updated ES response to get entityType and total non tiered entities

* fix(dataInsight): removed fields=*

* fix(dataInsight): Added entity migration

* cleaned up branch

* feat: Added entity URL for most viewed entity

* feat(dataInsight): Added href to entity

* feat(DataInsight): Added backend logic for entity href

* Fix conflict resolving typos

* Update openmetadata-service/src/main/resources/json/data/dataInsight/mostViewedEntities.json

Co-authored-by: Nahuel <nahuel@getcollate.io>

* fix: addressed comments from PR review

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-12-13 10:39:09 +00:00
Teddy
4a5cb870ec
Fixes #9192 - Update ES response for DI (#9221)
* staging commit

* updated terms to french

* fix(dataInsight): updated ES response to get entityType and total non tiered entities

* fix(dataInsight): removed fields=*

* fix(dataInsight): Added entity migration

* cleaned up branch
2022-12-09 18:53:17 -08:00
Milan Bariya
2f9f169de9
Fix: Safer default Mark Deleted Tables (#9065)
* Fix: Safer default Mark Deleted Tables

* Fix: Change Based On Comments

* Fix: Change Based On Comments

* Add: Migration Files

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-12-01 16:26:43 +05:30
Sriharsha Chintalapani
4610478688
Fix #8933: Add fixes to initSeedData from 0.12 upgrade (#8934)
* Fix #8933: Add fixes to initSeedData from 0.12 upgrade

* Fix #8933: Add fixes to initSeedData from 0.12 upgrade

* Fix #8933: Add fixes to initSeedData from 0.12 upgrade

* Fix #8933: Add fixes to initSeedData from 0.12 upgrade
2022-11-21 22:50:33 -08:00
Pere Miquel Brull
85ed009daa
Fix openmetadata connection migration (#8891) 2022-11-19 21:07:26 -08:00
Sriharsha Chintalapani
148eb5bf1b
Entity timeseries table missing index on entityFQN causing slower queries (#7968) 2022-11-17 11:22:59 -08:00
Pere Miquel Brull
34ba9d95c5
Ingestion Pipeline deployed, Athena tests and pydantic extras (#8682)
* Always run python tests

* Fix athena tests and types

* Update deployed prop in IngestionPipeline

* Fix #8554

* Format

* Use true as default deployed migration

* Remove repeated req

* Pydantic wiggle room
2022-11-13 11:59:43 +01:00
Suresh Srinivas
6306c9ac8d
Fixes #8688 Enforce mutuallyExclusive tag labels (#8689) 2022-11-12 18:44:17 -08:00
Mohit Yadav
2dbd43a589
Added MetadataServices for creating DataInsight and ElasticSearch Reindex (#8565)
* Added SystemServices for creating DataInsight and ElasticSearch Reindexing Pipeling with Sink config

* Add Amundsen config

* Changed SystemService to MetadataService

* Change to MetadataConnection

* Fixed failing test

* Fixed Failing test

* Added Metadata Service Test

* Bootstrapped Openmetadat service

* checkstyle fix
2022-11-11 13:16:54 +05:30
Suresh Srinivas
86c3ae30f1
Fixes #8596 Introduce mutually exclusive tags and glossary terms (#8597) 2022-11-10 16:47:21 -08:00
Mohit Yadav
5bd4e56b3f
Feat kpis (#8584)
* Added Kpi Resource

* Fix typo

* Fixed failing test

* Review Suggestions
2022-11-09 13:06:13 +05:30
NiharDoshi99
1c1f8660f2
Fixed: dagster to connect to cloud (#8387)
* Fixed: dagster to connect to cloud

* Fixed: changed test_cases

* Fixed: lineage between the assets and jobs

* Changes as per comments

* Fix python checkstyle

* Fix as per comments

* Fix added migration scripts

* minor fixes
2022-11-03 18:19:50 +01:00
Teddy
f4cc2b57a7
Fixes #8353 - Implement Aggregation Endpoint for dataInsight (#8464)
* Moved webanalytics type in its own folder

* Added data insight chart api endpoint

* Jave formatting

* Added resource descriptor

* Added metadata entity endpoint

* Added aggregation endpoint for dataInsight

* Fix tag name

* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight

* Fix pageViewEvent import issue

* Changed dataType to PERCENTAGE and NUMBER

* Cleaned up PR by removing metadata resource and ingestionPipeline sink logic

* Update openmetadata-service/src/main/resources/json/data/dataInsight/totalEntitiesByTier.json

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>

* Update openmetadata-service/src/main/resources/json/data/dataInsight/totalEntitiesByType.json

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-11-03 14:51:14 +05:30
Suresh Srinivas
d47f5738cb
Add support for system provided Tags, Glossaries, Policies, and Roles #8484 (#8508) 2022-11-02 14:56:31 -07:00
Suresh Srinivas
229b56e7b3
Fixes #8236 - Initialize bots from JSON data files (#8304) 2022-10-21 08:38:52 -07:00
Teddy
342e4a24da
Fixes #8154 - Add collect endpoint for web analytic events (#8193)
* Initial implementation of report endpoint

* Modified model files

* Added migration

* Added tests for analytics/report endpoint

* Finalized implementation of the reports endpoint

* Fixed maven test failure

* Fixed java style

* Added policy for reportDefinition

* Fixed minor typos in code

* Added database migration for web analytic event type

* Added logic for collect endpoint

* Added JSON Schema for collect endpoint

* Added index on table columns + added resource description for web event entities

* Fixed policy tests

* Updated add web analytic function to return the body of the request + updated test

* Added referer URL to PageView event

* Changed collection name to align with API endpoint

* updated initSeed method to initializeEntity

* cleaned up PR by removing unused filed from branch

* Fixed requested changes from review

* Clean up names

* Fixed comma in SQL statement
2022-10-20 16:38:54 +02:00
Mayur Singal
ca51fa0245
Fix #7801: Fixed Glue Pipeline (#7832)
Co-authored-by: Apple <apple@Apples-MacBook-Pro.local>
2022-10-02 16:43:35 -07:00
NiharDoshi99
8adcf2ee3d
Fixed: Dagster to work with its API (#7579) 2022-09-23 17:46:40 +05:30
Teddy
90abd73394
Fixes #7105 -- Add support for dataType filter for TestDefinition request (#7640)
* Updated json schema to support dataType filter

* Added migration for test definition support of data type filtering

* Added backend logic to support filter on data type for testdefinitions

* Added migration for:
- supported test types
- renaming of certain tests from starting uppercase to lowercase

* removed openmetadata_db from migration

* added datatype filter in column test

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-09-23 10:31:00 +02:00
Pere Miquel Brull
1578fa7f1b
Fix #7121 - Support Spark metastore DB connection (#7520)
* Fix #7121 - Support Spark metastore DB connection

* appname

* Update docs

* test validation

* Address PR comments

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-09-20 16:47:57 +02:00
Teddy
e22036cc09
Fixes #7504 Added migration steps to delete profiler pipeline from entity relationship (#7511)
* Added migration steps to delete profiler pipeline from entity relationship

* Added migration to remove rows from entity_relationship with profiler pipelines

* Fixed sample data TestDefinition name after renaming testDefinition

* Added user_tokens table to v005

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-09-20 08:14:29 +02:00
mohitdeuex
4c6932e172
[Basicauth] update table name (#7538)
* Fixed issue with testCases not posted in slack

* Return empty string in case of faling to build url

* [BasicAuth] Added Basic Authentication APIs

* [BasicAuth] Checkstyle

* [BasicAuth] Test yaml updated

* [BasicAuth] Update review comments

* [BasicAuth] Refresh logic added

* [BasicAuth] Add sql updates

* [BasicAuth] Maintain dependency version in main pom.xml

* [BasicAuth] Updated as per new module

* [BasicAuth] Fixes for refresh and missing edge scenario handled

* [BasicAuth] remove quotes

* [BasicAuth] Fix merge conflicts

* [BasicAuth] Update table name
2022-09-19 12:03:12 +05:30
mohitdeuex
4b594d0d6e
[BasicAuth] Added Basic Authentication (#7390)
* Fixed issue with testCases not posted in slack

* Return empty string in case of faling to build url

* [BasicAuth] Added Basic Authentication APIs

* [BasicAuth] Checkstyle

* [BasicAuth] Test yaml updated

* [BasicAuth] Update review comments

* [BasicAuth] Refresh logic added

* [BasicAuth] Add sql updates

* [BasicAuth] Maintain dependency version in main pom.xml

* [BasicAuth] Updated as per new module

* [BasicAuth] Fixes for refresh and missing edge scenario handled

* [BasicAuth] remove quotes

* [BasicAuth] Fix merge conflicts
2022-09-19 11:38:45 +05:30
Vivek Ratnavel Subramanian
2f84bfbfa8
Fix #7100 Upgrade issues for 0.12 release (#7101) 2022-09-01 22:22:42 -07:00
Teddy
704385da0a
Clean up profile and test data when migrating from 0.11 to 0.12 (#7039)
* Added SQL to clean up profile and test data when migrating from 0.11 to 0.12

* Added deletion query for ingestion pipeline mysql

* Added logic to delete profiler ingestion pipeline from DB

* Update v004__create_db_connection_info.sql

added semicolon
2022-08-30 18:43:12 +02:00
Sriharsha Chintalapani
f9a3f9988e
Fix #6936: testPlatforms is required field in testSuite and createTes… (#6944)
* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* fixed glossary test

* Fixed sample data ingestion

* Add filters for testCaseResultSuccess
testCaseResultFailed, testCaseResultAborted

* Fix #6936: fix testcase listing

* Fix #6936: fix java style

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-08-27 13:01:24 +05:30
mohitdeuex
c8ab6fa59b
[Backend][Settings] Settings through UI (#6514) 2022-08-25 06:37:44 -07:00
Milan Bariya
994b769d25
Databricks Connection - User & Password clean (#6866)
* Databricks Connection - User & Password clean

* Fix: Python checkstyle failed

* Fix: update the migration scripts
2022-08-23 14:36:58 +02:00
Sriharsha Chintalapani
821d70eae4
Fix #6782: Separate TableProfile and ColumnProfile api calls (#6783)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6782: Separate TableProfile and ColumnProfile api calls

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix setFields

* Fix tests

* Update pipeline status endpoint

* updated ui side as per new schema for profiler tab

* updated profiler details with new API

* Fix Profiler tests and validation errors (#6827)

* add profilerSample field in TableProfile

* add profilerSample field in TableProfile

* get columnProfile with field profile

* get columnProfile with field profile

* Fixed sample data and python tests

* fixed date range filter change issue

* handled empty profiler case

* Added column level test case and results

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-22 21:31:24 +05:30
Pere Miquel Brull
3f1fc5344a
Remove Athena hostPort (#6825) 2022-08-19 14:11:53 +02:00
Sriharsha Chintalapani
0a2e2d8b18
Fix #6571: Add EntityLink for the testCase to ID columns (#6755)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* added entityLink to python tests

* Fix tests

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-18 00:29:59 -07:00
Parth Panchal
e451a9ca5f
Fixed#5400: Added Kafka Event Publisher (#6697)
* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher
2022-08-11 20:27:15 +05:30
Suresh Srinivas
7825986105
Fixes #6554 Add support for team of type Group (#6588)
* Fixes #6554 Add support for team of type Group

* Fixing test failure
2022-08-05 10:49:42 -07:00
Mayur Singal
5e1b124d57
Fix #6530 & Fix #4975: Oracle Connection Enhancements (#6579)
* Fix #6530 & Fix #4975: Oracle Connection Enhancements

* updated migrations & docs
2022-08-05 10:28:04 +05:30
Sriharsha Chintalapani
1a42428e42
Add time series extention (#6416)
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-08-04 07:22:47 -07:00
Pere Miquel Brull
157fcd8dcf
DeltaLake does not support profiler (#6495) 2022-08-02 08:45:43 +02:00
Vivek Ratnavel Subramanian
62877c2e5d
Fix #6333 Backend: Add API support to create and list announcements (#6446) 2022-07-31 21:02:49 -07:00
Sriharsha Chintalapani
5f8e2a7f4f
Generic Test Case definition to capture different testcases (#5647) 2022-07-28 13:58:41 -07:00