147 Commits

Author SHA1 Message Date
Cristian Calugaru
3f61c3e05d
Issue 9064 - New Object Store Service and Container models (#10283)
* First cut of the store service model

* Second cut of the store service model

* reverted accidental rename of property

* Third version for the object container schema definition.

* final draft version for container and object store service related entities

* final first draft of api's and entities for the object store service and container

* formatting

* removed test file

* formatting

* test fixes

* formatting

* more fixed tests

* fixed tests, addressed PR comments

* added type for object store connections

* addressed PR comments

* formatting

* renamed objectstoreservice to service in schemas. Changed type to fullyQualifiedEntityName in createContainer schema
2023-02-24 16:34:08 -08:00
Mayur Singal
0fafbf70ba
Support postgres multiple DB ingestion for user with no access (#10207) 2023-02-23 15:42:40 +05:30
Ayush Shah
785142d86a
Add policy tags from Bigquery (#10189) 2023-02-20 19:13:45 +00:00
Suresh Srinivas
57093e583a
Fixes #10212 User email address should be unique (#10214) 2023-02-15 21:04:23 -08:00
Nahuel
da9b534194
Fix: bug in migration when removin sample data services (#10032) 2023-01-31 15:11:09 +00:00
Mayur Singal
939adf887b
Fix #8973: Superset fetch metadata from DB (#9645)
* Docs #8973: Superset with SSO docs

* Superset support for mysql & postgres db

* remove unwanted field from yaml

* Fix pylint

* Fix superset tests

* Fix sample data connection

* ci fix & review comments

* cypress & defualt provider fix
2023-01-19 16:39:43 +01:00
Suresh Srinivas
ac157a8f0b
Fixes #9618 Allow apostrophe in entity names (#9642)
* Fixes #9618 Allow apostrophe in entity names

* Add support for space in entity names (with exception) and column names

* Add regex checks to entity names, table column names to allow only required characters

* fix: allowed characters in entityName

* Add all the allowed special characters in the entity name in tests

* Allow '&', '/', ':' only for the table

* Override basic entityName by defining entity specific entityName for Python code gen

* fix: dagster migration + json schema path

* fix: python style check

* Fix test failure

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-01-17 11:34:29 -08:00
Teddy
903209bc70
Fixes #9656 -- Add support for datetime type for col between (#9671)
* feat (profiler): Added support for datetime type for sqa profiler

* feat (profiler): migration file

* fix: added comment for ms timestamp logic
2023-01-11 16:51:36 +01:00
Onkar Ravgan
e9ac8ecb85
Fix 7972: Added dbt tag classification name (#9575)
* Added dbt tag classification name

* resolved merge conflicts

* handle ui side change for dbt schema change

* Fixed migration scripts

* fixed failing cypress for dbt

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-01-06 11:08:21 +05:30
Mayur Singal
0a34e18e26
Fix #9562: Add duration in usage ingestion (#9564) 2023-01-05 06:33:07 +00:00
Pere Miquel Brull
329d7ef69e
Fix bigquery migration and deprecate sample data type (#9533) 2022-12-27 19:09:17 +01:00
Suresh Srinivas
758c976cba
Fixes #9259 Change Tags APIs to conform with rest of the APIs (#9260) 2022-12-26 12:32:17 -08:00
Sriharsha Chintalapani
b9242c1390
Fix topic indexes and migrations (#9439)
* Fix topic indexing

* Fix topic db migrations

* Fix topic db migrations

* Fix topic db migrations

* Fix test; add missing topic indexes on python side; fix ranking for tables using usage count
2022-12-21 10:33:12 +01:00
Mohit Yadav
08f6536558
Alert Fixes : removed redundant files, updated Triggwe (#9346)
* Removed Redundant Code + Removed Unnecessary Triggers, Activity Feed Filterin

* all should have intersecting filters

* updated by should have entities index
2022-12-16 21:59:09 +05:30
NiharDoshi99
3190ac2516
Fix: making token as optional field (#9314) 2022-12-16 14:17:47 +00:00
Ashish Gupta
7c181a5a89
Dbt worflow (#9302) 2022-12-16 17:00:28 +05:30
Nahuel
819001182f
Fix#9251: DB2 connection config and ingestion update (#9322)
* DB2 connection config and ingestion update

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v007__create_db_connection_info.sql

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-12-16 07:43:18 +01:00
Mohit Yadav
bfdc32c8c1
Alerts (#9267)
* Schema for Alerts #9248

* 1. Api for alerts
2. Api for alertAction

API for Alert and AlertAction #9266

* missing file

* fix ui failures

* Added Api to give function parameter type and additional Context for UI

* Added default Triggers in Alerts

* checkstyle fix

* docs update

* function names updated

* added timestamp for alert status update

* receiver should be unique in json schema and alerts publishers

* updated bootstrap filters

* updatated review comments

* added functions to match eventType for entities

* added migrations for alerts

* added alert config in alertACtionConfig

* Fix issue in sending mails

* removed commented part

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-12-15 00:14:10 +05:30
Teddy
387cd570f7
Fixes 9192 - Add entity link in response (#9239)
* staging commit

* updated terms to french

* fix(dataInsight): updated ES response to get entityType and total non tiered entities

* fix(dataInsight): removed fields=*

* fix(dataInsight): Added entity migration

* cleaned up branch

* feat: Added entity URL for most viewed entity

* feat(dataInsight): Added href to entity

* feat(DataInsight): Added backend logic for entity href

* Fix conflict resolving typos

* Update openmetadata-service/src/main/resources/json/data/dataInsight/mostViewedEntities.json

Co-authored-by: Nahuel <nahuel@getcollate.io>

* fix: addressed comments from PR review

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-12-13 10:39:09 +00:00
Teddy
4a5cb870ec
Fixes #9192 - Update ES response for DI (#9221)
* staging commit

* updated terms to french

* fix(dataInsight): updated ES response to get entityType and total non tiered entities

* fix(dataInsight): removed fields=*

* fix(dataInsight): Added entity migration

* cleaned up branch
2022-12-09 18:53:17 -08:00
Milan Bariya
2f9f169de9
Fix: Safer default Mark Deleted Tables (#9065)
* Fix: Safer default Mark Deleted Tables

* Fix: Change Based On Comments

* Fix: Change Based On Comments

* Add: Migration Files

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-12-01 16:26:43 +05:30
Sriharsha Chintalapani
4610478688
Fix #8933: Add fixes to initSeedData from 0.12 upgrade (#8934)
* Fix #8933: Add fixes to initSeedData from 0.12 upgrade

* Fix #8933: Add fixes to initSeedData from 0.12 upgrade

* Fix #8933: Add fixes to initSeedData from 0.12 upgrade

* Fix #8933: Add fixes to initSeedData from 0.12 upgrade
2022-11-21 22:50:33 -08:00
Pere Miquel Brull
85ed009daa
Fix openmetadata connection migration (#8891) 2022-11-19 21:07:26 -08:00
Sriharsha Chintalapani
148eb5bf1b
Entity timeseries table missing index on entityFQN causing slower queries (#7968) 2022-11-17 11:22:59 -08:00
Pere Miquel Brull
34ba9d95c5
Ingestion Pipeline deployed, Athena tests and pydantic extras (#8682)
* Always run python tests

* Fix athena tests and types

* Update deployed prop in IngestionPipeline

* Fix #8554

* Format

* Use true as default deployed migration

* Remove repeated req

* Pydantic wiggle room
2022-11-13 11:59:43 +01:00
Suresh Srinivas
6306c9ac8d
Fixes #8688 Enforce mutuallyExclusive tag labels (#8689) 2022-11-12 18:44:17 -08:00
Mohit Yadav
2dbd43a589
Added MetadataServices for creating DataInsight and ElasticSearch Reindex (#8565)
* Added SystemServices for creating DataInsight and ElasticSearch Reindexing Pipeling with Sink config

* Add Amundsen config

* Changed SystemService to MetadataService

* Change to MetadataConnection

* Fixed failing test

* Fixed Failing test

* Added Metadata Service Test

* Bootstrapped Openmetadat service

* checkstyle fix
2022-11-11 13:16:54 +05:30
Suresh Srinivas
86c3ae30f1
Fixes #8596 Introduce mutually exclusive tags and glossary terms (#8597) 2022-11-10 16:47:21 -08:00
Mohit Yadav
5bd4e56b3f
Feat kpis (#8584)
* Added Kpi Resource

* Fix typo

* Fixed failing test

* Review Suggestions
2022-11-09 13:06:13 +05:30
Pere Miquel Brull
a7347e1873
Fix Dagster migration script (#8539) 2022-11-04 09:43:52 +01:00
Sriharsha Chintalapani
5c5514a8b9
Fix mysql migration script syntax (#8536) 2022-11-04 08:31:33 +01:00
NiharDoshi99
1c1f8660f2
Fixed: dagster to connect to cloud (#8387)
* Fixed: dagster to connect to cloud

* Fixed: changed test_cases

* Fixed: lineage between the assets and jobs

* Changes as per comments

* Fix python checkstyle

* Fix as per comments

* Fix added migration scripts

* minor fixes
2022-11-03 18:19:50 +01:00
Teddy
f4cc2b57a7
Fixes #8353 - Implement Aggregation Endpoint for dataInsight (#8464)
* Moved webanalytics type in its own folder

* Added data insight chart api endpoint

* Jave formatting

* Added resource descriptor

* Added metadata entity endpoint

* Added aggregation endpoint for dataInsight

* Fix tag name

* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight

* Fix pageViewEvent import issue

* Changed dataType to PERCENTAGE and NUMBER

* Cleaned up PR by removing metadata resource and ingestionPipeline sink logic

* Update openmetadata-service/src/main/resources/json/data/dataInsight/totalEntitiesByTier.json

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>

* Update openmetadata-service/src/main/resources/json/data/dataInsight/totalEntitiesByType.json

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-11-03 14:51:14 +05:30
Suresh Srinivas
d47f5738cb
Add support for system provided Tags, Glossaries, Policies, and Roles #8484 (#8508) 2022-11-02 14:56:31 -07:00
Finn McClusky
3f7c8dc2e2
FIX 8196: Remove the create-ingestion-bot and update-ingestion-bot from bootstrap_storage.sh (#8202)
* ISSUE-8196: removed create-ingestion-bot, update-ingestion-bot options and unused functions from table initializer

* java checkstyle fix

Co-authored-by: Finn McClusky <finn.mcclusky@gmail.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-10-21 22:41:02 +05:30
Suresh Srinivas
229b56e7b3
Fixes #8236 - Initialize bots from JSON data files (#8304) 2022-10-21 08:38:52 -07:00
Teddy
342e4a24da
Fixes #8154 - Add collect endpoint for web analytic events (#8193)
* Initial implementation of report endpoint

* Modified model files

* Added migration

* Added tests for analytics/report endpoint

* Finalized implementation of the reports endpoint

* Fixed maven test failure

* Fixed java style

* Added policy for reportDefinition

* Fixed minor typos in code

* Added database migration for web analytic event type

* Added logic for collect endpoint

* Added JSON Schema for collect endpoint

* Added index on table columns + added resource description for web event entities

* Fixed policy tests

* Updated add web analytic function to return the body of the request + updated test

* Added referer URL to PageView event

* Changed collection name to align with API endpoint

* updated initSeed method to initializeEntity

* cleaned up PR by removing unused filed from branch

* Fixed requested changes from review

* Clean up names

* Fixed comma in SQL statement
2022-10-20 16:38:54 +02:00
Mayur Singal
ca51fa0245
Fix #7801: Fixed Glue Pipeline (#7832)
Co-authored-by: Apple <apple@Apples-MacBook-Pro.local>
2022-10-02 16:43:35 -07:00
NiharDoshi99
8adcf2ee3d
Fixed: Dagster to work with its API (#7579) 2022-09-23 17:46:40 +05:30
Teddy
90abd73394
Fixes #7105 -- Add support for dataType filter for TestDefinition request (#7640)
* Updated json schema to support dataType filter

* Added migration for test definition support of data type filtering

* Added backend logic to support filter on data type for testdefinitions

* Added migration for:
- supported test types
- renaming of certain tests from starting uppercase to lowercase

* removed openmetadata_db from migration

* added datatype filter in column test

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-09-23 10:31:00 +02:00
Pere Miquel Brull
1578fa7f1b
Fix #7121 - Support Spark metastore DB connection (#7520)
* Fix #7121 - Support Spark metastore DB connection

* appname

* Update docs

* test validation

* Address PR comments

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-09-20 16:47:57 +02:00
Teddy
e22036cc09
Fixes #7504 Added migration steps to delete profiler pipeline from entity relationship (#7511)
* Added migration steps to delete profiler pipeline from entity relationship

* Added migration to remove rows from entity_relationship with profiler pipelines

* Fixed sample data TestDefinition name after renaming testDefinition

* Added user_tokens table to v005

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-09-20 08:14:29 +02:00
mohitdeuex
4c6932e172
[Basicauth] update table name (#7538)
* Fixed issue with testCases not posted in slack

* Return empty string in case of faling to build url

* [BasicAuth] Added Basic Authentication APIs

* [BasicAuth] Checkstyle

* [BasicAuth] Test yaml updated

* [BasicAuth] Update review comments

* [BasicAuth] Refresh logic added

* [BasicAuth] Add sql updates

* [BasicAuth] Maintain dependency version in main pom.xml

* [BasicAuth] Updated as per new module

* [BasicAuth] Fixes for refresh and missing edge scenario handled

* [BasicAuth] remove quotes

* [BasicAuth] Fix merge conflicts

* [BasicAuth] Update table name
2022-09-19 12:03:12 +05:30
mohitdeuex
4b594d0d6e
[BasicAuth] Added Basic Authentication (#7390)
* Fixed issue with testCases not posted in slack

* Return empty string in case of faling to build url

* [BasicAuth] Added Basic Authentication APIs

* [BasicAuth] Checkstyle

* [BasicAuth] Test yaml updated

* [BasicAuth] Update review comments

* [BasicAuth] Refresh logic added

* [BasicAuth] Add sql updates

* [BasicAuth] Maintain dependency version in main pom.xml

* [BasicAuth] Updated as per new module

* [BasicAuth] Fixes for refresh and missing edge scenario handled

* [BasicAuth] remove quotes

* [BasicAuth] Fix merge conflicts
2022-09-19 11:38:45 +05:30
Sriharsha Chintalapani
656b50dd3a
Fix #7469: Refactor OpenMetadata code modules (#7474) 2022-09-14 23:14:02 -07:00
Vivek Ratnavel Subramanian
2f84bfbfa8
Fix #7100 Upgrade issues for 0.12 release (#7101) 2022-09-01 22:22:42 -07:00
Teddy
704385da0a
Clean up profile and test data when migrating from 0.11 to 0.12 (#7039)
* Added SQL to clean up profile and test data when migrating from 0.11 to 0.12

* Added deletion query for ingestion pipeline mysql

* Added logic to delete profiler ingestion pipeline from DB

* Update v004__create_db_connection_info.sql

added semicolon
2022-08-30 18:43:12 +02:00
Sriharsha Chintalapani
f9a3f9988e
Fix #6936: testPlatforms is required field in testSuite and createTes… (#6944)
* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* fixed glossary test

* Fixed sample data ingestion

* Add filters for testCaseResultSuccess
testCaseResultFailed, testCaseResultAborted

* Fix #6936: fix testcase listing

* Fix #6936: fix java style

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-08-27 13:01:24 +05:30
mohitdeuex
c8ab6fa59b
[Backend][Settings] Settings through UI (#6514) 2022-08-25 06:37:44 -07:00
Milan Bariya
994b769d25
Databricks Connection - User & Password clean (#6866)
* Databricks Connection - User & Password clean

* Fix: Python checkstyle failed

* Fix: update the migration scripts
2022-08-23 14:36:58 +02:00