415 Commits

Author SHA1 Message Date
Teddy
f4cc2b57a7
Fixes #8353 - Implement Aggregation Endpoint for dataInsight (#8464)
* Moved webanalytics type in its own folder

* Added data insight chart api endpoint

* Jave formatting

* Added resource descriptor

* Added metadata entity endpoint

* Added aggregation endpoint for dataInsight

* Fix tag name

* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight

* Fix pageViewEvent import issue

* Changed dataType to PERCENTAGE and NUMBER

* Cleaned up PR by removing metadata resource and ingestionPipeline sink logic

* Update openmetadata-service/src/main/resources/json/data/dataInsight/totalEntitiesByTier.json

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>

* Update openmetadata-service/src/main/resources/json/data/dataInsight/totalEntitiesByType.json

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-11-03 14:51:14 +05:30
Suresh Srinivas
d47f5738cb
Add support for system provided Tags, Glossaries, Policies, and Roles #8484 (#8508) 2022-11-02 14:56:31 -07:00
Finn McClusky
3f7c8dc2e2
FIX 8196: Remove the create-ingestion-bot and update-ingestion-bot from bootstrap_storage.sh (#8202)
* ISSUE-8196: removed create-ingestion-bot, update-ingestion-bot options and unused functions from table initializer

* java checkstyle fix

Co-authored-by: Finn McClusky <finn.mcclusky@gmail.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-10-21 22:41:02 +05:30
Suresh Srinivas
229b56e7b3
Fixes #8236 - Initialize bots from JSON data files (#8304) 2022-10-21 08:38:52 -07:00
Teddy
342e4a24da
Fixes #8154 - Add collect endpoint for web analytic events (#8193)
* Initial implementation of report endpoint

* Modified model files

* Added migration

* Added tests for analytics/report endpoint

* Finalized implementation of the reports endpoint

* Fixed maven test failure

* Fixed java style

* Added policy for reportDefinition

* Fixed minor typos in code

* Added database migration for web analytic event type

* Added logic for collect endpoint

* Added JSON Schema for collect endpoint

* Added index on table columns + added resource description for web event entities

* Fixed policy tests

* Updated add web analytic function to return the body of the request + updated test

* Added referer URL to PageView event

* Changed collection name to align with API endpoint

* updated initSeed method to initializeEntity

* cleaned up PR by removing unused filed from branch

* Fixed requested changes from review

* Clean up names

* Fixed comma in SQL statement
2022-10-20 16:38:54 +02:00
Mayur Singal
ca51fa0245
Fix #7801: Fixed Glue Pipeline (#7832)
Co-authored-by: Apple <apple@Apples-MacBook-Pro.local>
2022-10-02 16:43:35 -07:00
NiharDoshi99
8adcf2ee3d
Fixed: Dagster to work with its API (#7579) 2022-09-23 17:46:40 +05:30
Teddy
90abd73394
Fixes #7105 -- Add support for dataType filter for TestDefinition request (#7640)
* Updated json schema to support dataType filter

* Added migration for test definition support of data type filtering

* Added backend logic to support filter on data type for testdefinitions

* Added migration for:
- supported test types
- renaming of certain tests from starting uppercase to lowercase

* removed openmetadata_db from migration

* added datatype filter in column test

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-09-23 10:31:00 +02:00
Pere Miquel Brull
1578fa7f1b
Fix #7121 - Support Spark metastore DB connection (#7520)
* Fix #7121 - Support Spark metastore DB connection

* appname

* Update docs

* test validation

* Address PR comments

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-09-20 16:47:57 +02:00
Teddy
e22036cc09
Fixes #7504 Added migration steps to delete profiler pipeline from entity relationship (#7511)
* Added migration steps to delete profiler pipeline from entity relationship

* Added migration to remove rows from entity_relationship with profiler pipelines

* Fixed sample data TestDefinition name after renaming testDefinition

* Added user_tokens table to v005

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-09-20 08:14:29 +02:00
mohitdeuex
4c6932e172
[Basicauth] update table name (#7538)
* Fixed issue with testCases not posted in slack

* Return empty string in case of faling to build url

* [BasicAuth] Added Basic Authentication APIs

* [BasicAuth] Checkstyle

* [BasicAuth] Test yaml updated

* [BasicAuth] Update review comments

* [BasicAuth] Refresh logic added

* [BasicAuth] Add sql updates

* [BasicAuth] Maintain dependency version in main pom.xml

* [BasicAuth] Updated as per new module

* [BasicAuth] Fixes for refresh and missing edge scenario handled

* [BasicAuth] remove quotes

* [BasicAuth] Fix merge conflicts

* [BasicAuth] Update table name
2022-09-19 12:03:12 +05:30
mohitdeuex
4b594d0d6e
[BasicAuth] Added Basic Authentication (#7390)
* Fixed issue with testCases not posted in slack

* Return empty string in case of faling to build url

* [BasicAuth] Added Basic Authentication APIs

* [BasicAuth] Checkstyle

* [BasicAuth] Test yaml updated

* [BasicAuth] Update review comments

* [BasicAuth] Refresh logic added

* [BasicAuth] Add sql updates

* [BasicAuth] Maintain dependency version in main pom.xml

* [BasicAuth] Updated as per new module

* [BasicAuth] Fixes for refresh and missing edge scenario handled

* [BasicAuth] remove quotes

* [BasicAuth] Fix merge conflicts
2022-09-19 11:38:45 +05:30
Sriharsha Chintalapani
656b50dd3a
Fix #7469: Refactor OpenMetadata code modules (#7474) 2022-09-14 23:14:02 -07:00
Vivek Ratnavel Subramanian
2f84bfbfa8
Fix #7100 Upgrade issues for 0.12 release (#7101) 2022-09-01 22:22:42 -07:00
Teddy
704385da0a
Clean up profile and test data when migrating from 0.11 to 0.12 (#7039)
* Added SQL to clean up profile and test data when migrating from 0.11 to 0.12

* Added deletion query for ingestion pipeline mysql

* Added logic to delete profiler ingestion pipeline from DB

* Update v004__create_db_connection_info.sql

added semicolon
2022-08-30 18:43:12 +02:00
Sriharsha Chintalapani
f9a3f9988e
Fix #6936: testPlatforms is required field in testSuite and createTes… (#6944)
* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* fixed glossary test

* Fixed sample data ingestion

* Add filters for testCaseResultSuccess
testCaseResultFailed, testCaseResultAborted

* Fix #6936: fix testcase listing

* Fix #6936: fix java style

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-08-27 13:01:24 +05:30
mohitdeuex
c8ab6fa59b
[Backend][Settings] Settings through UI (#6514) 2022-08-25 06:37:44 -07:00
Milan Bariya
994b769d25
Databricks Connection - User & Password clean (#6866)
* Databricks Connection - User & Password clean

* Fix: Python checkstyle failed

* Fix: update the migration scripts
2022-08-23 14:36:58 +02:00
Sriharsha Chintalapani
821d70eae4
Fix #6782: Separate TableProfile and ColumnProfile api calls (#6783)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6782: Separate TableProfile and ColumnProfile api calls

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix setFields

* Fix tests

* Update pipeline status endpoint

* updated ui side as per new schema for profiler tab

* updated profiler details with new API

* Fix Profiler tests and validation errors (#6827)

* add profilerSample field in TableProfile

* add profilerSample field in TableProfile

* get columnProfile with field profile

* get columnProfile with field profile

* Fixed sample data and python tests

* fixed date range filter change issue

* handled empty profiler case

* Added column level test case and results

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-22 21:31:24 +05:30
Pere Miquel Brull
3f1fc5344a
Remove Athena hostPort (#6825) 2022-08-19 14:11:53 +02:00
Sriharsha Chintalapani
0a2e2d8b18
Fix #6571: Add EntityLink for the testCase to ID columns (#6755)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* added entityLink to python tests

* Fix tests

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-18 00:29:59 -07:00
Parth Panchal
e451a9ca5f
Fixed#5400: Added Kafka Event Publisher (#6697)
* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher
2022-08-11 20:27:15 +05:30
Suresh Srinivas
7825986105
Fixes #6554 Add support for team of type Group (#6588)
* Fixes #6554 Add support for team of type Group

* Fixing test failure
2022-08-05 10:49:42 -07:00
Mayur Singal
5e1b124d57
Fix #6530 & Fix #4975: Oracle Connection Enhancements (#6579)
* Fix #6530 & Fix #4975: Oracle Connection Enhancements

* updated migrations & docs
2022-08-05 10:28:04 +05:30
Sriharsha Chintalapani
1a42428e42
Add time series extention (#6416)
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-08-04 07:22:47 -07:00
Pere Miquel Brull
157fcd8dcf
DeltaLake does not support profiler (#6495) 2022-08-02 08:45:43 +02:00
Vivek Ratnavel Subramanian
62877c2e5d
Fix #6333 Backend: Add API support to create and list announcements (#6446) 2022-07-31 21:02:49 -07:00
Sriharsha Chintalapani
5f8e2a7f4f
Generic Test Case definition to capture different testcases (#5647) 2022-07-28 13:58:41 -07:00
mohitdeuex
160806c013
[Backend][Slack] Slack integration with UI (#6261)
* [Backend][Slack] Slack integration with UI

* [Backend][Slack] Slack integration with UI - removed unused jdbi

* [Backend][Slack] Slack integration with UI - URL fix + init webhook url during initialization

* [Backend][Slack] Slack integration with UI - remove slack

* [Backend][Slack] add Slack generic to exisiting jsons

* [Backend][Slack] PG fix and Java CheckStyle
2022-07-28 23:45:12 +05:30
Pere Miquel Brull
7560635114
Fix #6280 - Bump DeltaLake version, tests and docs (#6307)
Fix #6280 - Bump DeltaLake version, tests and docs (#6307)
2022-07-24 18:49:15 +02:00
Pere Miquel Brull
810e6c7dfc
Fix #5822 - Remove looker env and rename props (#5700)
Fix #5822 - Remove looker env and rename props (#5700)
2022-07-24 16:02:26 +02:00
Onkar Ravgan
1673b1aa30
DynamoDB Topology refactor (#5967)
DynamoDB Topology refactor (#5967)
2022-07-20 10:42:40 +02:00
Suresh Srinivas
4c62237bcb
Fixes #6133 - Add support for Team Hiearchy (#6134)
* Adding different team types for supporting organization team hierarchy

* Add support for team hierarchy

* Create organization during initalization

* Team hierachy test for POST methods

* Add PUT and PATCH support for changing team hiearchy

* Added tests for the team hiearchy feature

* Restored v003 sql files and added v004

* Merge conflicts with main and adopt new JSON based configuration

* Remove configuration Organization name to simplify user MUST do configurations

* Snowflake Make Query tag Optional (#6114)

* Snowflake make query tag optional

* Modified Description

* [Backend][UsageResource] Add PUT to enable updating Usage #6117 :::::: and allow adding space to tagCAtegory and Tags (#6119)

* [Backend][UsageResource] Add PUT to enable updating Usage #6117

* [Backend][UsageResource] Allow space in tags category and primary tags #6121

* [Backend][UsageResource] Update in more genric way to make use of space in resourcePath

* ui changes for allowing space in tag and tag category name

* [Backend][Improvement]add test for put and post

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>

* Adding different team types for supporting organization team hierarchy

* Add support for team hierarchy

* Create organization during initalization

* Team hierachy test for POST methods

* Add PUT and PATCH support for changing team hiearchy

* Added tests for the team hiearchy feature

* Restored v003 sql files and added v004

* Merge conflicts with main and adopt new JSON based configuration

* Remove configuration Organization name to simplify user MUST do configurations

* Python jsonSchema code generation not respecting  defaults

* fixed failing cypress

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: mohitdeuex <105265192+mohitdeuex@users.noreply.github.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2022-07-19 11:10:52 -07:00
Sriharsha Chintalapani
70bcfb34d2
Fix #6016: Add update-ingesiton-bot to update the JWT Token (#6017) 2022-07-12 22:08:44 -07:00
Pere Miquel Brull
7e0a46394a
Update missing FQN (#5966) 2022-07-08 16:27:49 +02:00
Mayur Singal
dd97e849a5
Fix #5905: Bootsrap Migration Fix (#5930)
* Fix #5905: Bootsrap Migration Fix

* Delete from ingestion_pipeline_entity
2022-07-08 06:57:51 +02:00
Parth Panchal
0460b63188
added condition to echo only with debug flag (#5845) 2022-07-05 15:28:59 +05:30
Pere Miquel Brull
97761b97a6
Prepare postgres migrations (#5814) 2022-07-01 10:18:24 +02:00
Sriharsha Chintalapani
d42783fbe0
Fix #5415 - Add migrations (#5767)
* Add migrations

* Migration Extension

* removed repeated code

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2022-06-30 18:40:26 +05:30
Vivek Ratnavel Subramanian
e826c98acf
Fix migration script for migrations to 0.11 (#5773) 2022-06-30 00:38:24 -07:00
mohitdeuex
17bfa5f2d2
Remove log and add debug option (#5750)
* [Backend] Disable logging to console

* [Backend][removeLogs]] Added logback.xml to disable logs for TablesInitializer , added -debug for controlling Console logs

* [Backend][removeLogs]] Printing User.json
2022-06-30 10:31:47 +05:30
Sriharsha Chintalapani
ce49247011
Fix #5590: Add bootstrap command to instantiate ingestion-bot user with JWTToken (#5591) 2022-06-23 18:05:30 +05:30
Vivek Ratnavel Subramanian
296f580304
Fix #4679 Backend: Support assigning a task to user/team and resolving the task (#5501) 2022-06-22 16:17:29 -07:00
Pere Miquel Brull
a14de6eee1
Fix #5367 - MlFlow connector & MLServices (#5446)
Fix #5367 - MlFlow connector & MLServices (#5446)
2022-06-21 14:54:36 +02:00
Parth Panchal
f8f79ebb2d
Fixed#2730: added status filter support for webhooks (#5214)
* Fixed#2730: added status filter support for webhooks

* added status param

* addressing comments

* postgres query change
2022-05-30 15:53:23 +05:30
Suresh Srinivas
9d25596227
Fixes #5122 Add backend and API support for column-level lineage (#5123)
* Fixes #5122 Add backend and API support for column-level lineage

* amend

* Fix test failures
2022-05-24 16:00:12 -07:00
Suresh Srinivas
0ba22c1d2f
Fixes #4977 Add APIs for adding custom fields to an existing entity (#4978) 2022-05-16 15:26:20 -07:00
Suresh Srinivas
6ce528d70a
Fixes #4921 Add support for type system (#4922) 2022-05-14 10:57:19 -07:00
mosiac1
bb131710e3
Add IF EXISTS checks to postgres migration (#4573) 2022-04-29 09:20:52 -07:00
Suresh Srinivas
f0139c3ea5
Make bootstrap sql files for mysql and postgres consistent with each other (#4541)
* Make mysql bootstrap file consistent with postgres

* Adding a java code change to trigger all the precommit checks

* Set test container reuse flag to true

* Make Postgres bootstrap files consistent with MySQL

* Test pagination using random jump of limit between 1-5 instead of 1 to reduce test time

* Starting the test container outside @BeforeAll in a static block for container reuse

* Comment printing out list output in pagination tests

* Move starting the test container to @BeforeAll setup method instead of static initialization

* Remove unnecessary tests in PolicyResourceTest to speed up tests

* Add test container reuse for github workflows

* Add test container reuse for github workflows

* Add test container reuse for github workflows

* Change timeouts for the TestContainer

* Add test container reuse for github workflows

* Add test container reuse for github workflows (#4563)

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2022-04-28 07:30:56 -07:00