156 Commits

Author SHA1 Message Date
Suresh Srinivas
d47f5738cb
Add support for system provided Tags, Glossaries, Policies, and Roles #8484 (#8508) 2022-11-02 14:56:31 -07:00
Tushar Mittal
2a65df5f36
feat: add Kinesis connector (#8452)
Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
2022-11-02 16:12:45 +05:30
Mayur Singal
fa68a1f18f
Fix #6754: Added User Details for Usage (#8449) 2022-11-02 14:07:41 +05:30
Mohit Yadav
6efe6df96b
[LDAP] Added Ldap-Authentication (#8256)
* [LDAP] Add Ldap-Authentication

* Refactor Basic to Service Layer

* added read login config

* pom updates

* review comments

* review comments

* move authenticator after authorizer

* merge conflicts resolve

* add init method , for initializing authenticator

* sonar fix
2022-10-27 22:56:20 +05:30
Teddy
83b048c07c
Fixes #8206 -- Implement data insight refinement for entity report data (#8351)
* added data insight module to refine and ingest data

* Added json schema for data insight workflow

* Added mixin for data insight

* Added tests for data insight workflow and methods

* Fixed python style

* Fixed python linting

* Fixed unit test

* Added data insight worflow + workflow logging

* Fix python style

* Renamed analytic -> data_insight + added test to make + fixed style

* Fixed java style

* Fixed code smells + python style

* Ignore ES linting for now

* Fixed comments from review

* fix python formatting

* Fixe PipelineType attribute for data insight
2022-10-26 11:18:08 +02:00
Suresh Srinivas
f12b09687d
Fixes #8326 Add roles to system bots during initialization and use po… (#8327)
* Fixes #8326 Add roles to system bots during initialization and use policy based authorization for bots

* Remove var usage and instead use specific types for variables

* Fix bot can't create entities
2022-10-24 17:53:11 -07:00
Suresh Srinivas
82b81572ac
Update javadoc version (#8329) 2022-10-23 22:07:58 -07:00
Suresh Srinivas
0d75eb9ff2
Fixes #8313 Deprecate botPrincipals from open metadata configuration (#8314) 2022-10-21 20:49:41 -07:00
Suresh Srinivas
229b56e7b3
Fixes #8236 - Initialize bots from JSON data files (#8304) 2022-10-21 08:38:52 -07:00
Teddy
342e4a24da
Fixes #8154 - Add collect endpoint for web analytic events (#8193)
* Initial implementation of report endpoint

* Modified model files

* Added migration

* Added tests for analytics/report endpoint

* Finalized implementation of the reports endpoint

* Fixed maven test failure

* Fixed java style

* Added policy for reportDefinition

* Fixed minor typos in code

* Added database migration for web analytic event type

* Added logic for collect endpoint

* Added JSON Schema for collect endpoint

* Added index on table columns + added resource description for web event entities

* Fixed policy tests

* Updated add web analytic function to return the body of the request + updated test

* Added referer URL to PageView event

* Changed collection name to align with API endpoint

* updated initSeed method to initializeEntity

* cleaned up PR by removing unused filed from branch

* Fixed requested changes from review

* Clean up names

* Fixed comma in SQL statement
2022-10-20 16:38:54 +02:00
Teddy
816111a671
Fixes #8121 -- Report endpoint for data insight (#8122)
* Initial implementation of report endpoint

* Modified model files

* Added migration

* Added tests for analytics/report endpoint

* Finalized implementation of the reports endpoint

* Fixed maven test failure

* Fixed java style

* Added policy for reportDefinition

* Fixed minor typos in code

* Renamed classes from AnalyticReport to ReportDefinition

* Java style formatting

* Update add report results to return request body

* Refactor report endpoint to simplify it and remove dependency on entity resource.

* Removed import *

* Fixed java style

* fixed typos in code

* Fix import path
2022-10-20 16:37:09 +02:00
Pere Miquel Brull
ee658bfb1f
Revert "Fixes #8236 - Initialize bots from JSON data files (#8237)" (#8281)
This reverts commit 9facb03cafce52735ef2b615d75205486985fc55.
2022-10-20 11:08:47 +02:00
Suresh Srinivas
9facb03caf
Fixes #8236 - Initialize bots from JSON data files (#8237) 2022-10-19 16:21:57 -07:00
Onkar Ravgan
d9b50b5be9
Added project id to DBT cloud config (#8062)
* Added project id to DBT cloud config

* added field in cloud config source Dbt config

* fix unit test issue

* Fixed pylint

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-10-19 18:59:33 +05:30
NiharDoshi99
e086fb7c25
Add Domo Database Connector (#8221)
* added domo database connector

* added domo database connector

* changes based on comment
2022-10-19 12:50:58 +02:00
Milan Bariya
1565aa7733
FIX: Trino&Presto catalogs as databases (#8189)
* FIX: Trino&Presto catalogs as databases

* FIX: Trino&Presto catalogs as databases

* Change based on comments
2022-10-18 20:00:17 +05:30
NiharDoshi99
84a89d7181
Added DOMO Pipeline (#8133) 2022-10-18 17:52:28 +05:30
Milan Bariya
e7a3770240
Add: databricks usage and lineage added (#7386) 2022-10-14 07:06:28 +05:30
Mohit Yadav
71fcb17e59
Added Context Info in ES reindexing (#8123)
* Added Context Info in ES reindexing

* Fix File Formatting

* Update openmetadata-spec/src/main/resources/json/schema/settings/eventPublisherJob.json

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>

* revert python file

Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-10-13 18:32:06 +05:30
Mohit Yadav
cba9240fe9
[Email] Added explicit senderMail in configuration (#8126)
* [Email] Added explicit senderMail in configuration

* Fix failing tests

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-10-13 18:24:32 +05:30
Nahuel
48905a7f6e
Fix#7532: Allow overriding service connection config when ingesting from CLI (#8091)
* Allow overriding service connection config when ingesting from CLI

* Rename field in JSON schema

* Address comment
2022-10-13 12:34:03 +02:00
NiharDoshi99
0e44f7d35d
Fixed: Intergrated DOMO Connector APIs (#7989) 2022-10-13 10:55:48 +05:30
Ayush Shah
aa9ef323e2
FIx Nonetype Bigquery Taxonomy (#8110) 2022-10-12 19:54:13 +05:30
Ayush Shah
58765379a1
Fix Policy Tags from diff project ids (#8004) 2022-10-12 15:17:21 +05:30
Mohit Yadav
e7ec822679
[ESReindexing] Fixed slowness in ES write + added controlling params (#7974)
* [ESReindexing] Fixed slowness in ES write + added controlling params

* migrate-all failure fix during script run

* Update createEventPublisherJob.json
2022-10-06 15:58:44 +05:30
Suresh Srinivas
7499277fcb
Update dependencies for release 0.13 (#7913)
* Update openmetadata-client module dependencies

* Update common module dependencies

* Update openmetadata-spec module dependencies

* Update openmetadata-ui module dependencies

* Update openmetadata-service module dependencies

* Upgrade remaining dependencies
2022-10-04 09:30:05 -07:00
Mayur Singal
ca51fa0245
Fix #7801: Fixed Glue Pipeline (#7832)
Co-authored-by: Apple <apple@Apples-MacBook-Pro.local>
2022-10-02 16:43:35 -07:00
Mohit Yadav
5b3b9ec811
update Batch job status + missing place stream status update (#7842)
* [ES Reindex] Added ES Reindexing from API

* [ES Reindex] checkstyle

* [ES Reindex] return complete record for status info

* fixes

* added limit param to get no of records

* Review Comments + Stream mode

* Review Comments

* Fix for failing tests

* Global ES monitoring with Stream + batch mode to allow Reindexing

* Updated Status Message + Status update on ESIndexDefinition Create and Delete

Co-authored-by: mohitdeuex <105265192+mohitdeuex@users.noreply.github.com>
2022-10-01 17:58:19 +05:30
Nahuel
593ca3a4a0
Fix bot creation and edition logic (#7796)
* Fix bot creation and edition logic

* Fix minor error creating user

* Fix failing test

* Minor fixes

* Add missing tests for new flows

* Fix put_failIfBotHasARelationshipToAnotherUser test

* Changes after manual testing

* Move where auth_provider is retrieved in the secret manager
2022-10-01 11:25:56 +02:00
Mohit Yadav
d04ce5341d
[Basic auth fixes] Added check to prevent brute-force attack (#7803)
* [Basic Auth] Updated messages , and added changepasswor

* checkstyleFix

* Added ChangePassword feature in user page

* unpushed file

* changes as per comments

* unpushed file

* close the modal on save

* Added Admin Create User Password + Logout

* revert test

* revert yaml

* Fix failing test

* Remove token on logout

* fix for test

* added api for random pwd

* update error messages as per UI

* Added password generator in user creation

* Fix issue with CP

* Fix issue with CP

* [Basic Auth] Added Brute forcing check , blocks user login for 10 mins + send update on mail

* Review Comments updated

Co-authored-by: mohitdeuex <105265192+mohitdeuex@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-10-01 02:03:43 +05:30
mohitdeuex
b934ed4870
Stream and Batch Global ES Error Logging added (#7780)
* [ES Reindex] Added ES Reindexing from API

* [ES Reindex] checkstyle

* [ES Reindex] return complete record for status info

* fixes

* added limit param to get no of records

* Review Comments + Stream mode

* Review Comments

* Fix for failing tests

* Global ES monitoring with Stream + batch mode to allow Reindexing
2022-09-28 22:55:00 +05:30
Teddy
962866a30e
Fixes #7280 - Implements support for BYTEA DType (#7749)
* Added support for postgres bytea data type

* Added BYTEA support for profiler and converts bytes to hex on results read
2022-09-28 13:28:35 +02:00
mohitdeuex
859317494b
Basic auth fixes (#7747)
* [Basic Auth] Updated messages , and added changepasswor

* checkstyleFix

* Added ChangePassword feature in user page

* unpushed file

* changes as per comments

* unpushed file

* close the modal on save

* Added Admin Create User Password + Logout

* revert test

* revert yaml

* Fix failing test

* Remove token on logout

* fix for test

* added api for random pwd

* update error messages as per UI

* Added password generator in user creation

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-09-28 16:57:30 +05:30
Mayur Singal
24daae88e9
Add flag use fqn for filter patterns (#7755) 2022-09-28 16:41:22 +05:30
mohitdeuex
8d33da7f3d
[ES Reindex] Added ES Reindexing from API (#7717)
* [ES Reindex] Added ES Reindexing from API

* [ES Reindex] checkstyle

* [ES Reindex] return complete record for status info

* fixes

* added limit param to get no of records

* Review Comments + Stream mode

* Review Comments

* Fix for failing tests
2022-09-27 21:34:07 +05:30
Onkar Ravgan
8afbbb8642
Ingest DBT tags (#7726)
* added tags

* Added dbt tags ingestion

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-09-27 11:12:53 +05:30
Vivek Ratnavel Subramanian
e2daf73fbd
UI: TeamsSelectable component should use REST API to build teams hierarchy (#7711) 2022-09-26 16:33:48 -07:00
Nahuel
e49ad000ee
Fix#6615: Configure a bot account and provider service or jwtAuthToken from the UI (#7637)
* Update JSON schemas

* Update create and update endpoint and remove /generateToken endpoint

* Add AuthMechanism and Token Expiry Field on the Add bot page

* Remove unwanted tests

* Add logic to show authMechanism based on auth provider

* Add authConfig to bots details page

* Backend implementation with secret manager

* Fix invalid date issue

* Add support for update bot token details

* Add logic for SSO auth

* Add fields for google sso auth client

* Add Edit Option.

* Updated ingestion CLI and secret manager issues

* Updated tests with errors

* Removed unused method

* Removed test files

* Fix broken JSON schema

* Fix typescript error

* Pass the bot permissions from the bot details page

* Minor errors related to the secret manager

* Restore Airflow configuration for retrocompability

* Add support for all SsoServiceType client config

* Fix Unit test

* Update tests

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-09-26 18:26:24 +05:30
Onkar Ravgan
35efd49256
Added control for DBT descriptions (#7653)
* Added control for DBT descriptions

* Fixed tests

* Added UI changes

* fixed maven ci tests

* Java formatting changes

* ui review fixes

* Fixed pytests

* Fixed python integration tests

* fixed airflow tests

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-09-26 16:19:47 +05:30
NiharDoshi99
8adcf2ee3d
Fixed: Dagster to work with its API (#7579) 2022-09-23 17:46:40 +05:30
Chirag Madlani
93ffbbbebf
feat(ui): login via email and password - Basic auth (#7558)
* feat(ui): login via email and password - Basic auth

* create admin users

* update basic auth

* create user on bootstrap

* update basic auth flow

* fix lint

* update basic auth ui forms

* update error messages

* fix cypress tests

* Added private key /public key in conf
Default invitation for invited users on bootstrapping user with invitation mail
Added self/signup flags , handled login scenarios around it
On migration persisting previous user auth mechanism also handled

* Missing fixes

* Move self signup to config

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-09-23 16:05:54 +05:30
Teddy
90abd73394
Fixes #7105 -- Add support for dataType filter for TestDefinition request (#7640)
* Updated json schema to support dataType filter

* Added migration for test definition support of data type filtering

* Added backend logic to support filter on data type for testdefinitions

* Added migration for:
- supported test types
- renaming of certain tests from starting uppercase to lowercase

* removed openmetadata_db from migration

* added datatype filter in column test

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-09-23 10:31:00 +02:00
Suresh Srinivas
569e8b9085
Fixes #7322 #7145 - Add ViewBasic operation and add access control to get, list, getVersion, listVersions (#7644) 2022-09-21 21:06:35 -07:00
Abhishek Pandey
d40be7a9aa
postgres-sql-query-updated (#7421) 2022-09-21 07:04:06 +05:30
Pere Miquel Brull
1578fa7f1b
Fix #7121 - Support Spark metastore DB connection (#7520)
* Fix #7121 - Support Spark metastore DB connection

* appname

* Update docs

* test validation

* Address PR comments

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-09-20 16:47:57 +02:00
Nahuel
67ae82286f
Fix#7424: Create Bots on default authorizer init (#7559)
* Create Bots on default authorizer init

* Apply suggestions from code review

Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>

Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
2022-09-20 11:28:39 +02:00
Suresh Srinivas
c4eb9260be
Change camelCase mlModel to lowercase in package name (#7565) 2022-09-19 18:54:02 -07:00
Nahuel
2a6c6134f4
Fix#7272: Improve logging when initializing workflow from CLI (#7522)
* Improve logging when initializing workflow from CLI

* Fix broken tests
2022-09-19 08:00:00 -07:00
mohitdeuex
4b594d0d6e
[BasicAuth] Added Basic Authentication (#7390)
* Fixed issue with testCases not posted in slack

* Return empty string in case of faling to build url

* [BasicAuth] Added Basic Authentication APIs

* [BasicAuth] Checkstyle

* [BasicAuth] Test yaml updated

* [BasicAuth] Update review comments

* [BasicAuth] Refresh logic added

* [BasicAuth] Add sql updates

* [BasicAuth] Maintain dependency version in main pom.xml

* [BasicAuth] Updated as per new module

* [BasicAuth] Fixes for refresh and missing edge scenario handled

* [BasicAuth] remove quotes

* [BasicAuth] Fix merge conflicts
2022-09-19 11:38:45 +05:30
Suresh Srinivas
78bd0c40c6
Change camel case package name to lower case (#7534) 2022-09-18 20:09:08 -07:00