145 Commits

Author SHA1 Message Date
Louie
956ec58930
Add ES RestHighLevelClient TCP keep alive timeout (#11731)
* Add RestHighLevelClient TCP keep alive timeout

* Merge RestHighLevelClient configs
2023-05-28 10:10:46 -07:00
Sriharsha Chintalapani
160984baf1
Fix #11673: Service connection details will be viewable based on view permissions and by default masked for users and unmasked for bots (#11738) 2023-05-24 08:54:14 +02:00
Pere Miquel Brull
50ad38ea0f
Fix #11548 - Secrets Managers comms with OMeta (#11602)
* Remove secretsManagerCredentials from backend

* Remove secretsManagerCredentials from backend

* Add secrets manager loader

* Load SM in the ometa client

* Fix tests

* Fix tests

* Fix Lint

* Mock AWS region

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-19 09:43:11 +02:00
Pere Miquel Brull
80af2b20bb
Centralize SSL Schemas & Add SSL for Airflow REST (#11363) 2023-05-08 21:05:10 +02:00
Pere Miquel Brull
cdd473c3b7
Add ingestionIpInfoEnabled configuration (#10870)
* Add ingestionIpInfoEnabled configuration

* Format

* hide ip address if api response if not 200

* removing dependancy on ip-modal and checking default state

* updated status code for ip to 204 to check default state

* fixed failing cypress

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-04-06 16:27:07 +02:00
Sriharsha Chintalapani
eb92ea9069
Fix #10808: Add configuration to enable HSTS, X-Frame-Options, X-Cont… (#10809)
* Fix #10808: Add configuration to enable HSTS, X-Frame-Options, X-Content-Type-Options etc..

* Fix #10808: Add configuration to enable HSTS, X-Frame-Options, X-Content-Type-Options etc..
2023-04-05 16:50:12 +02:00
07Himank
4eb1ec81e1
Update query task for activity feed (#10895)
* solved change event query update for activity feed

* working on url config

* testing

* fix bug in slack alerts

* fixing alerts url

* fixed space issue url issue and edited the testcases for changeeventparser

* fixed email markers bug

* default value added

* removed my gmail config

* fix

* addressing comments

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2023-04-04 17:02:08 +05:30
Parth Panchal
8f424e1787
Fixed#10565: Make Prometheus Mappings Configurable (#10671)
* Fixed#10565: Make Prometheus Mappings Configurable

* Fixed#10565: Make Prometheus Mappings Configurable

* Fixed#10565: Make Prometheus Mappings Configurable

* Addressing comments and fixed nullpointer in tests

* Fixed#10565: Make Prometheus Mappings Configurable

* Addressing comments

* Fixing tests
2023-03-28 12:51:52 +05:30
Mohit Yadav
11032a2d48
SAML feature (#10669)
* Added Saml Feature

* add localization

* fix tests

* added collate copyright

* remove thread.getType

* fixed keystore issues and updated review comments

* remove private key getter

* update test saml config

* update test saml config

* Update openmetadata-service/src/main/java/org/openmetadata/service/security/saml/OMMicrometerHttpFilter.java

Co-authored-by: Nahuel <nahuel@getcollate.io>

* review comments

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-24 21:56:49 +05:30
Nahuel
247016307d
Fix#8648: Mask sensitive info from API responses (#10307)
* Mask sensitive info from API responses

* Rename converter classes

* Add missing Java classes from JSON schemas and class converters

* Update test service connection schema

* Update datalakeConnection JSON schema and fix some tests

* Fix AlertsRuleEvaluatorResourceTest and minor error in run_local_docker.sh

* Fix Pipeline and Database service tests

* Minor refactor

* Fix CsvUtilTest

* Fix EventMonitorFactoryTest

* Fix CloudWatchEventMonitorTest

* Update datalake metadata

* Update bigquery metadata

* Fix test connection functionality

* Fix OMeta service api test

* Update gcsValues title and revert changes in GH actions

* Mask sensitive enabled by default for local docker

* Add missing tests

* Address PR comments

* Address PR comments

* fix ui breaks on gcsValues.json

* Address PR comments

* Minor refactor

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-03 18:10:01 +00:00
Sachin Chaurasiya
4ce038557e
feat(ui)#10274: UI - allow to configure OM logo (#10316)
* feat(ui)#10274: UI - allow to configure OM logo

* chore: use logo config to render the brand and login page logo

* fix : cy tests

* doc: update the doc

* chore: remove check for filePath logo location type

* test: add unit test

* - removed FilePath
- Added configs to docker

* address comment

* remove config from test yaml

* update logo aspect ratio

* remove type

* chore: update ui logic as per backend config

* update doc

* address comments

* Revert "address comments"

This reverts commit c4f9da88918343cab08cd5e603cba29bd05b2ec9.

* address comments

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-03-01 14:09:44 +05:30
Mohit Yadav
960ae2918e
- Config Changes for Logo and Jwt Expiry (#10287)
* - Changes for Logo
- Make Jwt Token Expiry Configurable

* change config time

* fix tests

* review comments

* fix failing python

* moved login to application config as well

* fix login Config

* Added Nav Bar and Login Page Config for Logos

* Removed UI Api calls

* fix: unit test

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-02-23 22:57:57 +05:30
Parth Panchal
9bd22369d7
Fixed#10211: Enable LDAP configuration to be configured via environment variable (#10252)
* Fixed#10211: Enable LDAP configuration to be configured via environment variable

* Fixed#10211: Enable LDAP configuration to be configured via environment variable
2023-02-22 12:53:24 +05:30
Pere Miquel Brull
1835de8bf2
Group airflowConfiguration into pipelineService config (#10120)
Group airflowConfiguration into pipelineService config (#10120)
2023-02-07 16:30:42 +01:00
Suresh Srinivas
1990ac56d9
10041 Part1 - Refactor and clean up System APIs (#10042)
* 10041 Part1 - Refactor and clean up System APIs

* Update the config resource endpoint url
2023-02-03 16:25:40 -08:00
Teddy
1455d5eb29
Update webanalytics prometheus metrics (#9939)
* feature: added web analytic events handler

* feature: exclude web analytic events from audit logger

* empty commit to re-run CI
2023-01-27 15:51:53 +01:00
Sriharsha Chintalapani
3c9d39e9af
Fix #9158: OpenSearch/ElasticSearch - Language based mappings (#9163)
* Add language for Search mappings

* Fix #9158: OpenSearch/ElasticSearch - Language based mappings

* Add analyzer to search in Japanese

* Adding JP es analyzer for all the indexes

* Fix review comments

* rebase with main

Co-authored-by: Megumi AIKAWA <megumi.aikawa@gmail.com>
2023-01-09 21:42:01 -08:00
Nahuel
e998ecc407
Deprecate configurations for auth provider configuration on Airflow (#8896)
* Deprecate configurations for auth provider configuration on Airflow

* Fix code after merging

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2022-12-15 12:28:46 -08:00
Pere Miquel Brull
ba9109d318
Fix #9044 - Add prometheus event monitor (#9176)
* Add prometheus event monitor

* Format

* Format
2022-12-09 11:51:39 +01:00
Talal
e73777cbe7
Fixes #9167 - Default value for SLACK_CHAT_SLACK_URL (#9174)
* Fixes #9167 - Added a default value for SLACK_CHAT_SLACK_URL

* Fixes #9167 - Added a default value for SLACK_CHAT_SLACK_URL
2022-12-07 17:17:30 +05:30
Pere Miquel Brull
dda8f8054e
Fix #8571 - Log IngestionPipeline status change to CloudWatch (#9013)
* Prep monitoring skeleton

* Fix naming

* Push Ingestion Pipeline status to cloudwatch

* Review comments and tests

* import

* Remove singleton object

* Format
2022-11-28 17:45:18 +01:00
Suresh Srinivas
0d75eb9ff2
Fixes #8313 Deprecate botPrincipals from open metadata configuration (#8314) 2022-10-21 20:49:41 -07:00
Mohit Yadav
cba9240fe9
[Email] Added explicit senderMail in configuration (#8126)
* [Email] Added explicit senderMail in configuration

* Fix failing tests

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-10-13 18:24:32 +05:30
Sriharsha Chintalapani
801012d2a6
Add remote debug port (#8045)
* Add remote debug port

* Add remote debug port

* Add remote debug port

* Add remote debug port
2022-10-11 11:13:15 +05:30
Suresh Srinivas
de7a616788
Add log layout for tests and also printing ISO8601 timestamp (#8071) 2022-10-10 18:42:59 -07:00
Mohit Yadav
d04ce5341d
[Basic auth fixes] Added check to prevent brute-force attack (#7803)
* [Basic Auth] Updated messages , and added changepasswor

* checkstyleFix

* Added ChangePassword feature in user page

* unpushed file

* changes as per comments

* unpushed file

* close the modal on save

* Added Admin Create User Password + Logout

* revert test

* revert yaml

* Fix failing test

* Remove token on logout

* fix for test

* added api for random pwd

* update error messages as per UI

* Added password generator in user creation

* Fix issue with CP

* Fix issue with CP

* [Basic Auth] Added Brute forcing check , blocks user login for 10 mins + send update on mail

* Review Comments updated

Co-authored-by: mohitdeuex <105265192+mohitdeuex@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-10-01 02:03:43 +05:30
Sriharsha Chintalapani
a946c3fc5a
Fix #7782: Add console appender to OpenMetadata logs (#7783)
* Fix #7782: Add console appender to OpenMetadata logs

* Add format

Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
2022-09-28 18:15:57 -07:00
Chirag Madlani
93ffbbbebf
feat(ui): login via email and password - Basic auth (#7558)
* feat(ui): login via email and password - Basic auth

* create admin users

* update basic auth

* create user on bootstrap

* update basic auth flow

* fix lint

* update basic auth ui forms

* update error messages

* fix cypress tests

* Added private key /public key in conf
Default invitation for invited users on bootstrapping user with invitation mail
Added self/signup flags , handled login scenarios around it
On migration persisting previous user auth mechanism also handled

* Missing fixes

* Move self signup to config

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-09-23 16:05:54 +05:30
mohitdeuex
4b594d0d6e
[BasicAuth] Added Basic Authentication (#7390)
* Fixed issue with testCases not posted in slack

* Return empty string in case of faling to build url

* [BasicAuth] Added Basic Authentication APIs

* [BasicAuth] Checkstyle

* [BasicAuth] Test yaml updated

* [BasicAuth] Update review comments

* [BasicAuth] Refresh logic added

* [BasicAuth] Add sql updates

* [BasicAuth] Maintain dependency version in main pom.xml

* [BasicAuth] Updated as per new module

* [BasicAuth] Fixes for refresh and missing edge scenario handled

* [BasicAuth] remove quotes

* [BasicAuth] Fix merge conflicts
2022-09-19 11:38:45 +05:30
Pere Miquel Brull
6700f169ab
Fix #7165 - Handle SSL certificates on ometa client (#7502)
Fix #7165 - Handle SSL certificates on ometa client (#7502)
2022-09-16 09:39:27 +02:00
Sriharsha Chintalapani
656b50dd3a
Fix #7469: Refactor OpenMetadata code modules (#7474) 2022-09-14 23:14:02 -07:00
Nahuel
b829a2cbf3
Fix: Improvements on secret manager implementation (#7282)
* Change local secret manager by noop

* Update openmetadata-secure-test.yaml
2022-09-07 09:18:59 +02:00
Pere Miquel Brull
7d2527bbeb
Fix #6961 - Fallback to another IP service (#7019)
Fix #6961  - Fallback to another IP service (#7019)
2022-09-01 06:39:12 +02:00
Nahuel
0762e39172
Minor fixes related to Secrets Manager implementation (#7046)
* Minor fixes related to Secrets Manager implementation

* Fix failing test
2022-08-31 14:14:55 +02:00
Nahuel
db41be4c4a
Fix#6929: docker-compose services should wait until OpenMetadata DB is created (#7025)
* Added the docker compose for postgres

* Added changes as per the comments on PR

* Airflow Connectivity to Postgres

* Changes are done as per the comments

* Replocate changes to local dockerfile

* Fixed Airflow Dockerfile

* docker-compose services should wait until OpenMetadata DB is created

* remove unnecesary wait for db service

* Add dependencies to airflow image

* Add missing dependencies to airflow Dockerfile

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2022-08-30 23:55:46 +02:00
mohitdeuex
c8ab6fa59b
[Backend][Settings] Settings through UI (#6514) 2022-08-25 06:37:44 -07:00
Vivek Ratnavel Subramanian
f23357ba68
Update slack chat to redirect to a URL (#6909) 2022-08-25 15:25:11 +05:30
Nahuel
65517beb62
Fix#5919: Implementation of migration process when a Secrets Manager is configured (#6584)
* Implementation of migration process when a Secrets Manager is configured

* Update conf/openmetadata.yaml

* Address PR comments

* Minor change in SQL statment in CollectionDAO

* Address Sonar security hotspot issue
2022-08-09 16:11:57 +02:00
Nahuel
cf2cb6d531
Fix#6517: Add clusterName property to the application config yaml (#6610)
* Add cluster name in the app configuration and start using it to create secrets id

* Update secret manager client in openmetadata for using default auth provider

* Add missing property in test config file
2022-08-09 09:00:43 +02:00
Vivek Ratnavel Subramanian
4f61169df6
feat(ui): Add support for slack chat (#6015)
* feat(ui): Add support for slack chat

* updated icons and styling

* Add to all the pages

* Fix UT failure
2022-07-14 19:31:27 +05:30
Nahuel
65260263fc
Fix #5915: Implementation of AWS Secret Manager (#5925)
* Implement Secrets Manager abstraction + refactor service repositories

* Rename secrets manager factory method for consistency

* Fix typo in openmetadata conf yaml file

* Bump version

* Fix SonarCloud code smells

* Add secrets manager provider as an Enum + send it in the server configuration in the ingestion pipeline

* Apply suggestions from code review

Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>

* Change secret manager implementations to be singleton

Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
2022-07-13 20:49:27 +02:00
Pere Miquel Brull
048801067a
Add fernet back (#6063) 2022-07-13 14:03:46 +02:00
Akash Jain
fa5105191b
Fix/issue-5883 (#5958) 2022-07-12 19:55:38 +05:30
Sriharsha Chintalapani
8528c51b9c
Fix #5516: Refactor elasticsearch indexes to be same as entity specifications (#5573)
* Fix #5516: Refactor elasticsearch indexes to be same as entity specifications

Co-authored-by: darth-coder00 <aashit@getcollate.io>
Co-authored-by: Sachin-chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-06-27 01:19:27 -07:00
Sriharsha Chintalapani
e8c5ddc342
Fix #5235: AIRFLOW_AUTH_PROVIDER value to be custom-oidc instead of customOidc (#5537) 2022-06-24 20:43:20 -07:00
Sriharsha Chintalapani
ad0c6dad88
Fix #5613: Add fernet encryption and remove connection details for non admin users (#5614)
* Fix #5613: Add fernet encryption and remove connection details for non-admin users
2022-06-24 16:44:37 -07:00
mohitdeuex
be18feb928
[Backend] Websocket implementation added (#5440)
* [Backend] Websocket implementation added

* [Backend] Websocket implementation added [google formatter fix]

* [backend] Updated Reiew Comments

* [backend] Updated CheckStyle
2022-06-15 10:14:47 +05:30
Vivek Ratnavel Subramanian
d65c97c4e8
Fix #5382 Backend: Update timestamp of thread for any reaction (#5443) 2022-06-13 23:12:20 -07:00
Sriharsha Chintalapani
bd98919d64
Fix openmetadata conf (#5403)
* Fix openmetadata conf
2022-06-09 22:08:01 -07:00
Sriharsha Chintalapani
1923591a94
Fix #5275: OpenMetadata database driver should be configurable via Env (#5402) 2022-06-09 21:02:32 -07:00