263 Commits

Author SHA1 Message Date
07Himank
569539f280
updated elastic search version from 7.16.0 to 7.16.1 (#12203)
* updated elastic search version from 7.16.0 to 7.16.1

* dummy commit

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-06-28 07:05:00 -07:00
Anuj359
73f5859ed9
Docker script update (#12115)
* Create manual.yml

* Fixing Typo

* Removing manual test github action file

* Testing IP runner

* Updating docker script for sandbox db task

* removing the ip test action

* Updating docker script for sandbox db task postgres

* Volume cleaning argument addition

* Volume cleaning argument addition
2023-06-26 15:44:53 +05:30
Anuj359
b2f9f36ac0
Dockerfile Update (#11066)
* Updating Dockerfile with multistage

* Updating reviewed changes

* Docker Development changes

* Docker workflow changes

* Arguments update

* Script path update

* Arguments update

* Resolving the reviewed suggestions
2023-06-26 10:05:23 +00:00
07Himank
9355c0677d
Open search elastic search client task (#11879)
* Added diffrent clients for elasticSearch and openSearch and did the interface implementation

* removed some comments and some changes

* Add Elastic Search Test Container

* upgarded es version

* added test cases for elastic search

* added test case for, when tag is deleted cehck its relationships also get deleted

* changed package of open,elastic to openSearch and elasticSearch

* added support for skip search test case if index is not present of that entity

* Refactor bunch of files

* unnecessary semicolon

* refactor test case indexing

* fix checkstyle

* checkstyle

* added hardcoded configs and changed package name

* fixed no response from hits

* stopped elasticsearch container and some fixes

* closed elasticsearch container

* revert

* closed client after used in test

* style

* edit es config in junit

* checking if using already es version release memory for maven ci

* revert docker file changes

* remove es container for testing the ci

* commented out all test cases for elasticsearch

* fix

* creating elastic search with reusable enabled

* trying to fix ci error

* start container only if it is not running

* revert docker changes and done changes for elastic search container

* changed var name of sort_order for es

* lower the version of es client

* fixes

* all latest version of es including docker

* Use Single sort order

* Fix Keyword issue

* Immediate refresh policy for entity created

* increase sleep time

* disable test

* added reindexing logic in run_local_docker.sh

* enabled test case

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-06-22 16:55:05 +05:30
Pere Miquel Brull
520d5e2657
Add Pipeline Service Status Job to publish to Prometheus (#12046)
* Add Pipeline Service Status Job to publish to Prometheus

* Add Pipeline Service Status Job to publish to Prometheus

* Add config for health check interval
2023-06-22 06:53:56 +02:00
Akash Jain
78eaa06fab
fix(docker): Syntax issues (#11982) 2023-06-14 10:25:37 +05:30
Akash Jain
c96e239b36
fix(docs): Update Airflow Docker environment variables (#11974) 2023-06-13 10:38:32 -07:00
Anuj359
b2ae9d498c
Update docker for sandbox (#11938)
* Stopping the docker before maven build
2023-06-12 10:21:05 +00:00
Snyk bot
f2d202eaf8
fix: docker/development/Dockerfile to reduce vulnerabilities (#11902)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-ALPINE315-OPENSSL-5661569
- https://snyk.io/vuln/SNYK-ALPINE315-OPENSSL-5661569
2023-06-06 22:35:13 +05:30
Sriharsha Chintalapani
bd00914a9b
fix: docker/docker-compose-quickstart/Dockerfile to reduce vulnerabilities (#11899)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-ALPINE315-OPENSSL-5661569
- https://snyk.io/vuln/SNYK-ALPINE315-OPENSSL-5661569

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2023-06-06 22:09:13 +05:30
Louie
956ec58930
Add ES RestHighLevelClient TCP keep alive timeout (#11731)
* Add RestHighLevelClient TCP keep alive timeout

* Merge RestHighLevelClient configs
2023-05-28 10:10:46 -07:00
Akash Jain
92d5bfa94e
fix: versions in main branch (#11478)
* fix: versions in main branch

* Prepare main branch for next release 1.1.0

* prepare main for latest release 1.0.1
2023-05-15 10:42:29 +05:30
Sachin Chaurasiya
2ca039865a
fix: custom logo variable names (#11576) 2023-05-12 12:34:26 -07:00
Mayur Singal
a5e6acbed4
Fix Dockerfile Path (#10840) 2023-03-30 10:39:14 +00:00
Mayur Singal
54b635dd60
1.0.0-alpha version upgrade (#10826) 2023-03-29 23:24:39 +05:30
Nahuel
551df24d35
Fix: pass correct env var to the ingestion docker image (#10598) 2023-03-16 12:28:23 +01:00
Sriharsha Chintalapani
de10cd254b
New docker structure (#10438)
* Adding the different docker-compose file openmetadata and ingestion

* Added two different env files for mysql and postgres

* Updated the docker file path

* Updated the path of docker folder structure

* Fix docker

* Updating the PR with necessary changes required

---------

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-03-06 08:58:53 -08:00
Nahuel
247016307d
Fix#8648: Mask sensitive info from API responses (#10307)
* Mask sensitive info from API responses

* Rename converter classes

* Add missing Java classes from JSON schemas and class converters

* Update test service connection schema

* Update datalakeConnection JSON schema and fix some tests

* Fix AlertsRuleEvaluatorResourceTest and minor error in run_local_docker.sh

* Fix Pipeline and Database service tests

* Minor refactor

* Fix CsvUtilTest

* Fix EventMonitorFactoryTest

* Fix CloudWatchEventMonitorTest

* Update datalake metadata

* Update bigquery metadata

* Fix test connection functionality

* Fix OMeta service api test

* Update gcsValues title and revert changes in GH actions

* Mask sensitive enabled by default for local docker

* Add missing tests

* Address PR comments

* Address PR comments

* fix ui breaks on gcsValues.json

* Address PR comments

* Minor refactor

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-03 18:10:01 +00:00
Sachin Chaurasiya
4ce038557e
feat(ui)#10274: UI - allow to configure OM logo (#10316)
* feat(ui)#10274: UI - allow to configure OM logo

* chore: use logo config to render the brand and login page logo

* fix : cy tests

* doc: update the doc

* chore: remove check for filePath logo location type

* test: add unit test

* - removed FilePath
- Added configs to docker

* address comment

* remove config from test yaml

* update logo aspect ratio

* remove type

* chore: update ui logic as per backend config

* update doc

* address comments

* Revert "address comments"

This reverts commit c4f9da88918343cab08cd5e603cba29bd05b2ec9.

* address comments

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-03-01 14:09:44 +05:30
Ayush Shah
785142d86a
Add policy tags from Bigquery (#10189) 2023-02-20 19:13:45 +00:00
Pere Miquel Brull
1835de8bf2
Group airflowConfiguration into pipelineService config (#10120)
Group airflowConfiguration into pipelineService config (#10120)
2023-02-07 16:30:42 +01:00
Suresh Srinivas
1990ac56d9
10041 Part1 - Refactor and clean up System APIs (#10042)
* 10041 Part1 - Refactor and clean up System APIs

* Update the config resource endpoint url
2023-02-03 16:25:40 -08:00
Nahuel
b9a3c06104
Bump main branch to version 1.0.0 (#10040)
* Bump to version 0.13.2

* Bump mvn projects to 1.0.0-SNAPSHOT

* Bump python projects to 1.0.0.dev0

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-02-02 12:56:14 +01:00
Sriharsha Chintalapani
9004434c18
Fix #9752: docker openmetadata_ingestion should only depend on db (#9753)
* Fix #9752: docker openmetadata_ingestion should only depend on db

* Fix the typo

* Adding service_started for local docker

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
2023-01-18 08:45:15 +01:00
Pere Miquel Brull
e3ae3a8a33
[WIP] - Add Scarf gateway (#9652)
* Update ingestion Readme with scarf

* Test scarf docker

* Update ingestion Readme with scarf

* Test scarf docker

* Add docker domain
2023-01-11 09:33:09 +01:00
Nahuel
2c43ebba6f
Fix#9448: Add ES volumes (#9506)
* Add ES volumes

* Fix run_local_docker script

* Fix error run_local_docker script

* Update Es volumes in docker-compose files
2022-12-23 17:33:30 +01:00
Pere Miquel Brull
8806244348
Removing ES volumes (#9503)
* Fix ES volumes permissions

* Remove ES volumes
2022-12-22 18:02:09 +01:00
Vj-L
f3855a1909
Added the default volumes for ElasticSearch (#9452)
* Added the default volumes for ElasticSearch

* Added docs for ES Volumes

* Updated as per the comments

* Updated as per the comments

* Added the docs for bare-metal upgrade

* Apply suggestions from code review

* Update docker/local-metadata/docker-compose.yml

* Pylint check addressed

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-12-21 18:29:49 +00:00
Pere Miquel Brull
ddf2adf52c
Add port mapping for MySQL (#9436) 2022-12-20 11:13:17 -08:00
Nahuel
a2b34dd0f4
Fix: Update Ingestion docker images and fix python libraries dependencies (#9342)
* Update Ingestion docker images and fix python libraries dependencies

* Install also apache-airlfow-providers-http
2022-12-16 14:46:25 +00:00
Nahuel
e998ecc407
Deprecate configurations for auth provider configuration on Airflow (#8896)
* Deprecate configurations for auth provider configuration on Airflow

* Fix code after merging

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2022-12-15 12:28:46 -08:00
Ayush Shah
231b28fc87
Fix 7278: add test connection (#9196)
* Add Test Connection to Profiler

* remove Code Lint

* Fix AttributeError

* Fix Pytest

* Fix Bigquery Partition tests

* Fix Lint
2022-12-10 17:16:28 +01:00
Vj-L
0a570c8585
Fixed the mounting of volumes (#8963)
* Fixed the mounting of volumes

* Update docker/local-metadata/docker-compose-postgres.yml

Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>

* Update docker/metadata/docker-compose-postgres.yml

Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>

* Added permission step

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
2022-11-23 14:37:53 +05:30
Ayush Shah
b25336a55c
Add Generation of Pydantic Models via script (#8923) 2022-11-22 11:19:22 +05:30
Vj-L
2c055641be
Added host volumes for Database service (#8722)
* Added host volumes for Database service

* Formated the python file

* Done changes according to the comments on PR

* Changes done as per the comments on Draft PR

* Fix PyLint

* Changes done as per the comments on Draft PR

* Updated the path of volume for Database

* Added cleanup state in workflows

* Added cleanup state in workflows

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-11-18 14:05:33 +01:00
Ayush Shah
2d7d89754c
Remove Click and Add ArgParse (#8182) 2022-10-31 18:12:26 +05:30
Rahul Surwade
a286a0fcbc
Removed unnecessary docker-compose files (#8085)
* Delete docker-compose-dev-postgres.yml

* Delete docker-compose-dev.yml

* Update openmetadata-server.md

* Update openmetadata-server.md

* Update openmetadata-server.md
2022-10-31 12:50:12 +05:30
Pere Miquel Brull
339f123f92
Fix #7477 - Paginate Airflow logs (#8145)
* Paginate last dag logs

* clean logic

* Prep pagination

* Prep pagination

* Format

* Update docker/local-metadata/docker-compose.yml

Co-authored-by: Nahuel <nahuel@getcollate.io>

* UI : Add pagination support for airflow logs

* Addressing review comments

Co-authored-by: Nahuel <nahuel@getcollate.io>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-10-21 17:32:56 +05:30
Vj-L
e1b4229074
Fix/env variable (#8199)
* Updated env variable for heap

* Updated env variable for heap

* Updated the env variables

* Updated the env variables

* Updated the env variables

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
2022-10-19 11:00:11 +05:30
Pere Miquel Brull
19b2158d56
Use postgres 14 (#8207) 2022-10-18 18:29:25 +05:30
Sam Firke
58b86a24bc
Remove inconsistent space in docker-compose files (#8138)
* remove extra space for consistency
2022-10-13 15:33:22 -07:00
Sriharsha Chintalapani
65ee573a7d
Fix HEAP OPTS param (#8094) 2022-10-11 16:30:09 -07:00
Vj-L
0926c754aa
Updated env variable for heap (#8067)
* Updated env variable for heap

* Updated env variable for heap

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
2022-10-11 19:29:22 +05:30
Pere Miquel Brull
339abc5bf3
Prepare Ingestion Base Docker image (#8065)
Prepare Ingestion Base Docker image (#8065)
2022-10-11 07:50:49 +02:00
Sriharsha Chintalapani
801012d2a6
Add remote debug port (#8045)
* Add remote debug port

* Add remote debug port

* Add remote debug port

* Add remote debug port
2022-10-11 11:13:15 +05:30
kmalki
69ddc7081c
Add environment variables in docker compose for elasticsearch (#8026) 2022-10-08 23:51:59 +05:30
Akash Jain
1e31245bbe
fix: Remove Lineage Backend Configs (#7836) 2022-10-01 11:23:24 +05:30
Akash Jain
c4b0bb0538
fix path for airflow volumes (#7827) 2022-09-30 18:20:38 +05:30
Vj-L
489d506e3a
Removed the volume for Database (#7798)
* Removed the volume for Database

* Removed the volume for Database

* Added named volume

* Added named volume

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
2022-09-29 21:44:12 +05:30
Vj-L
6cfd16925a
Added default volumes (#7741)
* Added default volumes

* Ran make py_format for formatting

* Added sample data in mount directories

* Updated as per the comments

* Done changes for --ingest_sample_data flag

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
2022-09-29 14:53:55 +05:30