66 Commits

Author SHA1 Message Date
Mohit Yadav
3a01ad7da5
[Fix-20125] OIDC: Allow max_age to be optional (#20721)
* Make Max Age Optional

* spotless fix
2025-04-09 15:09:57 +05:30
Mohit Yadav
c28f3274d1
Adds new param to docker files (#20338) 2025-03-19 18:13:22 +05:30
Pere Miquel Brull
69c9102da1
MINOR - Bump Ingestion versions (#19836)
* MINOR - Bump Ingestion versions

* MINOR - Bump Ingestion versions

* fix

* fix db_scheme for airflow +2.9.1

* fix
2025-02-18 07:56:46 +01:00
Chirag Madlani
a43835df32
Revert "fixes #18820: updated docker compose files (#18821)" (#19297)
This reverts commit 69dd8b99f991da78344eeb35b31cdee9a6ca8521.
2025-01-09 15:34:07 +05:30
tarunpandey23
69dd8b99f9
fixes #18820: updated docker compose files (#18821) 2025-01-09 10:50:00 +05:30
Ethan
e708a3242e
feat: update version (#18259) 2024-10-17 16:18:37 -07:00
Pere Miquel Brull
6a1cd0ef8b
GEN-1493 - Fix paginate_es in opensearch (#17858)
* GEN-1493 - Fix opensearch pagination

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch
2024-09-17 14:21:10 +02:00
Prajwal214
1c90eaaf3d
Docs: Updated Domain in Docker Compose & Docs (#17603)
* Minor: Updating Domain in Docker Compose & Docs

* replace openmetadata to open-metadata

---------

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-09-09 11:47:09 +05:30
Mohit Yadav
05baba9dc1
Change Default Value of Batch Size and Payload Size (#17391) 2024-08-12 11:18:18 +05:30
Mohit Yadav
aa5b9b2985
Increase Payload Size to 100MIB for ELastic Search (#17366)
* Increase Payload Size to 100MIB

* Increase batchSize Size to 500

* Add Payload Size as option in openmetadataoperations , and searchIndexJob

* //Comment

* Add Logs
2024-08-11 21:58:55 +05:30
sonika-shah
8d5661a7bf
Support Elasticsearch cluster alias for E2E tests  (#17117)
* Fix backend tests and have index alias with clusterAlias appended

* Fix backend tests and have index alias with clusterAlias appended

* Fix failing tests

* alias setup

* fix suggestion not working due to alias

* fix getIndexOrAliasName method for multiple indexes

* update openmetadata.yaml

* update childAliases with clusterAlias

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-08-10 11:26:49 -07:00
Sriharsha Chintalapani
ed2115067a
fix: docker/development/Dockerfile to reduce vulnerabilities (#16855)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-ALPINE319-OPENSSL-7413523
- https://snyk.io/vuln/SNYK-ALPINE319-OPENSSL-7413523
- https://snyk.io/vuln/SNYK-ALPINE319-OPENSSL-7413527
- https://snyk.io/vuln/SNYK-ALPINE319-OPENSSL-7413527

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2024-07-31 21:26:51 +05:30
k.nakagaki
9a31a35296
Fixes 9875: supporting gcp secret manager (#16505)
* Split ExternalSecretsManagerTest to new ExternalSecretsManagerTest and AWSBasedSecretsManagerTest

* implement SecretsManagerFactory to create GCPSecretsManager

* implement GCPSecretsManager

* implements gcp secret manager.

* Fix it for the GCP's rule.

* create a template of GCP

* fix compile error

* implements to use project_id

* add library for the google cloud secret manager

* add test code for using google credential in the docker container

* modify docker-compose.yml for GCP

* add google_crc32c module

* modify ways to get project id

* create a new docker-compose.yml for Google Cloud

* create a new document

* create compose file for gcp secret manager

* fix invalid styles and formats

* downgrade google library to avoid conflicting protoc versions
2024-06-28 21:09:02 -07:00
Mohit Yadav
37e1472fa3
Add Jwt Claim Mapping Env (#16780) 2024-06-24 23:15:19 +05:30
Mohit Yadav
a5295396bd
Add WebConf Env to Compose (#16713) 2024-06-19 12:34:33 +05:30
Shailesh Parmar
1d4e3e0825
Revert alias setup (#16610)
* Revert "fix(CI): update `run_local_docker.sh` script with cluster alias ES endpoint (#16604)"

This reverts commit baab52076f71f76e8bc48d89564ba733ac77a1e6.

* Revert "chore(CI): Enable Elasticsearch cluster alias by default (#16599)"

This reverts commit a3524e14d3a1244db19199ab1eae3796ba15b499.
2024-06-11 19:14:25 +05:30
Akash Jain
a3524e14d3
chore(CI): Enable Elasticsearch cluster alias by default (#16599) 2024-06-11 11:34:24 +05:30
sonika-shah
c010939072
Added payload size check logic for Elasticsearch (#15540)
* Added payload size check logic for Elasticsearch

* Added payload size check logic for Elasticsearch - made review changes

* Added payload size check logic for Elasticsearch - added env in docker files

* Optimize payload logic, fix search application error stats issue
2024-04-15 16:25:06 +05:30
Shin-ichi Hashiba
ba7b5d248a
delete duplicate env (#15709) 2024-04-01 18:21:51 +05:30
Mohit Yadav
e8f8271b77
- Add docker vars (#15619)
- Modified Azure refresh token logic
2024-03-20 11:47:25 +05:30
Imri Paran
aade838020
Fixes #15388: Use native backup tools (#15393)
* feat: use native backup tools

1. added mysqldump 8.3 to the ingestion container.
2. documented how to use native tools to back up and restore.
3. added deprecated message on the cli backup and restore.

* added deprecation notice for 1.3 backup

* removed 1.3.x deprecation notice

* added another backup page in 1.3 introducing SQL dump tools

* added --set-gtid-purged=OFF to the mysql dump process
2024-03-12 06:23:05 +01:00
Pere Miquel Brull
62c0cc7563
#13985 - Azure KV Secrets Manager (#15192)
* #13985 - Azure KV Secrets Manager

* Format

* #13985 - Azure KV Secrets Manager

* #13985 - Azure KV Secrets Manager

* Simplify credentials loading

* Simplify credentials loading

* Simplify credentials loading
2024-02-20 07:18:35 +01:00
Akash Jain
065a8e30da
fix: docker-compose failed randomly without healthcheck of elasticsearch. #14854 (#15034) 2024-02-05 18:28:00 +05:30
Akash Jain
8574e9a742
feat(docker-compose): Use openmetadata-ops.sh script for migrations (#14990) 2024-02-05 11:22:53 +05:30
Akash Jain
69e6f50769
chore(docker-compose): fix DB_PARAMS for Docker Compose files (#14931) 2024-01-29 19:28:07 +05:30
Dhruvin Maniar
8b5faae720
minor: fix alpine package to avoid vulnerabilities (#14926) 2024-01-29 11:33:24 +00:00
Akash Jain
cb893cdd65
fix(CI): Optimize Server Docker Image Layers (#14835) 2024-01-24 16:03:18 +05:30
Akash Jain
07eb1ccdf5
fix: Add SAML and LDAP env configs; disabled by default (#14453) 2023-12-25 18:08:38 -08:00
Pere Miquel Brull
d8984d267e
#14340 & #13849 - Clean secret ID and improve encrypt/decrypt exception management (#14356)
* Fix supported characters in SM

* Update SM

* Fixes

* Fixes

* Improve class conversion exceptions

* Comments

* Rename noop to db secrets manager providee

* Update sm

* Fix

* db SM

* db SM

* Fix test

* UI

* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts

* update default
2023-12-18 06:43:20 +01:00
Akash Jain
6eefd51180
fix: AIRFLOW_DB_PROPERTIES default value in docker compose files (#14274) 2023-12-06 19:56:19 +05:30
Akash Jain
a5a3347fd3
fix: Include DB_PROPERTIES env for docker-compose files (#14264) 2023-12-06 13:13:49 +05:30
Preet Shah
20cf8f63ab
Update ingestion_dependency.sh (#14139)
* Update ingestion_dependency.sh

* Add parameter for DB connection properties

---------

Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
2023-12-04 18:09:42 +05:30
Preet Shah
597c49985e
Update omd-server's base image to alpine:3.18.5 (#14199)
* Update server's base image to alpine:3.18.5

* Update server's base image tags

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-12-04 14:54:22 +05:30
Pere Miquel Brull
7fcdf08ca4
#11626 & #14131 - Lineage with other Entities & attr-based xlets (#14191)
* Add OMEntity model

* Test OMEntity

* Update repr

* Fix __str__

* Add entity ref map

* Test serializer for backend

* Fix tests

* Fix serializer

* Test runner

* Add runner tests

* Update docs

* Format
2023-12-01 06:29:44 +01:00
Mohit Yadav
19dbc217b0
Add code flow Configuration (#14026) 2023-11-20 08:44:13 +05:30
Mohit Yadav
2ae0333551
Fix Email Issues (#13962)
* - Added Unauthenticated Server Emails
- Removed redundant OM_URI
- Add Test Email in SystemResources

* remove default port

* remove fernet
2023-11-14 09:04:53 +05:30
Preet Shah
81d695b1a2
Update Dockerfiles to run OMD as non-root user (#13726)
* Update Dockerfiles to run OMD as non-root user

* Update Dockerfile to use alpine:3 image

* Update Dockerfile to use alpine:3 docker image

* Update Dockerfile to use alpine:3 docker image
2023-11-09 17:52:10 +05:30
Mohit Yadav
2577490700
- Removed Logo Configuration from the openmetadata.yaml (#13873)
* - Removed Logi Configuration from the openmetadata.yaml
- Migrated Login Configuration to the UI

* Removed ApplicationConfig from tests

* Added Custom Header Logo Url

* Update field to favicon

* support login conifguration

* chore(ui): update logo and login config md files

* fix validation issue

* add cypress

* restructure folders and set default fallback for favicon

* autofocus to first field of edit form

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-11-08 20:05:00 +05:30
Teddy
208241780f
fix: revert compose changes (#13560) 2023-10-13 12:32:11 +05:30
Teddy
1cbdfb3ae7
Fixes #12601 - column filter for profiler workflow (#13535)
* fix: sample data ingestion to match entity profiler column setting

* fix: python linting

* fix: updated fn call

* fix: added logic to handle json filed in datalake connector

* fix: handle NA values in parsing

* fix: reverted sampler changes from #13338

* fix: reverted metric changes from #13338

* fix: added datalake profiler ingestion test

* fix: python linting

* fix: removed normalization of json blob in NoSQL db
2023-10-12 14:51:38 +02:00
Teddy
e57849b732
Fixes #12298 - Update report data type to camel case (#13505)
* fix: updated DI to camelCase

* fix: ran linting

* fix: added migration

* fix: remove extra parenthesis in migration file

* fix: psql migration query

* fix: OS compose host

* fix: removed commented code block
2023-10-11 08:14:21 +02:00
Akash Jain
8b028a737b
fix(CI): postgresql ci with opensearch (#13500)
* fix(CI): Postgresql Docker Compose Services Dependency

* fix(CI): update `SEARCH_TYPE` to `opensearch`
2023-10-09 20:18:13 +05:30
Akash Jain
9549df48b4
fix(CI): Postgresql Docker Compose Services Dependency (#13492) 2023-10-09 07:42:20 -07:00
Sriharsha Chintalapani
35c4e18663
Upgrade search versions (#13479)
* working on maven shading

* update es in docker

* add maven shade for elasticsearch/opensearch dependencies

* add submodule for es/os dependencies

* upgrade docker elasticsearch to latest 8.10.2

* upgrade docker elasticsearch to latest 8.10.2

* Update Postgres dev to use OpenSearch

* Fix container service dependency

* revert conf change

---------

Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-08 11:55:34 -07:00
Sriharsha Chintalapani
03ca972887
Add Java 17 support (#12895)
* Add Java 17 support

* Change Test HTTP client provider

* Create Tests HTTP Client once

* Create Tests HTTP Client once

* fix(CI): Update CI to use jdk 17 and dockerfiles as well

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-10-03 01:35:50 -07:00
Akash Jain
7d4957537c
fix(CI): enable force migrations for CI (#13026) 2023-08-29 13:39:08 +05:30
Akash Jain
0a7f30977a
fix(chore): Remove OPENMETADATA_DEBUG Param (#12997) 2023-08-24 20:17:50 +05:30
Mohit Yadav
9712337b01
[SecurityHeaders] Fixed issue with Security Headers (#12921) 2023-08-18 17:48:17 +05:30
Dhruvin Maniar
403e407478
[FEAT] Implement Docker InitContainer for migration (#12714)
* [FEAT] Implement Docker InitContainer for migration

* [FEAT] Implement Docker InitContainer for migration

* [FEAT] Implement Docker InitContainer for migration

* [FEAT] Implement Docker InitContainer for migration

---------

Co-authored-by: Dhruvin Maniar <dhruvinmaniar@192.168.1.27>
Co-authored-by: Dhruvin Maniar <dhruvinmaniar@Dhruvins-MacBook-Pro.local>
Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
2023-08-03 16:01:20 +05:30
Akash Jain
1b54a9d6bf
fix: Docker Compose Postgres Command Args (#12693) 2023-08-01 18:56:51 +05:30