230 Commits

Author SHA1 Message Date
Akash Jain
015d71c9de
chore: Update main branch for 1.5.0 Release (#16401) 2024-05-23 18:46:15 +05:30
Ayush Shah
a47191ecaf
MINOR: Add PROCESS Priviledge (#16275)
* Fix PROCESS Priviledge issue

* Add Flush priviledges
2024-05-15 23:21:42 -07:00
Pere Miquel Brull
4d75ce03bd
MINOR - Fix validate compose CI (#16205) 2024-05-10 06:44:27 +02:00
sonika-shah
c010939072
Added payload size check logic for Elasticsearch (#15540)
* Added payload size check logic for Elasticsearch

* Added payload size check logic for Elasticsearch - made review changes

* Added payload size check logic for Elasticsearch - added env in docker files

* Optimize payload logic, fix search application error stats issue
2024-04-15 16:25:06 +05:30
Pere Miquel Brull
a1404e6b4a
MINOR - Clean ingestion dependencies (#15679)
* WIP - MINOR - Clean ingestion dependencies

* test

* test

* Clean imports

* add pyiceberg for test

* Revert "add pyiceberg for test"

This reverts commit ab26942736586f089a57a644ffd727aca200db62.

* add pyiceberg for test

* Remove docker dep

* clean local docker sh

* MINOR - AKS Airflow troubleshooting docs

* Fix action

* clean local docker sh
2024-04-11 14:30:40 +02:00
Shin-ichi Hashiba
ba7b5d248a
delete duplicate env (#15709) 2024-04-01 18:21:51 +05:30
Mohit Yadav
e8f8271b77
- Add docker vars (#15619)
- Modified Azure refresh token logic
2024-03-20 11:47:25 +05:30
Imri Paran
aade838020
Fixes #15388: Use native backup tools (#15393)
* feat: use native backup tools

1. added mysqldump 8.3 to the ingestion container.
2. documented how to use native tools to back up and restore.
3. added deprecated message on the cli backup and restore.

* added deprecation notice for 1.3 backup

* removed 1.3.x deprecation notice

* added another backup page in 1.3 introducing SQL dump tools

* added --set-gtid-purged=OFF to the mysql dump process
2024-03-12 06:23:05 +01:00
Imri Paran
18c22c4178
Fixes #10013: Implement first stage of NoSQL profiler (#15189)
* feat(nosql-profiler): row count

1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.

* added profiler support for mongodb connection

* doc

* use int_admin_ometa in test setup

* - fixed linting issue in gx
- removed unused inheritance

* moved the nosql function into the metric class

* formatting

* validate_compose: raise exception for bad status code.

* fixed import

* format
2024-02-22 11:46:19 +01:00
Pere Miquel Brull
62c0cc7563
#13985 - Azure KV Secrets Manager (#15192)
* #13985 - Azure KV Secrets Manager

* Format

* #13985 - Azure KV Secrets Manager

* #13985 - Azure KV Secrets Manager

* Simplify credentials loading

* Simplify credentials loading

* Simplify credentials loading
2024-02-20 07:18:35 +01:00
Akash Jain
95bb221025
feat: Prepare main for next release cycle (#15139) 2024-02-12 16:56:50 +05:30
Akash Jain
065a8e30da
fix: docker-compose failed randomly without healthcheck of elasticsearch. #14854 (#15034) 2024-02-05 18:28:00 +05:30
Akash Jain
8574e9a742
feat(docker-compose): Use openmetadata-ops.sh script for migrations (#14990) 2024-02-05 11:22:53 +05:30
Akash Jain
69e6f50769
chore(docker-compose): fix DB_PARAMS for Docker Compose files (#14931) 2024-01-29 19:28:07 +05:30
Akash Jain
6e2daed38b
fix(CI): Dockerfile Instructions (#14927) 2024-01-29 17:39:33 +05:30
Dhruvin Maniar
8b5faae720
minor: fix alpine package to avoid vulnerabilities (#14926) 2024-01-29 11:33:24 +00:00
Akash Jain
dcc87394ee
Revert "fix: docker/docker-compose-quickstart/Dockerfile to reduce vulnerabilities (#14862)" (#14913)
This reverts commit 390e731eb42e8b1757cefd0b302d20bc83e049a9.
2024-01-29 11:14:20 +05:30
Sriharsha Chintalapani
390e731eb4
fix: docker/docker-compose-quickstart/Dockerfile to reduce vulnerabilities (#14862)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-ALPINE318-OPENSSL-6152404
- https://snyk.io/vuln/SNYK-ALPINE318-OPENSSL-6152404
- https://snyk.io/vuln/SNYK-ALPINE318-OPENSSL-6160000
- https://snyk.io/vuln/SNYK-ALPINE318-OPENSSL-6160000

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2024-01-28 10:27:34 -08:00
Akash Jain
cb893cdd65
fix(CI): Optimize Server Docker Image Layers (#14835) 2024-01-24 16:03:18 +05:30
Shiyang Xiao
241f3c68d7
Fixes #14413: Add SAS connector (#14415)
* feat: SAS Viya connector

* refactor SASCatalog to SAS

* add SAS logo to UI and connection documentation

* doc changes

* modify ingestion logic

* revert original changes

* added support for dataflow & perfect logic for reports/datatables

* add filter doc

* more updates to perfect ingestion for each asset type

* fix a bug with table lineage not created properly

* Delete ingestion/pipelines/sasCatalog.yaml

* precomit fix

* Conversion to database connector

* minor fixes

* make custom properties type generic

* Add SAS javaEnum

* add dummy variable for sas.yaml

---------

Co-authored-by: lizmc <liz.mcintosh@sas.com>
Co-authored-by: Shiyang Xiao <Shiyang.Xiao@sas.com>
2024-01-11 06:46:57 -08:00
Ayush Shah
9c6d202555
Add Sample data, modify regex pattern (#14467) 2024-01-11 14:23:33 +05:30
Akash Jain
07eb1ccdf5
fix: Add SAML and LDAP env configs; disabled by default (#14453) 2023-12-25 18:08:38 -08:00
Pere Miquel Brull
d8984d267e
#14340 & #13849 - Clean secret ID and improve encrypt/decrypt exception management (#14356)
* Fix supported characters in SM

* Update SM

* Fixes

* Fixes

* Improve class conversion exceptions

* Comments

* Rename noop to db secrets manager providee

* Update sm

* Fix

* db SM

* db SM

* Fix test

* UI

* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts

* update default
2023-12-18 06:43:20 +01:00
Akash Jain
6eefd51180
fix: AIRFLOW_DB_PROPERTIES default value in docker compose files (#14274) 2023-12-06 19:56:19 +05:30
Akash Jain
a5a3347fd3
fix: Include DB_PROPERTIES env for docker-compose files (#14264) 2023-12-06 13:13:49 +05:30
Preet Shah
20cf8f63ab
Update ingestion_dependency.sh (#14139)
* Update ingestion_dependency.sh

* Add parameter for DB connection properties

---------

Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
2023-12-04 18:09:42 +05:30
Preet Shah
597c49985e
Update omd-server's base image to alpine:3.18.5 (#14199)
* Update server's base image to alpine:3.18.5

* Update server's base image tags

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-12-04 14:54:22 +05:30
Pere Miquel Brull
7fcdf08ca4
#11626 & #14131 - Lineage with other Entities & attr-based xlets (#14191)
* Add OMEntity model

* Test OMEntity

* Update repr

* Fix __str__

* Add entity ref map

* Test serializer for backend

* Fix tests

* Fix serializer

* Test runner

* Add runner tests

* Update docs

* Format
2023-12-01 06:29:44 +01:00
Mohit Yadav
19dbc217b0
Add code flow Configuration (#14026) 2023-11-20 08:44:13 +05:30
Mohit Yadav
3f8a931e39
Bump Pom Version to 1.3.0-SNAPSHOT (#14008)
* Bump Pom Version to 1.3.0-SNAPSHOT

* chore: Fix Makefile recipe

* fix: Prepare Main Branch for Next Feature Release

* fix: Syntax issue

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-11-17 11:33:47 +05:30
Mohit Yadav
2ae0333551
Fix Email Issues (#13962)
* - Added Unauthenticated Server Emails
- Removed redundant OM_URI
- Add Test Email in SystemResources

* remove default port

* remove fernet
2023-11-14 09:04:53 +05:30
Preet Shah
81d695b1a2
Update Dockerfiles to run OMD as non-root user (#13726)
* Update Dockerfiles to run OMD as non-root user

* Update Dockerfile to use alpine:3 image

* Update Dockerfile to use alpine:3 docker image

* Update Dockerfile to use alpine:3 docker image
2023-11-09 17:52:10 +05:30
Mohit Yadav
bd6e2962c0
Remove Migration Extension Path Env (#13908) 2023-11-09 14:04:28 +05:30
Mohit Yadav
2577490700
- Removed Logo Configuration from the openmetadata.yaml (#13873)
* - Removed Logi Configuration from the openmetadata.yaml
- Migrated Login Configuration to the UI

* Removed ApplicationConfig from tests

* Added Custom Header Logo Url

* Update field to favicon

* support login conifguration

* chore(ui): update logo and login config md files

* fix validation issue

* add cypress

* restructure folders and set default fallback for favicon

* autofocus to first field of edit form

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-11-08 20:05:00 +05:30
Akash-Jain
36c0984f97 fix(docker-env): Update syntax for ENV files 2023-10-25 11:49:30 +05:30
Akash Jain
08d1747a7d
fix(docker-env): Update syntax for ENV files (#13680) 2023-10-23 11:12:08 +05:30
Teddy
208241780f
fix: revert compose changes (#13560) 2023-10-13 12:32:11 +05:30
Teddy
1cbdfb3ae7
Fixes #12601 - column filter for profiler workflow (#13535)
* fix: sample data ingestion to match entity profiler column setting

* fix: python linting

* fix: updated fn call

* fix: added logic to handle json filed in datalake connector

* fix: handle NA values in parsing

* fix: reverted sampler changes from #13338

* fix: reverted metric changes from #13338

* fix: added datalake profiler ingestion test

* fix: python linting

* fix: removed normalization of json blob in NoSQL db
2023-10-12 14:51:38 +02:00
Teddy
e57849b732
Fixes #12298 - Update report data type to camel case (#13505)
* fix: updated DI to camelCase

* fix: ran linting

* fix: added migration

* fix: remove extra parenthesis in migration file

* fix: psql migration query

* fix: OS compose host

* fix: removed commented code block
2023-10-11 08:14:21 +02:00
Mohit Yadav
63ac994371
Point Reindex request to new Applications api (#13508) 2023-10-10 17:05:07 +05:30
Akash Jain
8b028a737b
fix(CI): postgresql ci with opensearch (#13500)
* fix(CI): Postgresql Docker Compose Services Dependency

* fix(CI): update `SEARCH_TYPE` to `opensearch`
2023-10-09 20:18:13 +05:30
Akash Jain
9549df48b4
fix(CI): Postgresql Docker Compose Services Dependency (#13492) 2023-10-09 07:42:20 -07:00
Ayush Shah
1a8b11b29e
Fix Github Actions, Timeout (#13488) 2023-10-09 15:01:23 +05:30
Sriharsha Chintalapani
35c4e18663
Upgrade search versions (#13479)
* working on maven shading

* update es in docker

* add maven shade for elasticsearch/opensearch dependencies

* add submodule for es/os dependencies

* upgrade docker elasticsearch to latest 8.10.2

* upgrade docker elasticsearch to latest 8.10.2

* Update Postgres dev to use OpenSearch

* Fix container service dependency

* revert conf change

---------

Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-08 11:55:34 -07:00
Sriharsha Chintalapani
03ca972887
Add Java 17 support (#12895)
* Add Java 17 support

* Change Test HTTP client provider

* Create Tests HTTP Client once

* Create Tests HTTP Client once

* fix(CI): Update CI to use jdk 17 and dockerfiles as well

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-10-03 01:35:50 -07:00
Anuj359
12e2813ae0
Fix:Updating Docker Label for version update via makefile (#13241)
* Fix:Updating Docker Label for version update via makefile

* Fix:Updating Docker Label for version update via makefile

* Fix:Updating Docker Label for version update via makefile
2023-09-18 12:37:44 +05:30
Anuj359
9d649964fd
Creating make script for code freeze automation (#12976)
* Disk space test

* Disk space test with other action

* Disk space test with other action

* Disk space test with first action

* checking labeler

* updating version of wait action

* xvfb-run check

* Update disk-space-test.yml

* xvfb-run check using sudo

* checking large packages

* Update disk-space-test.yml

* Update disk-space-test.yml

* Update disk-space-test.yml

* Update disk-space-test.yml

* Checking the logic

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update update_version.py

* Update Makefile

* Update update_version.py

* Update Makefile

* permission issue check

* Update Makefile

* Update Makefile

* Update Makefile

* Update Makefile

* Update Makefile

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update Makefile

* Creating make script for code freeze automation

* updating dockerfile version with main branch

* Updating paths in makefile

* Updating the python code with logger

* Using argparse to pass the arguments in python code

* feat:Updating script path

* Delete update_version.py from root directory
2023-09-13 15:10:28 +05:30
Akash Jain
7d4957537c
fix(CI): enable force migrations for CI (#13026) 2023-08-29 13:39:08 +05:30
Akash Jain
0a7f30977a
fix(chore): Remove OPENMETADATA_DEBUG Param (#12997) 2023-08-24 20:17:50 +05:30
Mohit Yadav
9712337b01
[SecurityHeaders] Fixed issue with Security Headers (#12921) 2023-08-18 17:48:17 +05:30