60 Commits

Author SHA1 Message Date
Pere Miquel Brull
0e92a975e3
#14425 - Create ingestion-base-slim image (#14426)
* #14425 - Create ingestion-base-slim image

* Format

* Bump airflow

* Bump constraints
2023-12-19 11:09:38 +01:00
Pere Miquel Brull
6d046f7e3b
Revert "Update Dockerfile.ci file to use new MySQL Keys (#14400)" (#14416)
This reverts commit 33f2fdd49cae50c93cbf1f8544402bda41070a01.
2023-12-16 16:46:33 +01:00
Preet Shah
33f2fdd49c
Update Dockerfile.ci file to use new MySQL Keys (#14400)
* Update Dockerfile.ci file to use new MySQL Keys

* Update ingestion Dockerfiles files to use new MySQL Keys

* Update ingestion Dockerfiles files

* Update Dockerfiles to not hardcode the new keys

* Update comments
2023-12-15 15:46:30 +05:30
Mohit Yadav
3f8a931e39
Bump Pom Version to 1.3.0-SNAPSHOT (#14008)
* Bump Pom Version to 1.3.0-SNAPSHOT

* chore: Fix Makefile recipe

* fix: Prepare Main Branch for Next Feature Release

* fix: Syntax issue

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-11-17 11:33:47 +05:30
Pere Miquel Brull
8c6777f820
Fix #12406 - Ingestion image with python 3.10 (#13949) 2023-11-13 15:08:06 +05:30
Nguyen Huu Loc
7ff6738527
Fix looker missing git execution on container image (#13457)
* - Add git execution to ingestion Dockerfile
- [Looker] Update missing function

* Fix pylint

* Add git execution to Dockerfile

* Remove log

---------

Co-authored-by: Loc Nguyen <loc.nguyenhuu@xendit.co>
2023-10-06 06:51:07 +02:00
Anuj359
9d649964fd
Creating make script for code freeze automation (#12976)
* Disk space test

* Disk space test with other action

* Disk space test with other action

* Disk space test with first action

* checking labeler

* updating version of wait action

* xvfb-run check

* Update disk-space-test.yml

* xvfb-run check using sudo

* checking large packages

* Update disk-space-test.yml

* Update disk-space-test.yml

* Update disk-space-test.yml

* Update disk-space-test.yml

* Checking the logic

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update update_version.py

* Update Makefile

* Update update_version.py

* Update Makefile

* permission issue check

* Update Makefile

* Update Makefile

* Update Makefile

* Update Makefile

* Update Makefile

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update Makefile

* Creating make script for code freeze automation

* updating dockerfile version with main branch

* Updating paths in makefile

* Updating the python code with logger

* Using argparse to pass the arguments in python code

* feat:Updating script path

* Delete update_version.py from root directory
2023-09-13 15:10:28 +05:30
Anuj359
e5489912ea
Upgrade: airflow base image from 2.5.3 to 2.6.3 (#13151) 2023-09-12 20:04:25 +05:30
Akash Jain
fd0b1b9fed
fix: Align Docker Base Image to use airflow 2.5.3 (#12737) 2023-08-04 12:08:58 +05:30
Matias Puerta
e203ece20c
[CHORE] Reduce docker image size by removing pip cache (#12708)
* [CHORE] Reduce docker image size by removing pip cache

* [CHORE] Reduce image size for ingestion/operators as well

* [CHORE] Reduce image size for CI
2023-08-02 14:36:27 +02:00
Anuj359
1417621ddc
security-scan fix (#12585) 2023-07-27 11:41:25 +05:30
Anuj359
22e87df5af
Fixing/security scan (#12562)
* Updating Dockerfile with multistage

* Updating reviewed changes

* Docker Development changes

* Docker workflow changes

* Arguments update

* Script path update

* Arguments update

* Resolving the reviewed suggestions

* Updating omd api version for security scan
2023-07-24 11:52:45 +05:30
Pere Miquel Brull
a3bff293a4
Fix #12190 - Bring support for Airflow 2.6 & Use Airflow 2.6.3 as the new ingestion base (#12398)
* Bump Airflow version

* Support Airflow 2.6

* Lint

* Bump airflow version

* Fix airflow 2.6 lineage

* Fix airflow 2.6 lineage
2023-07-13 06:41:59 +02:00
Anuj359
7f55b808e5
fix(chore): Prepare Main Branch with next release (1.2.0) (#12364)
* Updating Dockerfile with multistage

* Updating reviewed changes

* Docker Development changes

* Docker workflow changes

* Arguments update

* Script path update

* Arguments update

* Resolving the reviewed suggestions

* Preparing for next release and updating docker sort value changes

* Adding .dev0 in the python related release version

* Adding .dev0 in the python related release version
2023-07-12 16:55:18 +05:30
Onkar Ravgan
9cbaf98435
remvoed ibm db arch (#12158) 2023-06-26 15:34:55 +02:00
Onkar Ravgan
23b153991b
fixed mysqlclient dep (#12087) 2023-06-22 15:18:39 +05:30
Akash Jain
ce9a10dd32
fix: CI Snyk Reports (#11874) 2023-06-05 10:43:52 +05:30
Pere Miquel Brull
7ff00d399e
Install ibm-db-sa if not ARM (#11798) 2023-05-30 10:36:48 +05:30
Pere Miquel Brull
c0ea11a477
Add libsasl deps (#11613) 2023-05-16 14:32:48 +02:00
Pere Miquel Brull
aaf92a9ae8
Bump python-daemon (#11477) 2023-05-08 09:59:57 +02:00
Pere Miquel Brull
e453593760
Rel to #10805 - Ingestion docs glue & Kerberos Dep (#11157)
* Glue

* Add kerberos system dep
2023-04-20 10:56:46 +02:00
Mayur Singal
54b635dd60
1.0.0-alpha version upgrade (#10826) 2023-03-29 23:24:39 +05:30
Nahuel
b9a3c06104
Bump main branch to version 1.0.0 (#10040)
* Bump to version 0.13.2

* Bump mvn projects to 1.0.0-SNAPSHOT

* Bump python projects to 1.0.0.dev0

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-02-02 12:56:14 +01:00
Pere Miquel Brull
b0058bdb65
Fix #9707 - Install instantclient and enable thick mode (#9761)
* Install instantclient

* Prep instant client

* Update docs

* Amend comments
2023-01-19 08:08:56 +01:00
Chirag Madlani
bf6fc5f93a
prepare(release) next release (#9479)
* prepare(release) next release

* airflow typo

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-12-27 20:15:46 +05:30
Nahuel
a2b34dd0f4
Fix: Update Ingestion docker images and fix python libraries dependencies (#9342)
* Update Ingestion docker images and fix python libraries dependencies

* Install also apache-airlfow-providers-http
2022-12-16 14:46:25 +00:00
Pere Miquel Brull
eccbb087d3
Add table query test (#9000)
* Add table query test

* Add table query test
2022-11-25 06:56:12 +01:00
Pere Miquel Brull
ff028a6eb0
Fix Mysql bigint (#8665)
* Fix mysql bigint cast

* Remove upgrade from dockerfiles
2022-11-11 10:12:23 +01:00
Pere Miquel Brull
600aecbdf8
Bump bind9 bullseye version for ingestion container (#8619)
* Bump bind9 bullseye version

* Remove bind9 fixed version
2022-11-09 21:17:20 -08:00
Pere Miquel Brull
bc0ce0cec1
Remove expat version (#8445) 2022-10-31 13:55:44 +05:30
Nahuel
3ed6ad8456
Update GID used in ingestion docker images (#8434) 2022-10-29 13:03:37 -07:00
Pere Miquel Brull
f834b28198
patch ingestion image (#8385) 2022-10-27 15:10:04 +02:00
Akash Jain
fc8312c5fb
feat: Refactor openmetadata/ingestion docker image (#7456)
* centralize openmetadata/ingestion docker image

* update volume mappings; fix ingestion tag!

* fix PR comments

* fix cypress mysql tests!
2022-09-19 09:20:54 +05:30
Pere Miquel Brull
7d2527bbeb
Fix #6961 - Fallback to another IP service (#7019)
Fix #6961  - Fallback to another IP service (#7019)
2022-09-01 06:39:12 +02:00
Nahuel
db41be4c4a
Fix#6929: docker-compose services should wait until OpenMetadata DB is created (#7025)
* Added the docker compose for postgres

* Added changes as per the comments on PR

* Airflow Connectivity to Postgres

* Changes are done as per the comments

* Replocate changes to local dockerfile

* Fixed Airflow Dockerfile

* docker-compose services should wait until OpenMetadata DB is created

* remove unnecesary wait for db service

* Add dependencies to airflow image

* Add missing dependencies to airflow Dockerfile

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2022-08-30 23:55:46 +02:00
NiharDoshi99
4c7fd03b5a
Configured ODBC drivers (#6924) 2022-08-25 08:19:20 -07:00
Vj-L
ffdb32193f
Added libxml2 in the ingestion Dockerfile (#6778)
* Added libxml2 in the ingestion Dockerfile

* Added libxml2 in the airflow Dockerfile

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
2022-08-18 15:54:00 +05:30
Ayush Shah
a6db2e8a84
Fix for profiler: modified filter patterns and added error handling (#6608) 2022-08-08 10:43:17 +05:30
Pere Miquel Brull
b7eb41a560
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
2022-07-28 14:46:25 +02:00
Pere Miquel Brull
e51aadae03
Fix curl vulnerability (#6311) 2022-07-24 08:24:35 -07:00
Rabi Narayan Sahoo
f606d4d636
[Fix] Removed volume instructions from dockerfile. (#5851)
Co-authored-by: Rabi Narayan Sahoo <106670146+Robby30ies@users.noreply.github.com>
2022-07-05 18:12:56 +05:30
Onkar Ravgan
f1969c4fce
added dependencies for pymssql (#4876)
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-05-11 10:41:13 +02:00
Pere Miquel Brull
23b95a267b
Fix #4828 - Remove lineage backend from airflow image (#4833)
* Remove callbacks from DAG

* Remove lineage backend from image
2022-05-11 07:36:33 +02:00
Vj-L
2c7a327c6b
Issue #3094 (#4500)
* Added the named volumes for improvement

* Mapped the named volumes

* Updated the Named Volumes

* Updated the Volumes on top level

* Added the named volumes
2022-05-03 14:19:31 +05:30
Akash Jain
792c67a8f1
Arguments to provide ingestion depenedency. defaults to 'all' (#4613) 2022-05-02 16:09:44 +05:30
codingwithabhi
32d392e0cb
deltalake-test-connection-fixed (#4387) 2022-04-23 00:40:44 -07:00
Mayur Singal
e4914cfae4
Fix #4354: Added Hive SASL dependency (#4355) 2022-04-22 19:52:59 +05:30
Pere Miquel Brull
d6b49f070c
Fix #3951 - OpenMetadata Airflow REST (#4034)
Fix #3951 - OpenMetadata Airflow REST (#4034)
2022-04-11 18:38:26 +02:00
Ayush Shah
116f8d333d
Airflow Auth fix + Version Downgrade + Increased Docker Min Memory Limit (#3233) 2022-03-08 13:51:07 -08:00
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup (#1730)
* Fix typo

* Clean setup

* Update ingestion local image to be barebone on connector dependencies

* Prepare ingestion connectors base image

* Add system dependencies

* Prepare docker CLI

* Add docker provider

* Prepare entrypoint for the image

* Remove DBT pipeline as per Issue 1658

* Add TODO for ingestion build

* Bind docker socket

* Update comment

* Update README

* Use DockerOperator in sample data

* Build images with latest tag

* Prepare symlink to pass the volume to the DockerOperator

* Update README

* Prepare Base image for CI

* COPY multiple files into dir

* COPY multiple files into dir

* Remove DBT source as is now part of table ingestion

* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00