26 Commits

Author SHA1 Message Date
Anuj359
9d649964fd
Creating make script for code freeze automation (#12976)
* Disk space test

* Disk space test with other action

* Disk space test with other action

* Disk space test with first action

* checking labeler

* updating version of wait action

* xvfb-run check

* Update disk-space-test.yml

* xvfb-run check using sudo

* checking large packages

* Update disk-space-test.yml

* Update disk-space-test.yml

* Update disk-space-test.yml

* Update disk-space-test.yml

* Checking the logic

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update update_version.py

* Update Makefile

* Update update_version.py

* Update Makefile

* permission issue check

* Update Makefile

* Update Makefile

* Update Makefile

* Update Makefile

* Update Makefile

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update Makefile

* Creating make script for code freeze automation

* updating dockerfile version with main branch

* Updating paths in makefile

* Updating the python code with logger

* Using argparse to pass the arguments in python code

* feat:Updating script path

* Delete update_version.py from root directory
2023-09-13 15:10:28 +05:30
Pere Miquel Brull
a3bfd4e696
Part of #11968 - Restructure Profiler Workflow and PII Processor (#13059)
* Structure PII

* Restructure Profiler Workflow

* Update signature for abc

* remove profiler sink

* Fix tests

* Fix lint

* Fix test

* Fix test
2023-09-04 11:02:57 +02:00
Pere Miquel Brull
6c0e9f5061
Part of #7272 - Centralize Workflows, Status, and Exception Management (#13029)
* Prep changes

* Prep changes

* prep changes

* Update imports

* Format

* Prep delete

* Prep delete

* Fix sink

* Prep test

* Commit

* passing either

* passing either

* Prep Either

* Metadata source with Either

* Update status

* Merge remote-tracking branch 'upstream/main' into issue-7272

* Format

* Linting

* Linting

* Linting

* Linting

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Comments
2023-08-30 15:49:42 +02:00
Matias Puerta
e203ece20c
[CHORE] Reduce docker image size by removing pip cache (#12708)
* [CHORE] Reduce docker image size by removing pip cache

* [CHORE] Reduce image size for ingestion/operators as well

* [CHORE] Reduce image size for CI
2023-08-02 14:36:27 +02:00
Anuj359
1417621ddc
security-scan fix (#12585) 2023-07-27 11:41:25 +05:30
Anuj359
7f55b808e5
fix(chore): Prepare Main Branch with next release (1.2.0) (#12364)
* Updating Dockerfile with multistage

* Updating reviewed changes

* Docker Development changes

* Docker workflow changes

* Arguments update

* Script path update

* Arguments update

* Resolving the reviewed suggestions

* Preparing for next release and updating docker sort value changes

* Adding .dev0 in the python related release version

* Adding .dev0 in the python related release version
2023-07-12 16:55:18 +05:30
Onkar Ravgan
9cbaf98435
remvoed ibm db arch (#12158) 2023-06-26 15:34:55 +02:00
Pere Miquel Brull
d0bc44127e
Set logger level for ingestion-base (#12137)
* Set logger level for ingestion-base

* Set logger level for ingestion-base
2023-06-25 08:20:46 -07:00
Onkar Ravgan
23b153991b
fixed mysqlclient dep (#12087) 2023-06-22 15:18:39 +05:30
Pere Miquel Brull
4a8554c313
Fix Redshift performance and memory issues for ingestion-base (#12002)
* base op

* Update redshift deps and ingestion-base image

* Remove vim

* Remove vim
2023-06-19 12:09:09 +02:00
Pere Miquel Brull
7ff00d399e
Install ibm-db-sa if not ARM (#11798) 2023-05-30 10:36:48 +05:30
Pere Miquel Brull
e02fc231da
Improve logging for test connection workflow (#11794) 2023-05-30 10:36:10 +05:30
Pere Miquel Brull
c0ea11a477
Add libsasl deps (#11613) 2023-05-16 14:32:48 +02:00
Pere Miquel Brull
e453593760
Rel to #10805 - Ingestion docs glue & Kerberos Dep (#11157)
* Glue

* Add kerberos system dep
2023-04-20 10:56:46 +02:00
Teddy
9b4e9132ae
fixed #9656 - Add support for date type to column values to be between (#10890)
* fix: renamed  to  submodule

* fix: linting

* fix: columnValuesToBeBetween test for date column type
2023-04-04 17:16:44 +02:00
Pere Miquel Brull
2e2c6a0cdf
Operators directory cleanup (#10834)
* Remove old connectors directory

* Add mssql test for url

* Format
2023-03-30 09:51:21 +02:00
Mayur Singal
54b635dd60
1.0.0-alpha version upgrade (#10826) 2023-03-29 23:24:39 +05:30
Pere Miquel Brull
78d7dd8789
[WIP] - Test Connection - Prepare the new test connection ingestion+UI logic (#10660)
* Prepare the new test connection ingestion logic

* Update test assert

* Update Test Connection for SQA Sources

* Correct return type and method doc

* Handle decryption

* Non SQA Database Sources

* Add the run_automation script in ingestion-base

* Dashboard Test Connection Changes

* Pipeline, Messagin, MlModel & Metadata Sources

* ui: test connect flow-1

* Unmask connection parameters before sending to Ariflow

* ui: test connect flow-2

* Address review comments and pylint

* pytest fix

* ui: test connect flow-3 (refactoring and style fix)

* ui: test connect flow-4 (fix test connection status logic)

* sync local file

* ui: test connect flow-5 (fix lowercase issue and styling)

* ui: test connect flow-5 (show toast notifications)

* test: add unit test

* ui: test connect flow-5 (update service page test connection button)

* Databrick fix & pytest fix

* pylint

* Update test

* Fix merge

* S3 Test connection

* add style for mandatory step

* sync locales

* chore: add service name in workflow request

* Unmask using original service connection parameters

* Fix test connection unmasking

* Wrap inspector function to eliminate error outside test conn

* Fix linting

* fix:cy test

* Fix linting

* address comment

* refactor and fix connection type casing issue

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-28 06:29:13 +02:00
Pere Miquel Brull
db292eaa0b
Prepare exit handler & Add missing arg to test connection (#10519)
* Prepare exit handler

* clean envs
2023-03-13 10:50:10 +01:00
Pere Miquel Brull
477a5223eb
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
2023-03-06 14:44:16 +01:00
Teddy
754074f1be
Fixes #7758 - Added Column value and Integer Range Partitionning (#10350)
* feat(profiler): renamed  module to

* feat(profiler): added dbt-artifacts-parser to test setup.py

* feat(profiler): refactor workflow and interface

* feat(profiler): linting

* feat(profiler): removed old profiler modules

* feat(profiler): added support for value and integer range partition

* feat(profiler): fixed linting

* feat(profiler): added partitionning support for datalake profiler

* feat(profiler): removed `ProfilerInterfaceArgs` class

* feat(profiler): address comments

* feat(profiler): Added `OTHER` as an `IntervalType` for UI type generation
2023-03-01 08:20:38 +01:00
Pere Miquel Brull
203e2b5fff
Simplify Pipeline Service Client Response and prep test connection for ingestion base (#10334)
* Simplify Response and prep test connection

* Format
2023-02-25 15:52:14 -08:00
Nahuel
b9a3c06104
Bump main branch to version 1.0.0 (#10040)
* Bump to version 0.13.2

* Bump mvn projects to 1.0.0-SNAPSHOT

* Bump python projects to 1.0.0.dev0

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-02-02 12:56:14 +01:00
Pere Miquel Brull
b0058bdb65
Fix #9707 - Install instantclient and enable thick mode (#9761)
* Install instantclient

* Prep instant client

* Update docs

* Amend comments
2023-01-19 08:08:56 +01:00
Pere Miquel Brull
615b1a9647
Add Airflow deps installation (#8131) 2022-10-13 11:06:27 +02:00
Pere Miquel Brull
339abc5bf3
Prepare Ingestion Base Docker image (#8065)
Prepare Ingestion Base Docker image (#8065)
2022-10-11 07:50:49 +02:00