215 Commits

Author SHA1 Message Date
Shirshanka Das
b010efa6cf
fix(ingest): kafka - revert deps back to < 1.9.0 (#5476) 2022-07-24 15:04:24 -07:00
Shirshanka Das
f387fa6149
fix(ingest): ensure upgrade checks run async (#5383) 2022-07-16 17:38:33 -07:00
Tamas Nemeth
4334248953
feat(airflow): Circuit breaker and python api for Assertion and Operation (#5196) 2022-07-13 10:17:38 -07:00
Navin Sharma
f3e5afdba9
feat(quickstart): Adding env variables and cli options for customizing mapped ports in quickstart (#5353)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-07-11 22:33:24 -07:00
Mayuri Nehate
4b515e022b
feat(ingest): salesforce - add connector (#5104)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Vincent Koc <koconder@users.noreply.github.com>
2022-07-06 10:01:16 -07:00
Tamas Nemeth
c1f8227693
fix(ingestion): Reverting sqlalchemy upgrade because it caused issues with mssql and redshift-usage (#5289) 2022-06-29 18:23:36 +02:00
Tamas Nemeth
d812fd18fa
fix(ingest) redshift: Adding missing dependencies and relaxing sqlalchemy dependency (#5284)
Relaxing sqlalchemy deps to make our plugins work with Airflow 2.3
2022-06-29 16:25:48 +02:00
Mugdha Hardikar
5455122267
feat(ingest): delta-lake: adding support for delta lake (#5259)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-06-27 16:16:33 -07:00
Aseem Bansal
8c8f1b987a
fix(build): pin confluent kafka dependency (#5224) 2022-06-22 13:53:30 +05:30
Aditya Radhakrishnan
82ca92f8f9
feat(ingest): adds csv enricher ingestion source (#5221) 2022-06-22 12:25:39 +05:30
Aseem Bansal
4541379024
feat(build): changes to decrease build time, cancel runs in case of multiple commits (#5187) 2022-06-17 18:05:10 +05:30
John Joyce
5c501fba2c
fix(cli): Bumping avro-gen3 to 0.7.4 (#5098) 2022-06-07 12:56:06 -07:00
Shirshanka Das
51854a768d
fix(ci): pin google-cloud-logging to avoid pip backtracking (#5109) 2022-06-07 23:14:00 +05:30
Aseem Bansal
596853ae1f
fix(build): remove dependencies added for compatibility (#5108) 2022-06-07 19:59:17 +05:30
Aseem Bansal
e567d70d37
fix(build): reduce time taken for resolution (#5106) 2022-06-07 17:32:33 +05:30
Shirshanka Das
7cef07490f
fix(deps): ingest - moving packaging to framework_common (#5096) 2022-06-06 16:34:23 -07:00
Lal Rishav
4f82e29697
feat(ingest): snowflake using oauth for Azure (#4647) 2022-06-06 17:49:49 +05:30
Shirshanka Das
e62c647693
feat(cli): suggest upgrades when appropriate (#5091) 2022-06-04 09:34:17 -07:00
Shirshanka Das
83d9f7d6e6
fix(ingest): fix build breakage due to traitlets 5.2.2 bug (#5051) 2022-05-31 14:42:56 -07:00
Shirshanka Das
65d96b9108
fix(build): m1 build fails to install hdb-cli (#5040) 2022-05-30 11:02:33 +02:00
Aseem Bansal
64bb918939
fix(parsing): improve sql parsing, some debugging redash (#5025) 2022-05-27 19:28:45 +05:30
Ravindra Lanka
d200103ea7
fix(ingestion): Remove hana from base_dev_requirements to unblock m1 users (#5024) 2022-05-27 14:20:15 +05:30
Ravindra Lanka
26c0acf786
fix(build): fix for hana build failure for aarch64. (#5019) 2022-05-26 15:29:21 -07:00
Ravindra Lanka
108b492ed1
feat(ingestion): Add Iceberg source (#5010)
Co-authored-by: cccs-eric <eric.ladouceur@cyber.gc.ca>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-05-26 08:05:57 -07:00
buggythepirate
92338c7912
feat(ingest): Added new ingestion source SAP HANA (#4376)
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-05-26 03:42:50 -07:00
Ebu (えぶ)
2911e1ed1b
feat(ingest): Add Source from Vertica (#4555)
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-05-26 03:26:28 -07:00
Aseem Bansal
c2bb02903c
fix(dbt): missing aws dependency (#5008) 2022-05-26 14:23:50 +05:30
Aseem Bansal
b976393780
fix(bigquery): restrict protobuf version (#5007) 2022-05-26 12:26:45 +05:30
Aseem Bansal
b5e1ed739c
fix(bigquery): add rate limiting for api calls made (#4967) 2022-05-23 20:14:36 +05:30
Aseem Bansal
5ac6ba58f8
fix(build): Fix breaking changes from GE 0.15.3 (#4905) 2022-05-12 08:46:23 -07:00
Tamas Nemeth
84f71dadf0
fix(ingestion): dependencies - Downgrading typing-extension dependency to work with Airflow 2.0.2 (#4855)
* Downgrading typing-extension dependency to work with Airflow 2.0.2 restricting typing-extension on python 3.7
2022-05-09 09:43:49 -07:00
Ravindra Lanka
842fb391eb
feat(ingestion): kafka - add protobuf schema support (#4819)
Co-authored-by: Luis Angel Vicente Sanchez <luis.vicentesanchez@aaqua.live>
2022-05-04 17:07:01 -07:00
Shirshanka Das
c15b44a1de
fix(ingest): databricks - hive ingestion should not fail on table comment (#4787) 2022-04-29 13:56:42 -07:00
vanmeete
74d6d35881
feat(ingestion): add Pulsar source (#4721) 2022-04-29 15:57:02 +05:30
Ravindra Lanka
8583f397f1
Fix breaking changes from GE 0.15.3 that are affecting our Python3.6 smoke_tests. (#4779) 2022-04-28 15:50:17 -07:00
Danilo Peixoto
d2a6bc06dc
feat(ingest): feast - add support for Feast 0.18, deprecate older integration (#4094) 2022-04-26 14:35:02 -07:00
Aseem Bansal
53d357b4eb
fix(bigquery-usage): missing dependency (#4661) 2022-04-13 14:29:31 +02:00
jchen0824
524d183d93
feat: add presto-on-hive metadata ingestion source (#4625)
* feat(metadata ingestion source): add presto-on-hive metadata ingestion source

Co-authored-by: Houren Chen <houren.chen@grabtaxi.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-11 17:46:44 -07:00
Tamas Nemeth
e2a617f183
Restricting pytest docker version (#4639) 2022-04-11 23:35:34 +02:00
Aseem Bansal
61a95f41ae
chore: fix lint and remove incorrect integration mark from unit tests (#4621)
* chore: fix lint and remove incorrect integration mark from unit tests

* add to test requirements

* revert athena source tests
2022-04-08 17:18:48 +02:00
Aseem Bansal
5ebb37ab4c
fix(bigquery): incorrect lineage when views are present (#4568)
* fix(bigquery): incorrect lineage when views are present

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-06 17:29:02 -07:00
Aseem Bansal
1478489c9c
fix(bigquery): missing dependency (#4567)
* fix(bigquery): missing dependency

* code review feedback
2022-04-04 08:34:34 -07:00
Aseem Bansal
40188d27c9
fix: dependencies for usage sources (#4561) 2022-04-01 09:56:52 -07:00
Corentin
2fc3a48bc5
feat(ingest): indent sql queries for usage sources (#3782)
* feat(ingest): indent sql queries for usage connectors.

Co-authored-by: EC2 Default User <ec2-user@ip-172-31-22-140.eu-west-1.compute.internal>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-03-31 15:15:09 -07:00
Sergio Gómez Villamor
bdf17f551e
feat(ingest): glue - adds platform instance capability (#4130) 2022-03-30 18:50:26 -07:00
MugdhaHardikar-GSLab
37aedfc87c
feat(s3): add s3 source (#4490)
* feat(data-lake): add containers and folder level dataset support

* docs(data-lake): Update readme for data lake

* doc(data-lake): fix examples, update doc

* lint fix

* feat(s3): add s3 source, restore old data-lake source

Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
2022-03-29 11:52:57 +02:00
Aseem Bansal
1a20f76225
fix(ingestion): pin Jinja2 to version < 3.1.0 (#4489) 2022-03-24 12:26:38 -07:00
Aseem Bansal
f770ed5fea
fix(ingestion): stop CLI build failures (#4484) 2022-03-24 17:03:56 +01:00
Sergio Gómez Villamor
9fbb521bfe
chore: acryl-data 0.6.12 (#4474) 2022-03-23 10:24:48 -07:00
pedro-iatzky
6a6d744667
fix(ingest): bigquery - fix ingestion of external tables (#4313) 2022-03-22 13:35:41 +01:00