187 Commits

Author SHA1 Message Date
Aseem Bansal
b5e1ed739c
fix(bigquery): add rate limiting for api calls made (#4967) 2022-05-23 20:14:36 +05:30
Aseem Bansal
5ac6ba58f8
fix(build): Fix breaking changes from GE 0.15.3 (#4905) 2022-05-12 08:46:23 -07:00
Tamas Nemeth
84f71dadf0
fix(ingestion): dependencies - Downgrading typing-extension dependency to work with Airflow 2.0.2 (#4855)
* Downgrading typing-extension dependency to work with Airflow 2.0.2 restricting typing-extension on python 3.7
2022-05-09 09:43:49 -07:00
Ravindra Lanka
842fb391eb
feat(ingestion): kafka - add protobuf schema support (#4819)
Co-authored-by: Luis Angel Vicente Sanchez <luis.vicentesanchez@aaqua.live>
2022-05-04 17:07:01 -07:00
Shirshanka Das
c15b44a1de
fix(ingest): databricks - hive ingestion should not fail on table comment (#4787) 2022-04-29 13:56:42 -07:00
vanmeete
74d6d35881
feat(ingestion): add Pulsar source (#4721) 2022-04-29 15:57:02 +05:30
Ravindra Lanka
8583f397f1
Fix breaking changes from GE 0.15.3 that are affecting our Python3.6 smoke_tests. (#4779) 2022-04-28 15:50:17 -07:00
Danilo Peixoto
d2a6bc06dc
feat(ingest): feast - add support for Feast 0.18, deprecate older integration (#4094) 2022-04-26 14:35:02 -07:00
Aseem Bansal
53d357b4eb
fix(bigquery-usage): missing dependency (#4661) 2022-04-13 14:29:31 +02:00
jchen0824
524d183d93
feat: add presto-on-hive metadata ingestion source (#4625)
* feat(metadata ingestion source): add presto-on-hive metadata ingestion source

Co-authored-by: Houren Chen <houren.chen@grabtaxi.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-11 17:46:44 -07:00
Tamas Nemeth
e2a617f183
Restricting pytest docker version (#4639) 2022-04-11 23:35:34 +02:00
Aseem Bansal
61a95f41ae
chore: fix lint and remove incorrect integration mark from unit tests (#4621)
* chore: fix lint and remove incorrect integration mark from unit tests

* add to test requirements

* revert athena source tests
2022-04-08 17:18:48 +02:00
Aseem Bansal
5ebb37ab4c
fix(bigquery): incorrect lineage when views are present (#4568)
* fix(bigquery): incorrect lineage when views are present

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-06 17:29:02 -07:00
Aseem Bansal
1478489c9c
fix(bigquery): missing dependency (#4567)
* fix(bigquery): missing dependency

* code review feedback
2022-04-04 08:34:34 -07:00
Aseem Bansal
40188d27c9
fix: dependencies for usage sources (#4561) 2022-04-01 09:56:52 -07:00
Corentin
2fc3a48bc5
feat(ingest): indent sql queries for usage sources (#3782)
* feat(ingest): indent sql queries for usage connectors.

Co-authored-by: EC2 Default User <ec2-user@ip-172-31-22-140.eu-west-1.compute.internal>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-03-31 15:15:09 -07:00
Sergio Gómez Villamor
bdf17f551e
feat(ingest): glue - adds platform instance capability (#4130) 2022-03-30 18:50:26 -07:00
MugdhaHardikar-GSLab
37aedfc87c
feat(s3): add s3 source (#4490)
* feat(data-lake): add containers and folder level dataset support

* docs(data-lake): Update readme for data lake

* doc(data-lake): fix examples, update doc

* lint fix

* feat(s3): add s3 source, restore old data-lake source

Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
2022-03-29 11:52:57 +02:00
Aseem Bansal
1a20f76225
fix(ingestion): pin Jinja2 to version < 3.1.0 (#4489) 2022-03-24 12:26:38 -07:00
Aseem Bansal
f770ed5fea
fix(ingestion): stop CLI build failures (#4484) 2022-03-24 17:03:56 +01:00
Sergio Gómez Villamor
9fbb521bfe
chore: acryl-data 0.6.12 (#4474) 2022-03-23 10:24:48 -07:00
pedro-iatzky
6a6d744667
fix(ingest): bigquery - fix ingestion of external tables (#4313) 2022-03-22 13:35:41 +01:00
Kevin Neville
d8e6f890a9
fix: Replace old repository link with new link (#4446) 2022-03-18 14:12:19 -07:00
Hassan Shahid
eb9a167e0d
build(ingestion): update markupsafe pinning for Airflow compatibility (#4388) 2022-03-14 16:19:59 -07:00
Ravindra Lanka
5aaf187371
fix(ingestion): Fix mypy error stateful committable & restore mypy version. (#4408) 2022-03-14 14:15:14 -07:00
Aseem Bansal
4bcc2b3d12
feat(ingestion): improve logging, docs for bigquery, snowflake, redshift (#4344) 2022-03-14 08:50:29 -07:00
mayurinehate
e19b12030e
fix(GE): fix dependencies for GE DataHubValidationAction, logic for s… (#4347) 2022-03-08 11:20:09 -08:00
John Joyce
9f1c5a8f75
feat(assertions): Adding Assertions Entity & Great Expectations BETA (#4305) 2022-03-04 11:51:31 -08:00
Tamas Nemeth
3158e49171
fix(ingest): Removing python 3.9.9 restriction (#4312) 2022-03-04 08:21:33 -08:00
Aseem Bansal
d52638a252
fix(ingest): lib - revert positional arg change, create new method (#4266)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-03-01 22:06:37 -08:00
mohdsiddique
e2f8db7926
feat: powerbi - add new source (#4201) 2022-02-28 17:37:22 -08:00
Kevin Hu
02fe05eb8f
feat(ingest): data-lake - remove spark requirement if not profiling (#4131) 2022-02-24 23:26:06 -08:00
Ravindra Lanka
1ac20ee764
feat(platform): add timeline api for dataset entity (#4252) 2022-02-24 19:40:55 -08:00
Edward Vaisman
6ff551cbcd
feat(ingest): lineage-file - add ability to provide lineage manually through a file (#4116) 2022-02-24 17:02:38 -08:00
Kevin Hu
46701319dc
feat(ingest): switch telemetry endpoint to Mixpanel (#4238) 2022-02-24 12:35:48 -08:00
Harshal Sheth
49a8ece02a
fix(ingestion): enable compat with avro 1.11 (#4205) 2022-02-22 22:13:50 -08:00
Swaroop Jagadish
c6cb549918
fix(ingest): superset - adding missing greenlet dep (#4203) 2022-02-21 09:07:44 -08:00
Alexander Chashnikov
c2065bd7fe
feat(ingest): clickhouse - add initial support (#4057)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-21 07:36:08 -08:00
Harshal Sheth
1b60fae014
test(airflow): fix airflow version parsing (#4142)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-19 18:13:01 -08:00
Sunil Patil
be64092c89
fix(ingest): elasticsearch - pinning down elasticsearch dependency to less than 8.0.0 (#4182)
Co-authored-by: Sunil Patil <spatil@twilio.com>
2022-02-19 11:44:32 -08:00
Swaroop Jagadish
eaf7b02b2a
docs(model): auto-generated docs and hand-written docs for the metadata model (#4189) 2022-02-18 09:45:45 -08:00
Tamas Nemeth
8cb312c4be
fix(ci): pinning markup_safe (#4188) 2022-02-18 08:53:28 -08:00
Harshal Sheth
75d2ec2a39
ci(ingestion): fix airflow 1 deps for tox (#4083) 2022-02-17 00:33:28 -08:00
Harshal Sheth
6f37ddfee2
chore(ingestion): upgrade mypy (#4141) 2022-02-17 00:31:20 -08:00
Tamas Nemeth
bfaec300b6
feat(ingest) Athena: Getting table properties for Athena datasets (#4123)
* Getting table properties for Athena datasets

* Isorting

* Fixing mypy error

* Addressing pr review comments
Adding tests

* Adding missing import

* black

* Fixing test run

* fixing flake8

* Adding athen to tox tests as well

* Not running athena tests on python < 3.7

* Adressing more pr comments
2022-02-14 13:51:45 -08:00
Aditya Radhakrishnan
b331106dc7
fix(ingest): okta - better use of asyncio and additional debug logging (#4077) 2022-02-10 20:02:23 -08:00
Dexter Lee
47db7b5e2c
fix(ingest): dependencies - fix for redshift-usage, mode, superset and others (#4103) 2022-02-09 23:12:41 -08:00
Ravindra Lanka
917720989a
fix(ingest): dependencies - Add psutil dependency for stateful ingestion reporting. (#4099) 2022-02-08 23:25:12 -08:00
John Joyce
2a9a076fc1
feat(ingest): Adding Tableau Source Connector [BETA] (#4063) 2022-02-08 14:26:44 -08:00
Kevin Hu
1a2f75b1e6
fix(ingest): data-lake - add aws dependencies (#4060) 2022-02-04 21:12:09 -08:00