588 Commits

Author SHA1 Message Date
Kevin Hu
1a2f75b1e6
fix(ingest): data-lake - add aws dependencies (#4060) 2022-02-04 21:12:09 -08:00
Swaroop Jagadish
aab622793d
feat(ci): pin tox requirements to speed up ci runs, remove airflow-1 suite until we can pin it (#4055) 2022-02-04 07:32:38 -08:00
Swaroop Jagadish
ded16809da
feat(ingest): add tests for platform instance (#4047) 2022-02-02 22:52:50 -08:00
Ravindra Lanka
f20382f956
feat(ingest): framework - client side changes for monitoring and reporting (#3807) 2022-02-02 13:19:15 -08:00
mayurinehate
faac911df8
feat(ingest): use trino python client with sqlalchemy, supports python3.6 (#3888) 2022-01-31 09:22:55 -08:00
Aseem Bansal
7180ae5ff6
fix(ci): fix linting lib version requirement (#3939) 2022-01-30 17:20:46 -08:00
Michael A. Schlosser
c36662f837
feat(ingest): snowflake - support for additional auth mechanisms (#4009) 2022-01-30 11:47:53 -08:00
Tamas Nemeth
cec541d827
fix(ci): ingestion - Restrict python to <=3.9.9 due to upstream looker sdk issues (#3961)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-25 19:11:53 -08:00
Ravindra Lanka
7eda2bf144
fix(ingest): snowflake_usage - emit usage workunits when include_operational_stats is enabled. (#3949) 2022-01-21 17:16:03 -08:00
Kevin Hu
9e30b42638
feat(ingest): data lake - support for profiling data lake assets (#3656) 2022-01-17 15:01:23 -08:00
Ravindra Lanka
a44b48a6b8
feat(ingest): elasticsearch - add Elasticsearch Source (#3893)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-14 13:10:12 -08:00
varunbharill
cc52e3e759
fix(ingest): bigquery - update dependencies for module. (#3874)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-13 22:12:11 -08:00
Aditya Radhakrishnan
c3e98645a2
feat(ingest): operational stats - show last updated for sql usage sources (#3845) 2022-01-07 17:27:09 -08:00
Tamas Nemeth
e5d8313788
feat(ingest): sql parsing - Replacing sqlmetadata sql parser lib with sqlineage parser lib (#3806) 2022-01-05 12:22:59 -08:00
Hyun Min Choi
e76d40b57a
fix(lkml): bump lkml version up to 1.1.2 to support sql_preamble expression (#3757) 2021-12-21 12:13:13 -08:00
Aseem Bansal
77e3641a50
fix(ingest): fix compatibility with google composer (#3774) 2021-12-20 12:47:56 -08:00
Ravindra Lanka
bd69e736ba
feat(Stateful Ingestion-2/3): Client side changes for checkpointing a source job state. (#3763)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-12-16 20:06:33 -08:00
Harshal Sheth
c6f3ddf077
build(ingest): restrict latest mypy version (#3756) 2021-12-16 08:48:15 -08:00
Sergio Gómez Villamor
c59c63e90d
feat: enables dbt metadata files to be loaded from URIs (#3739) 2021-12-15 09:11:39 -08:00
jawadqu
578590e795
feat(ingestion) : Add Metabase Source Connector (#3602)
Co-authored-by: Jawad Qureshi <jqureshi@petabloc.com>
2021-12-13 23:02:47 -08:00
Gabe Lyons
8394fc62b0
feat(mode): add mode analytics ingestion source (#3710) 2021-12-09 16:10:08 -08:00
Gabe Lyons
0fdd3352bd
feat(ingestion): Add lineage support for Redshift source (#3697)
Co-authored-by: treff7es <treff7es@gmail.com>
2021-12-08 23:41:18 -08:00
mayurinehate
1d7ec8dba8
feat(ingest): add nifi source (#3681) 2021-12-08 14:56:31 -08:00
Swaroop Jagadish
a16c432a1b
feat(metadata-model): adding metadata model doc generation and upload… (#3667) 2021-12-05 12:22:17 -08:00
Harshal Sheth
7d5da17d88
fix(ingest): workaround great-expectations compatibility issue (#3634) 2021-11-29 17:14:19 -08:00
Swaroop Jagadish
2fd919cb4d
fix(ci): pin great expectations version (#3629) 2021-11-28 17:28:57 -08:00
Harshal Sheth
cb2a3e6927
feat(profiler): dynamically combine queries (#3572) 2021-11-24 10:19:03 -08:00
Tamas Nemeth
a36fefaa33
feat(ingest): Starburst Trino usage (#3558) 2021-11-18 09:56:24 -08:00
Vincenzo Lavorini
6cd31b5bdc
feat(ingest): adding openapi ingestion (#2706) 2021-11-09 21:22:15 -08:00
Harshal Sheth
8a432898ba
fix(ingest): denylist botocore 1.23.0 (#3534) 2021-11-08 16:23:25 -08:00
Swaroop Jagadish
3984506f51
feat(cli): add support for deletion based on filters, soft deletes an… (#3527) 2021-11-07 22:13:50 -08:00
mayurinehate
192f0d33a2
feat(ingest): kafka connect source improvements (#3481) 2021-11-03 15:03:05 -07:00
Ravindra Lanka
a1bf95307b
feat(ingest): user configurable dataset profiling. (#3453) 2021-10-27 19:49:40 -07:00
varunbharill
7e27355837
fix(ingest): Add gcp logging dependency for bigquery source. (#3451)
fixes #3450
2021-10-25 17:50:07 -07:00
Remi
91d20e41c0
fix(ingest): fix lookml derived tables parsing (#3443) 2021-10-25 14:08:30 -07:00
Harshal Sheth
6e5d1fe42f
fix(ingest): switch to avro from deprecated avro-python3 (#3412) 2021-10-18 15:35:27 -07:00
aseembansal-gogo
ebe0b3f7c1
feat(ingest): add mariadb as possible source (#3245) 2021-10-12 23:27:47 -07:00
mayurinehate
d22d1c2795
feat(ingest): add trino source to datahub (#3307) 2021-10-06 11:57:06 -07:00
taufiqibrahim
ed38168f54
feat(ingest): add support for parsing lineage for redash connector (#3267)
Co-authored-by: tibrahim <taufiq.ibrahim@bizzy.co.id>
2021-10-05 23:08:02 -07:00
Chinmay Bhat
5a239905df
feat(ingest): add redshift usage source (#3277) 2021-09-23 13:39:33 -07:00
Swaroop Jagadish
9db7a34cfa
feat(ingest): looker and lookml improvements - connection, explores, folders (#3255) 2021-09-16 23:09:45 -07:00
Chinmay Bhat
9014afd9ed
refactor(ingest): rename azure source to azure_ad (#3181) 2021-09-02 09:24:10 -07:00
Shirshanka Das
88a3523fbb
feat(ingest): Add a business glossary source (#3164) 2021-09-01 15:10:12 -07:00
Chinmay Bhat
bda881cbf2
feat(ingest): Adding Azure Source integration to ingest users, groups and group memberships (#3161) 2021-08-29 08:32:40 -07:00
Swaroop Jagadish
65943cc190
deps(ingest): upgrade to pick up acryl-pyhive changes (#3145) 2021-08-23 20:18:11 -07:00
Swaroop Jagadish
72af0c317a
fix(deps): Adding min version to python-dateutil to guard against isoparse failures (#3139) 2021-08-21 18:41:28 -07:00
taufiqibrahim
dd1b81b727
feat(ingest): adding redash source (#2918)
Co-authored-by: tibrahim <taufiq.ibrahim@bizzy.co.id>
2021-08-18 12:03:03 -07:00
John Joyce
8c9c696cdd
feat(ingest): Adding an Okta Integration to extract Users, Groups, Group Membership (#3043) 2021-08-11 18:49:16 -07:00
Kevin Hu
f642d6c0be
fix(ingest): sqlalchemy-snowflake add constraints to make sure we don't pull in 1.2.5 (#3078) 2021-08-11 13:20:01 -07:00
Chinmay Bhat
7abcb45f8c
feat(ingest): add support for jdbc connector to kafka-connect source (#3076) 2021-08-11 08:50:38 -07:00