578 Commits

Author SHA1 Message Date
Harshal Sheth
f0ea79060b
chore(ingest): bug fix in sqlparse pin (#7848) 2023-04-18 16:05:23 -07:00
Harshal Sheth
cf7eb570a0
fix(ingest): pin sqlparse version (#7847) 2023-04-18 14:25:42 -07:00
Andrew Sikowitz
1ac1ccf26e
perf(ingest/bigquery): Improve bigquery usage disk usage and speed (#7825) 2023-04-14 18:09:43 -07:00
Tamas Nemeth
0cc12bcce7
feat(ingest): redshift - Redshift rework (#6906) 2023-04-12 19:15:43 +02:00
Mayuri Nehate
ec1228f67d
fix(dep): add sqllineage dependency for tableau (#7803) 2023-04-12 15:33:31 +02:00
Harshal Sheth
e99875cac6
chore(ingest): enable flake8 bugbear linting (#7763) 2023-04-10 14:14:42 -07:00
Harshal Sheth
89734587f7
feat(ingest): add urn modification helper (#7440)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-03-16 13:27:08 -07:00
Harshal Sheth
b87e4c312a
fix(ingest): pin typeguard version for feast (#7591) 2023-03-15 13:27:20 +05:30
Mayuri Nehate
dac3938077
chore(ingest): snowflake - bump up classification library version to 0.0.6 (#7542) 2023-03-12 10:20:32 -07:00
J Feldman
aa4228734c
feat(ingest/looker): upgrade to Looker API from 3.1 to 4.0 (#7411)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2023-03-07 16:49:37 -08:00
Mayuri Nehate
dc2a7d8a46
chore(ingest): remove unused dependency for bigquery (#7510)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-03-07 16:43:14 -08:00
cburroughs
cc0772f8d8
feat(ingest): unbundle airflow plugin emitter dependencies (#7493)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-03-07 09:07:42 -08:00
Harshal Sheth
b4dd1d7d82
chore(ingest): pin acryl-datahub-classify (#7485) 2023-03-03 11:36:20 -05:00
Aseem Bansal
1adbc2cab0
chore(ci): upgrade GE version (#7290) 2023-03-02 10:47:38 -08:00
Aseem Bansal
f8a73005d4
chore(ci): relax bigquery dependency (#7309) 2023-02-21 08:33:00 +01:00
Tamas Nemeth
097d4e6bbd
fix(dep/json-schema): Fixing json-schema dependencies (#7383) 2023-02-20 14:02:08 -08:00
Shirshanka Das
07e4d0696f
feat(ingest): json-schema - add json schema support for files and kaf… (#7361) 2023-02-19 08:43:13 -08:00
Andrew Sikowitz
a605f0752f
fix(deps): pin snowflake-connector-python (#7365)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-02-18 10:44:55 +01:00
Harshal Sheth
582fdf95cd
chore(ingest): upgrade to mypy 1.0.0 (#7313) 2023-02-10 13:24:05 -08:00
Tamas Nemeth
793f303a79
fix(ingest/bigquery): Lowering significantly the memory usage of the BigQuery connector (#7315) 2023-02-10 13:12:02 -08:00
Harshal Sheth
55442042ff
feat(cli): improve startup time (#7292) 2023-02-10 21:36:01 +05:30
Harshal Sheth
e3af6168d3
fix(ingest): upgrade feast to avoid build issues (#7218) 2023-02-02 15:24:28 +01:00
david-leifker
39920bb00f
feat(elasticsearch): Elasticsearch improvements (#6894) 2023-01-31 18:44:37 -06:00
Patrick Franco Braz
8ee9fa1930
feat(ingest): bigquery - extracts lineage metadata from catalog api (#7137) 2023-01-31 15:02:30 +01:00
Harshal Sheth
927d45dda9
feat(ingest): add --log-file option and show CLI logs in UI report (#7118)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-01-26 09:25:02 -08:00
Harshal Sheth
54c5017efd
feat(ingest): move datahub-lite to optional dep and add shim when missing (#7097) 2023-01-20 17:24:43 -08:00
Harshal Sheth
13cc16fbc2
fix(cli/lite): fix datahub lite serve command (#7089) 2023-01-20 10:21:24 +01:00
Shirshanka Das
bdcc356cc5
feat(datahub-lite): introduces a new experimental lightweight impleme… (#7052) 2023-01-18 19:18:56 -08:00
Harshal Sheth
890dae0199
fix(ingest): temporarily disable vertica tests (#7059) 2023-01-17 12:37:16 -08:00
Rajasekhar-Vuppala
cd9fc26a25
feat(ingest/vertica): Adding Vertica as source in Datahub UI (#7010)
Co-authored-by: Vishal <vishal.k@simplify3x.com>
Co-authored-by: VISHAL KUMAR <110387730+vishalkSimplify@users.noreply.github.com>
Co-authored-by: John Joyce <john@acryl.io>
2023-01-13 13:23:32 -08:00
Harshal Sheth
211c30fe30
fix(ingest): add missing dep for powerbi (#6969) 2023-01-06 18:16:32 -05:00
VISHAL KUMAR
96ac4c431f
feat(ingest/vertica): support projections and lineage in vertica (#6785)
Co-authored-by: mraman2512 <MY_mramaan2512@gmail.com>
Co-authored-by: Aman.Kumar <64635307+mraman2512@users.noreply.github.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-01-06 16:20:19 -05:00
Aseem Bansal
d55ad6ca14
fix(ci): restrict GE to fix build issues (#6967) 2023-01-06 18:25:36 +05:30
Harshal Sheth
9bb1c155bd
chore(ingest): partially revert pyspark dep from #6908 (#6954) 2023-01-04 16:51:44 -08:00
Harshal Sheth
e97903f7f6
chore(ingest): unpin pydantic dep (#6909) 2023-01-04 16:31:04 -08:00
mohdsiddique
54ea8244de
feat(ingestion): PowerBI# Improve PowerBI source ingestion (#6549)
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
2023-01-03 08:08:11 -08:00
Harshal Sheth
b9677229a1
chore(ingest): loosen pyspark and pydeequ deps (#6908) 2022-12-30 20:53:38 +01:00
Tamas Nemeth
ead0074169
deprecate(ingest): bigquery - Removing bigquery-legacy source (#6851)
Co-authored-by: John Joyce <john@acryl.io>
2022-12-29 13:19:05 -08:00
Aseem Bansal
b8664d6630
fix(lint): pin pydantic version (#6886) 2022-12-29 19:36:14 +05:30
cccs-eric
ec8a4e0eab
feat(ingest): upgrade pydantic version (#6858)
This PR also removes the requirement on docker-compose v1 and makes our tests use v2 instead.

Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-12-27 17:06:16 -05:00
Tamas Nemeth
a1970d2dce
feat(ingest/bigquery): add option to enable/disable legacy sharded table support (#6822)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: John Joyce <john@acryl.io>
2022-12-20 23:29:46 -05:00
Harshal Sheth
14a00f4098
chore(ingest): pin black version (#6807) 2022-12-19 19:35:49 +01:00
Harshal Sheth
7d63399d00
fix(ingest): fix serde for empty dicts in unions with null (#6745)
The code changes in https://github.com/acryldata/avro_gen/pull/16, but tests are written here.
2022-12-13 08:17:24 +01:00
Felix Lüdin
05e18a0ae7
feat(ingest): use entry point for registering transformers (#6628) 2022-12-07 23:08:08 -05:00
Mayuri Nehate
eeb7a9dfe5
feat(ingest): snowflake - update snowflake docs, add simple validations (#6636) 2022-12-07 14:56:03 +01:00
Harshal Sheth
fceef480a2
chore(ingest): remove feast-legacy (#6661) 2022-12-06 14:19:38 -08:00
Harshal Sheth
71bfa98f89
fix(ingest): fix lingering demo-data source issues (#6659) 2022-12-06 16:10:21 -05:00
Aseem Bansal
43c566ee4f
feat(ingest): add dummy data source for automated testing (#6550) 2022-12-06 16:57:12 +05:30
david-leifker
2de9d3d5bf
fix(logging): Remove lombok as source of slf4j-api, convert to compileOnly where possible (#6616) 2022-12-04 19:57:47 -08:00
Harshal Sheth
a1e62c723e
docs(ingest): add airflow docs that use the PythonVirtualenvOperator (#6604) 2022-12-02 19:56:17 +01:00