519 Commits

Author SHA1 Message Date
Shuixi Li
f147b51fc8
feat(ingest): add preset source (#10954)
Co-authored-by: MARK CHENG <hcheng@wealthsimple.com>
Co-authored-by: hwmarkcheng <94201005+hwmarkcheng@users.noreply.github.com>
2024-10-09 20:27:31 -07:00
Andrew Sikowitz
3c1dcf99b0
fix(ingest/sqlglot): Make detach_ctes more robust (#11449) 2024-09-23 11:42:55 -07:00
Harshal Sheth
f4033707d4
chore(ingest): bump acryl-sqlglot (#11331) 2024-09-09 21:09:44 -07:00
sid-acryl
3150d90bd1
fix(ingestion/tableau): restructure the tableau graphql datasource query (#11230)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-09-09 10:45:06 -07:00
Harshal Sheth
5467481f16
fix(py): fix issues with AvroException (#11311) 2024-09-09 09:02:05 +02:00
Mayuri Nehate
1f3688a1ed
feat(ingest/databricks): include metadata for browse only tables (#10766)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-09-02 19:09:45 +05:30
Aseem Bansal
15c1cfc386
ci(build): update outdated action & pin deepdiff lib (#11260) 2024-08-28 19:53:42 +05:30
Felix Lüdin
ce99bc4f22
feat(ingest): add ingestion source for SAP Analytics Cloud (#10958)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-26 11:29:15 -07:00
david-leifker
94e7706e3b
chore(bump): bump azure-identity (#11235)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-26 09:44:08 -05:00
Mayuri Nehate
223650dd7a
feat(ingest): add bigquery-queries source (#10994)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-25 22:51:00 -07:00
Mayuri Nehate
9568a4254d
feat: separate great-expectations action package (#11096)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-21 12:13:36 -04:00
sid-acryl
627c5abfd6
feat(ingestion/bigquery): Add ability to filter GCP project ingestion based on project labels (#11169)
Co-authored-by: Alice Naghshineh <alice.naghshineh@nytimes.com>
Co-authored-by: Alice Naghshineh <45885699+anaghshineh@users.noreply.github.com>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
2024-08-20 14:42:00 -04:00
skrydal
b46a9632cb
fix(tests): Bump databricks-sdk dependency to >=0.30.0 (#11209) 2024-08-20 15:07:30 +01:00
sid-acryl
cb33c0fef7
feat(ingestion/lookml): support looker -- if comments (#11113) 2024-08-16 15:27:59 -04:00
Tamas Nemeth
5e9188ca2c
fix(ingest/databricks): Updating code to work with Databricks sdk 0.30 (#11158) 2024-08-13 16:57:31 +02:00
sid-acryl
b1f16f9b11
fix(ingestion/lookml): fix for sql parsing error (#11079)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-09 15:06:42 -07:00
Harshal Sheth
f78b6c08fb
fix(py): remove dep on types-pkg_resources (#11076) 2024-08-02 17:27:54 +05:30
Harshal Sheth
e83550ba35
feat(ingest/tableau): add retry on timeout (#10995) 2024-07-31 12:20:48 -07:00
sagar-salvi-apptware
da72ba2113
fix(ingestion/transformer): replace the externalUrl container (#11013) 2024-07-30 15:17:04 +05:30
Harshal Sheth
1fa7998ed3
feat(ingest): support domains in meta -> "datahub" section (#10967) 2024-07-25 09:31:19 -07:00
Tamas Nemeth
20574cf1c6
feat(ingest/athena): Add option for Athena partitioned profiling (#10723) 2024-07-20 00:00:40 +02:00
Tamas Nemeth
4fe5f280b3
fix(ingest/setup): feast and abs source setup (#10951) 2024-07-19 19:00:43 +05:30
Joel Pinto Mata (KPN-DSH-DEX team)
13b6febce9
feat(ingest/abs): Adding azure blob storage ingestion source (#10813) 2024-07-17 11:06:05 +02:00
Aseem Bansal
437bacb0e6
feat(ingest): grafana connector (#10891)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-07-15 14:12:18 -07:00
Harshal Sheth
a4bce6af1c
feat(ingest): add snowflake-queries source (#10835) 2024-07-12 15:08:51 -07:00
Harshal Sheth
89bda5bdd9
fix(ingest/redshift): handle multiline alter table commands (#10727) 2024-07-11 16:32:37 -07:00
Harshal Sheth
44930dfd1e
fix(ingest/looker): add missing dependency (#10876) 2024-07-11 13:25:13 -07:00
sid-acryl
609847fa59
fix(ingestion/looker): Add sqlglot dependency and remove unused sqlparser (#10874) 2024-07-09 12:51:08 -07:00
Tamas Nemeth
1c8e8c32b5
chore(ingest): Mypy 1.10.1 pin (#10867) 2024-07-08 19:43:45 +02:00
sid-acryl
43bac365bc
fix(ingestion/lookml): liquid template resolution and view-to-view cll (#10542) 2024-07-08 09:26:39 -07:00
Mayuri Nehate
54b9d98177
ci(ingest): pin dask dependency for feast (#10865) 2024-07-08 18:52:11 +05:30
cburroughs
c24d7805bd
chore(ingest): update acryl-datahub-classify version (#10844) 2024-07-03 16:32:46 -07:00
sagar-salvi-apptware
640d42dc65
feat(ingest/transformer): tags to terms transformer (#10758)
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2024-07-02 15:30:05 +05:30
Oleksandr Simonchuk
8b4e302881
feat(ingest): add and use file system abstraction in file source (#8415)
Co-authored-by: oleksandrsimonchuk <oleksandr.si@appsflyer.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2024-07-01 10:47:07 -07:00
Harshal Sheth
45a8cc9ecf
feat(ingest): bump sqlglot (#10770) 2024-06-26 14:31:57 -07:00
Harshal Sheth
0816e7590c
fix(ingest): pin numpy<2 for classification (#10725) 2024-06-17 22:50:08 +02:00
Rajasekhar-Vuppala
a2d8a099d8
feat(ingest/vertica): use 3 part naming (#10636) 2024-06-14 17:04:55 -07:00
Harshal Sheth
62c6704f69
feat(ingest/snowflake): refactor + parallel schema extraction (#10653) 2024-06-14 13:23:07 -07:00
sagar-salvi-apptware
d69966074a
fix(ingest/bigquery): Map BigQuery policy tags to datahub column-level tags (#10669) 2024-06-14 16:43:12 +05:30
Harshal Sheth
6329153e36
fix(ingest): fix redshift query urns + reduce memory usage (#10691) 2024-06-13 11:27:06 -07:00
Harshal Sheth
25d48d2d09
fix(ingest/fivetran): fix fivetran bigquery support (#10693) 2024-06-13 11:26:47 -07:00
Harshal Sheth
894e25680b
feat(ingest): add snowflake-summary source (#10642) 2024-06-12 10:04:22 -07:00
Andrew Sikowitz
46dbb10940
docs(ingest): Rename csv / s3 / file source and sink (#10675) 2024-06-11 11:44:13 -07:00
Eric L (CCCS)
c04b3bc2e4
fix(ingest/iceberg): update iceberg source to support newer versions of pyiceberg at runtime (#10614) 2024-06-04 09:45:29 -07:00
Harshal Sheth
2e14f70864
test(ingest/sql): refactor CLL generator + add tests (#10580) 2024-05-23 18:11:22 -07:00
Harshal Sheth
2b6c78b776
feat(ingest): bump acryl-sqlglot dep (#10554) 2024-05-21 23:52:33 +02:00
sagar-salvi-apptware
5fbf781558
fix(ingest/transformer): Add dataset domains based on tags using transformer (#10458) 2024-05-15 14:13:03 +05:30
skrydal
9debbdd4a9
fix(ingestion): Explicitly set requirement on snowflake-connector-python to be newer or equal to 3.4.0 (#10445)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-05-08 14:12:30 +05:30
Tamas Nemeth
897e648eae
fix(ingest/mode): Improve query lineage (#10284)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-05-07 22:02:37 -07:00
Mayuri Nehate
f6627efe71
fix(ingest/snowflake): add more reporting for usage aggregation, handle lineage errors (#10279)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-05-07 08:42:39 -07:00