578 Commits

Author SHA1 Message Date
Harshal Sheth
894e25680b
feat(ingest): add snowflake-summary source (#10642) 2024-06-12 10:04:22 -07:00
Andrew Sikowitz
46dbb10940
docs(ingest): Rename csv / s3 / file source and sink (#10675) 2024-06-11 11:44:13 -07:00
Eric L (CCCS)
c04b3bc2e4
fix(ingest/iceberg): update iceberg source to support newer versions of pyiceberg at runtime (#10614) 2024-06-04 09:45:29 -07:00
Harshal Sheth
2e14f70864
test(ingest/sql): refactor CLL generator + add tests (#10580) 2024-05-23 18:11:22 -07:00
Harshal Sheth
2b6c78b776
feat(ingest): bump acryl-sqlglot dep (#10554) 2024-05-21 23:52:33 +02:00
sagar-salvi-apptware
5fbf781558
fix(ingest/transformer): Add dataset domains based on tags using transformer (#10458) 2024-05-15 14:13:03 +05:30
skrydal
9debbdd4a9
fix(ingestion): Explicitly set requirement on snowflake-connector-python to be newer or equal to 3.4.0 (#10445)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-05-08 14:12:30 +05:30
Tamas Nemeth
897e648eae
fix(ingest/mode): Improve query lineage (#10284)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-05-07 22:02:37 -07:00
Mayuri Nehate
f6627efe71
fix(ingest/snowflake): add more reporting for usage aggregation, handle lineage errors (#10279)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-05-07 08:42:39 -07:00
dushayntAW
c00ddb2a0d
fix(ingest/transformer): new transformer to clean user URN for datasetUsageStatistics aspect (#10398) 2024-05-03 13:24:48 +05:30
mrjefflewis
e4cf4de3e0
feat(ingest/mssql): improve docs on using odbc (#10370)
Co-authored-by: Jeff Lewis <jeff.lewis@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-04-24 13:00:45 -05:00
Harshal Sheth
08731055ba
feat(ingest): bump acryl-sqlglot dep (#10343) 2024-04-20 08:37:22 +02:00
Harshal Sheth
d1cc0af314
feat(ingest/classify): add pip dependency (#10335) 2024-04-19 12:52:51 -07:00
david-leifker
25ba1e1a8b
chore(pyiceburg): set minimum version (#10318) 2024-04-17 11:47:13 -07:00
Harshal Sheth
3cdc462a7b
fix(ingest): disallow src.* imports, fix powerbi/sigma (#10292) 2024-04-16 15:04:51 -07:00
Tamas Nemeth
d463a16b49
chore(ingest/presto-on-hive): Renaming presto-on-hive to hive-metastore source (#10278) 2024-04-16 23:35:16 +02:00
Shubham Jagtap
90c1249e7d
feat(ingest/sigma): Sigma connector integration (#10037)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-04-15 20:18:31 -07:00
dushayntAW
f860f7907d
fix(ingest/transformer): replace externalUrl in dataset properties (#10281) 2024-04-15 20:14:42 +05:30
Tamas Nemeth
8ed87d6a90
feat(ingest/mode): Mode improvements (#10273) 2024-04-12 09:01:16 +02:00
Dotan Mor
fa0c1b3fa9
feat(ingest/cockroachdb): add cockroachdb ingestion (#10226) 2024-04-09 18:36:51 -07:00
Mayuri Nehate
0949d8ca8b
fix(ingest/databricks): pin pandas for databricks ingestion (#10204) 2024-04-04 09:36:44 -07:00
Harshal Sheth
db33c8646a
fix(ingest): add classification dep for dynamodb (#10162) 2024-04-02 09:28:43 -05:00
Harshal Sheth
61c21e1a73
feat(ingest): bump sqlglot dep (#10144)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2024-03-28 17:53:28 -07:00
Mayuri Nehate
0361f2463d
feat(ingest/dynamodb): add support for classification (#10138) 2024-03-27 11:28:58 -07:00
Harshal Sheth
e97e6822ad
feat(ingest): loosen pyarrow dep (#10141) 2024-03-26 12:39:57 -07:00
dushayntAW
dd502ae662
fix(ingest): added new transformer to cleanup suffix/prefix in owner URN (#10067) 2024-03-22 15:23:03 +05:30
Aseem Bansal
9659d60867
feat(ingest/datahub-gc): gc source to cleanup things (#10085) 2024-03-21 15:21:17 +05:30
Mayuri Nehate
77c72dad01
feat(ingest): add classification to bigquery, redshift (#10031) 2024-03-13 22:45:28 -07:00
Harshal Sheth
b0163c4885
feat(ingest): utilities for query logs (#10036) 2024-03-12 23:20:46 -07:00
Mayuri Nehate
2de0e62ac4
feat(ingest): add classification for sql sources (#10013)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-03-12 09:23:20 -07:00
skrydal
2265ae9257
feat(ingestion): Support for Server-less Redshift (#9998)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2024-03-12 09:32:20 +01:00
Harshal Sheth
b6956f9a5c
feat(ingest): update sqlglot fork (#10022) 2024-03-11 15:22:30 +01:00
Harshal Sheth
d987707cde
feat(ingest): speed up to_obj() and validate() (#9969) 2024-03-04 13:31:39 +01:00
Harshal Sheth
1bcc9a67eb
feat(ingest): loosen sentry requirements (#9936) 2024-02-27 12:36:24 -08:00
Harshal Sheth
02f41b74b6
feat(ingest): use uv for python package installs (#9885) 2024-02-26 15:02:47 -08:00
Shubham Jagtap
a1f2216da7
feat(ingest/qlik): Qlik cloud connector integration (#9682)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-02-26 11:27:37 -08:00
Harshal Sheth
b01416c11b
feat(ingest): add timeout to sqlglot parser (#9868) 2024-02-23 11:11:02 -08:00
Harshal Sheth
ef3a8148ec
fix(ingest): bump sqlglot + add some debug info to tests (#9867) 2024-02-21 13:07:59 -08:00
Harshal Sheth
24da7a573b
test(ingest): use pytest-random-order for unit tests (#9753) 2024-02-09 18:20:05 -05:00
Aseem Bansal
45511644e7
feat(ingest/slack): source to get user info from slack (#9776) 2024-02-08 14:05:26 +05:30
cburroughs
3b898dbae9
fix(ingest/snowflake): stop pinning transitive spacy dependency (#9768) 2024-02-05 08:28:24 -08:00
cburroughs
eb8017d6eb
feat(ingestion/feast): loosen feast dependency (#9769) 2024-02-02 14:47:26 -08:00
cburroughs
b4d70dd13f
chore(ingest/feast): drop redundant typeguard constraint (#9770) 2024-02-02 14:46:54 -08:00
Harshal Sheth
0e418b527e
fix(ingest): upgrade pytest-docker (#9765) 2024-02-01 16:33:15 -08:00
Harshal Sheth
1498c36875
chore(cli): drop support for python 3.7 (#9731) 2024-01-29 10:50:47 -08:00
Tamas Nemeth
5735eb3a55
fix(ingest/test): Fixing breaking change in moto 5.0 library (#9736) 2024-01-29 19:42:30 +05:30
Harshal Sheth
2bb4b73f98
fix(ingest/metabase): add missing sql parser dep (#9725) 2024-01-26 23:03:16 +01:00
Harshal Sheth
b94d463fe0
feat(ingest/lookml): support complex lookml manifests (#9688) 2024-01-22 16:23:56 -08:00
ethan-cartwright
dfb2f7e596
feat(classifier): Add support for excluding list of exact column names (#9472)
Co-authored-by: Ethan Cartwright <ethan.cartwright@acryl.io>
2024-01-16 19:12:23 -08:00
cburroughs
acfc315b57
build(ingest/feast): upgrade to latest feast version (0.35.x) (#9627) 2024-01-16 17:28:56 -08:00