3975 Commits

Author SHA1 Message Date
Andrew Sikowitz
2d1584b12f
feat(ingest/datahub-source): Allow ingesting aspects from the entitiesV2 API (#9089) 2023-10-24 23:08:24 -04:00
Harshal Sheth
916235d31a
fix(ingest): update athena type mapping (#9061) 2023-10-24 19:59:42 -07:00
skrydal
378d84a346
fix(ui): Fixes handling of resources filters in UI (#9087) 2023-10-24 21:12:11 +01:00
Harshal Sheth
d13553f53a
feat(sqlparser): extract CLL from updates (#9078) 2023-10-24 12:24:50 -07:00
Andrew Sikowitz
eb0b03d2f2
fix(ingest/bigquery): Correctly apply table pattern to read events; fix end time calculation; deprecate match_fully_qualified_names (#9077) 2023-10-24 10:45:09 -04:00
Harshal Sheth
c849246e63
fix(ingest): add releases link (#9014) 2023-10-24 09:09:41 +02:00
Andrew Sikowitz
adf8c8db38
refactor(ingest): Move sqlalchemy import out of sql_types.py (#9065) 2023-10-24 08:59:56 +02:00
Ellie O'Neil
10456c5e3c
docs(ingest): update to get_workunits_internal (#9054) 2023-10-23 14:53:07 -07:00
Harshal Sheth
8fb95e88a1
feat(sqlparser): parse create DDL statements (#9002) 2023-10-23 12:40:42 -07:00
Tamas Nemeth
04216e30bb
feat(ingest/s3): S3 add partition to schema (#8900)
Co-authored-by: Pedro Silva <pedro@acryl.io>
2023-10-21 16:21:54 +01:00
Pedro Silva
86e0023a4e
feat(ingestion): Adds more advanced configurations for runtime debugging (#8998) 2023-10-21 16:20:59 +01:00
Harshal Sheth
2fea466d48
docs: fix lineage capability annotations (#8954) 2023-10-20 13:47:52 -07:00
Andrew Sikowitz
35d7770462
test(ingest/delta-lake): Fix integration tests (#9056) 2023-10-20 01:40:23 -04:00
Tim
1eaf9c8c5f
feature(ingest/athena): introduce support for complex and nested schemas in Athena (#8137)
Co-authored-by: dnks23 <dominik.s23@live.de>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Tim <tim@MBP-von-Tim.fritz.box>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-18 09:39:59 -07:00
Andrew Sikowitz
d2eb42373f
fix(ingest/sqlalchemy): Fix URL parsing when sqlalchemy_uri provided (#9032) 2023-10-18 17:34:45 +02:00
VISHAL KUMAR
5937937819
feat(ingestion/Vertica): Fixed vertica integration test Updated vertica dialect (#9011) 2023-10-18 14:52:07 +05:30
Mayuri Nehate
9bb0bf25ae
fix(ingest): fix typo in parsing list of groups (#9037) 2023-10-18 14:09:27 +05:30
Andrew Sikowitz
da6cc54d63
fix(ingest/bigquery): Remove table name restrictions (allow $ and @) (#9030) 2023-10-17 16:18:39 -04:00
ethan-cartwright
ae5fd90c73
add create dataproduct example (#9009) 2023-10-17 20:25:07 +05:30
Tamas Nemeth
10eb205cb8
fix(ingest) - Fix file backed collection temp directory removal (#9027) 2023-10-17 10:16:25 -04:00
Andrew Sikowitz
9fec6024fb
feat(ingest/unity-catalog): Support external S3 lineage (#9025) 2023-10-17 09:58:38 -04:00
Andrew Sikowitz
6366b63e48
feat(ingest/teradata): Add option to not use file backed dict for view definitions (#9024) 2023-10-16 19:13:23 -04:00
Andrew Sikowitz
9ccd1d4f5d
build(ingest/redshift): Add sqlglot dependency (#9021) 2023-10-16 14:34:15 -04:00
Mayuri Nehate
c81a339bfc
build(ingest): remove ratelimiter dependency (#9008) 2023-10-16 09:27:57 -07:00
Tamas Nemeth
c2e8041d77
Adding missing sqlparser libs to setup.py (#9015) 2023-10-13 20:59:18 +00:00
Tamas Nemeth
1007204cda
feat(ingest/teradata): view parsing (#9005) 2023-10-13 15:07:19 -04:00
Andrew Sikowitz
6bc7425353
feat(cli/datacontract): Add data quality assertion support (#8968)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2023-10-13 12:36:18 -04:00
Tamas Nemeth
a8f0080c08
feat(ingest/teradata): Teradata source (#8977) 2023-10-12 15:14:45 -07:00
Pedro Silva
f6e1312063
feat(ingestion): Adds support for memory profiling (#8856)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-12 18:43:14 +01:00
Harshal Sheth
8813ae2fb1
feat(ingest/s3): support .gzip and fix decompression bug (#8990) 2023-10-12 13:58:10 +02:00
Tamas Nemeth
c381806110
feat(ingestion): Adding config option to auto lowercase dataset urns (#8928) 2023-10-12 13:56:30 +02:00
Tamas Nemeth
dd418de76d
fix(ingest/bigquery): Fix shard regexp to match without underscore as well (#8934) 2023-10-12 13:10:59 +02:00
Harshal Sheth
84bba4dc44
feat(ingest): add output schema inference for sql parser (#8989) 2023-10-11 22:31:17 -07:00
Sergio Gómez Villamor
245c5c0008
fix(ingest/looker): stop emitting tag owner (#8942) 2023-10-11 17:06:19 -07:00
Mayuri Nehate
1b06c6a30c
fix(ingest/snowflake): fix sample fraction for very large tables (#8988) 2023-10-11 12:01:42 -07:00
Harshal Sheth
932fbcddbf
refactor(ingest/dbt): move dbt tests logic to dedicated file (#8984) 2023-10-11 11:17:02 -07:00
Harsha Mandadi
4b6b941a2a
fix(ingest): Fix postgres lineage within views (#8906)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Maggie Hays <maggiem.hays@gmail.com>
2023-10-11 14:01:46 -04:00
siddiquebagwan-gslab
10a190470e
feat(ingestion/redshift): CLL support in redshift (#8921) 2023-10-10 20:24:08 -07:00
Andrew Sikowitz
1a72fa499c
feat(ingest/tableau): Allow parsing of database name from fullName (#8981) 2023-10-10 17:34:06 -04:00
Mayuri Nehate
57f855ecd1
feat(ingest): refactor + simplify incremental lineage helper (#8976) 2023-10-09 23:48:21 -07:00
Mayuri Nehate
8d175ef7ef
feat(ingest): incremental lineage source helper (#8941)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-09 13:34:25 -07:00
Mayuri Nehate
8e7f286e71
feat(ingest/snowflake): support profiling with sampling (#8902)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-10-06 10:42:39 -04:00
Andrew Sikowitz
c80da8f949
feat(ingest/unity): Use ThreadPoolExecutor for CLL (#8952) 2023-10-06 10:06:36 -04:00
Andrew Sikowitz
26bc039b96
fix(ingest/unity): Remove metastore from ingestion and urns; standardize platform instance; add notebook filter (#8943) 2023-10-05 23:23:15 -04:00
Harshal Sheth
3cede10ab3
feat(ingest/dbt): support use_compiled_code and test_warnings_are_errors (#8956) 2023-10-05 10:29:47 -07:00
Jinlin Yang
6310e51eb0
feat(ingestion/dynamodb): implement pagination for list_tables (#8910) 2023-10-05 09:33:31 +05:30
Mayuri Nehate
2fcced6db9
docs(ingest): add permissions required for athena ingestion (#8948) 2023-10-05 09:31:57 +05:30
Aseem Bansal
2bc685d3b9
ci: tweak ci to decrease wait time of devs (#8945) 2023-10-05 09:31:32 +05:30
Harshal Sheth
817c371fbf
feat: data contracts models + CLI (#8923)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: John Joyce <john@acryl.io>
2023-10-04 20:11:06 -07:00
ethan-cartwright
e2afd44bfe
feat(dbt-ingestion): add documentation link from dbt source to institutionalMemory (#8686)
Co-authored-by: Ethan Cartwright <ethan.cartwright@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-04 20:38:58 +00:00