3904 Commits

Author SHA1 Message Date
Mayuri Nehate
837d00d391
fix(ingest/bq): fix ordering of queries for use_queries_v2 (#11333) 2024-09-10 12:17:23 -07:00
sagar-salvi-apptware
852a23b845
feat(mode/ingest): Add support for missing Mode datasets in lineage (#11290) 2024-09-10 10:24:55 -07:00
Harshal Sheth
f082287a07
docs(ingest/dbt): add docs on hiding sources (#11334) 2024-09-10 10:13:25 -07:00
Harshal Sheth
f4033707d4
chore(ingest): bump acryl-sqlglot (#11331) 2024-09-09 21:09:44 -07:00
Harshal Sheth
0d4eda6115
fix(ingest): avoid sqlite "too many SQL variables" error (#11332) 2024-09-09 20:16:24 -07:00
Mayuri Nehate
cf49f80e77
feat(ingest/sql): auto extract and use mode query user metadata (#11307)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-09-09 12:38:24 -07:00
Harshal Sheth
28310bb64b
feat(ingest): support full urns without owner_type in meta mapping (#11298)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2024-09-09 11:18:07 -07:00
Harshal Sheth
55a56be6b0
fix(ingest): disable reporting for dry-run pipelines (#11306) 2024-09-09 11:16:56 -07:00
sid-acryl
3150d90bd1
fix(ingestion/tableau): restructure the tableau graphql datasource query (#11230)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-09-09 10:45:06 -07:00
Harshal Sheth
5467481f16
fix(py): fix issues with AvroException (#11311) 2024-09-09 09:02:05 +02:00
Harshal Sheth
9b64b117c7
docs(ingest/dbt): clarify dbt ingestion docs (#11312) 2024-09-09 09:01:46 +02:00
Harshal Sheth
d0bad4a7ee
feat(ingest): make rest emitter version error messages more clear (#11295) 2024-09-08 11:07:18 -07:00
Harshal Sheth
607ad5e1bc
feat(cli): reject missing urns in datahub get (#11313) 2024-09-06 15:32:25 -07:00
Harshal Sheth
9589adebb3
feat(ingest): enable query usage stats by default (#11281) 2024-09-06 10:02:40 -07:00
Rohan Sunkari
c3a7850a11
feat(ingest/sql): add default dialect support to SqlQueriesSource (#11285) 2024-09-05 13:26:11 -07:00
Mayuri Nehate
d788cd753c
feat(ingest/salesforce): helpful error messages on failure (#11266) 2024-09-05 09:47:26 -07:00
Tamas Nemeth
9c7f62a779
feat(ingest/athena): Add option to disable partition extraction (#11286) 2024-09-04 15:01:03 -07:00
david-leifker
ede9520b87
feat(schemaField): populate schemaFields with side effects (#10928) 2024-09-04 15:36:12 -05:00
Harshal Sheth
401b787427
fix(ingest): add custom StrEnum type (#11270) 2024-09-04 08:09:36 -07:00
Mayuri Nehate
08a5956e26
fix(bigquery): followups on bigquery queries v2 integration (#11291) 2024-09-04 07:42:35 -07:00
sleeperdeep
e34a628afe
fix(ingest/mssql): remove lower() method from sql_common get_db_name (#10773) 2024-09-03 12:41:23 -07:00
Harshal Sheth
96b075d79f
feat(ingest): add python deps for apk (#11188) 2024-09-03 10:53:44 -07:00
Mayuri Nehate
1f3688a1ed
feat(ingest/databricks): include metadata for browse only tables (#10766)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-09-02 19:09:45 +05:30
Mayuri Nehate
a7fc7f519a
feat(ingest/bq): integrate bigquery-queries into main source (#11247) 2024-08-30 18:16:45 -07:00
matthew-coudert-cko
c513e17dbb
feat(ingest/fivetran): support filtering on destination ids (#11277) 2024-08-30 13:25:18 -07:00
Mayuri Nehate
3e5c18f5d2
fix(ingest): limit number of upstreams generated by sql parsing aggre… (#11267)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-30 13:20:18 -07:00
Andrew Sikowitz
fa1164aa63
feat(ingest/s3): Support reading S3 file type (#11177)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2024-08-30 12:15:12 +02:00
Alisa Aylward
675b225c5b
feat(ingest/snowflake): Add cluster formula to dataset properties (#11254) 2024-08-29 16:11:35 -07:00
dushayntAW
6204cba2bb
fix(ingestion/prefect-plugin): Prefect plugin (#10643)
Co-authored-by: shubhamjagtap639 <shubham.jagtap@gslab.com>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2024-08-29 15:40:10 +02:00
Aseem Bansal
efc42df3d9
ci: update outdated actions for java and python setup (#11261) 2024-08-29 14:27:35 +05:30
Aseem Bansal
15c1cfc386
ci(build): update outdated action & pin deepdiff lib (#11260) 2024-08-28 19:53:42 +05:30
Tamas Nemeth
1893b7b2dc
feat(ingest/dagster): Add automatic snowflake_pandas_io_manager asset capture (#11189)
Co-authored-by: shubhamjagtap639 <shubham.jagtap@gslab.com>
Co-authored-by: John Joyce <john@acryl.io>
2024-08-28 09:22:11 +02:00
Mayuri Nehate
dc88d11fa3
docs: fix great-expectations doc module_name (#11253) 2024-08-27 20:31:29 -07:00
Harshal Sheth
8f88858eef
fix(ingest/kafka): update warning reporting for kafka (#11171) 2024-08-27 09:42:26 -07:00
Alisa Aylward
ca2e134d34
fix(ingest/snowflake): exclude snowflake excluded tags (#11250) 2024-08-27 08:48:59 -07:00
Jay Feldman
56a563b8c5
feat(ingest/bigquery): Add query job retries for transient errors (#11162)
Co-authored-by: Gabe Lyons <gabe.lyons@acryl.io>
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2024-08-26 21:19:32 -07:00
Peter Esselius
414dc54b13
feat(ingest/superset): clickhousedb -> clickhouse mapping in superset ingestion (#11201) 2024-08-26 16:38:01 -07:00
Felix Lüdin
ce99bc4f22
feat(ingest): add ingestion source for SAP Analytics Cloud (#10958)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-26 11:29:15 -07:00
Aseem Bansal
e2e6d105d3
fix(cli/delete): change filter to include env (#11246) 2024-08-26 09:56:02 -07:00
david-leifker
94e7706e3b
chore(bump): bump azure-identity (#11235)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-26 09:44:08 -05:00
Mayuri Nehate
223650dd7a
feat(ingest): add bigquery-queries source (#10994)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-25 22:51:00 -07:00
Harshal Sheth
1a09cb2c2a
fix(ingest/snowflake): propagate table list from main to query extractor (#11222) 2024-08-23 16:42:32 -04:00
Harshal Sheth
e0c13fda27
feat(ingest/dbt): add support for urns in add_owner directive (#11221) 2024-08-23 13:52:21 -04:00
Harshal Sheth
2729925cb8
docs(ingest): update config docs on platform instances (#11206) 2024-08-23 13:50:17 -04:00
Tamas Nemeth
ef6a410091
feat(ingest/s3): Partition support improvements (#11083)
- Partition autodetection
- Option to find min/max/min-max partition of a dataset
- Generating Partition aspects
2024-08-22 17:55:43 +02:00
sagar-salvi-apptware
50ed448861
fix(ingest/sagemaker): ensure consistent STS token usage with refresh mechanism (#11170)
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2024-08-22 15:42:13 +05:30
sid-acryl
dc30c0a0b7
feat(ingestion/looker): filter Looker dashboards by folder (#11205)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-22 14:10:00 +05:30
Alisa Aylward
88b3893b5c
feat(ingest/snowflake): allow iceberg tables in lineage and access metadata (#10961) 2024-08-22 01:20:51 +00:00
Harshal Sheth
0927c6331c
fix(ingest/redshift): avoid asserts in redshift schemas (#11219) 2024-08-21 17:08:13 -04:00
Andrew Sikowitz
30c4fa91ce
feat(ingest/mongodb): Ingest databases as containers (#11178)
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
2024-08-21 14:53:47 -04:00