1732 Commits

Author SHA1 Message Date
Harshal Sheth
1fa7998ed3
feat(ingest): support domains in meta -> "datahub" section (#10967) 2024-07-25 09:31:19 -07:00
Tamas Nemeth
8d874ad1e4
fix(ingest/test): Fix for mssql integration tests (#10978) 2024-07-24 16:41:25 +05:30
Harshal Sheth
a0905180f2
feat(ingest/looker): include dashboard urns in browse v2 (#10955) 2024-07-22 15:26:37 -07:00
Aseem Bansal
9f570a7521
feat(datajob/flow): add environment filter using info aspects (#10814) 2024-07-22 16:03:48 +05:30
sagar-salvi-apptware
348d449d8a
fix(ingest/Glue): column upstream lineage between S3 and Glue (#10895) 2024-07-19 14:39:19 +05:30
Harshal Sheth
7f3da47e90
fix(ingest/snowflake): fix test connection (#10927) 2024-07-17 11:57:58 -07:00
Harshal Sheth
bccfd8f0a5
feat(ingest/snowflake): integrate snowflake-queries into main source (#10905) 2024-07-17 10:22:14 -07:00
Harshal Sheth
79e1e2eb58
feat(ingest): report extractor failures more loudly (#10908) 2024-07-17 10:21:35 -07:00
sagar-salvi-apptware
ec788df328
fix(ingest/bigquery): handle quota exceeded for project.list requests (#10912) 2024-07-17 17:17:52 +02:00
pie1nthesky
298c299cf1
feat(ingest/postgres): fetch table sizes for profile (#10864) 2024-07-16 16:06:42 -07:00
Patrick Franco Braz
4b83adfa9f
fix(ingest/bigquery): changes helper function to decode unicode scape sequences (#10845) 2024-07-16 15:50:54 -07:00
Mayuri Nehate
ff1c6b895e
feat(ingest/BigQuery): refactor+parallelize dataset metadata extraction (#10884) 2024-07-16 11:46:42 -07:00
sid-acryl
bb1ba091cd
fix(ingestion/tableau): human-readable message for PERMISSIONS_MODE_SWITCHED error (#10866)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-07-16 10:36:51 -07:00
Aseem Bansal
437bacb0e6
feat(ingest): grafana connector (#10891)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-07-15 14:12:18 -07:00
Harshal Sheth
a4bce6af1c
feat(ingest): add snowflake-queries source (#10835) 2024-07-12 15:08:51 -07:00
Gabe Lyons
423af83ef1
feat(rest-emitter): adding async flag to rest emitter (#10902)
Co-authored-by: Gabe Lyons <gabe.lyons@acryl.io>
2024-07-12 13:30:21 -07:00
Harshal Sheth
351e434856
fix(ingest/dbt): always encode tag urns (#10799) 2024-07-11 16:32:16 -07:00
Harshal Sheth
82bd3c248f
fix(ingest): only populate audit stamps where accurate (#10604) 2024-07-11 13:26:57 -07:00
haeniya
3e86192b29
feat(ingestion/tableau): optionally ingest multiple sites and create site containers (#10498)
Co-authored-by: Yanik Häni <Yanik.Haeni1@swisscom.com>
2024-07-09 11:49:41 -07:00
Shubham Jagtap
b6c7fe8267
refactor(ingestion): remove company domain for security reason (#10839) 2024-07-08 21:15:20 -07:00
Aseem Bansal
41b9e15235
feat(ingest/audit): add client id and version in system metadata props (#10829) 2024-07-08 09:38:12 -07:00
sid-acryl
43bac365bc
fix(ingestion/lookml): liquid template resolution and view-to-view cll (#10542) 2024-07-08 09:26:39 -07:00
ipolding-cais
97638c3b9a
fix(ingest/tableau): quote special characters in external URL (#10842) 2024-07-07 21:49:49 -07:00
John Joyce
fa3e381f83
refactor(ingest): Refactor structured logging to support infos, warnings, and failures structured reporting to UI (#10828)
Co-authored-by: John Joyce <john@Johns-MBP.lan>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-07-03 19:20:58 -07:00
sagar-salvi-apptware
b8af2b9d69
fix(ingestion/glue): ensure date formatting works on all platforms for aws glue (#10836) 2024-07-03 18:05:37 +05:30
skrydal
099021c7a3
feat(ingest/glue): allow ingestion of empty databases from Glue (#10666)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-07-03 11:43:12 +05:30
sagar-salvi-apptware
640d42dc65
feat(ingest/transformer): tags to terms transformer (#10758)
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2024-07-02 15:30:05 +05:30
sid-acryl
3e4749624f
feat(ingest/lookml): ingest field tags (#10792)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-07-01 11:15:51 -07:00
Oleksandr Simonchuk
8b4e302881
feat(ingest): add and use file system abstraction in file source (#8415)
Co-authored-by: oleksandrsimonchuk <oleksandr.si@appsflyer.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2024-07-01 10:47:07 -07:00
Tim Drahn
93616f7869
fix(ingestion): ingest emails as empty if no ldap attribute (#9433)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-07-01 11:53:41 +05:30
Teppo Naakka
b223281305
feat(ingest/powerbi): powerbi dataset profiling (#9355) 2024-06-28 14:50:08 -07:00
Harshal Sheth
f4be88d0a9
feat(ingest): set pipeline name in system metadata (#10190)
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
2024-06-27 15:00:35 -07:00
Harshal Sheth
fa2ab1bcee
fix(ingest): add status aspect to dataProcessInstance (#10757) 2024-06-27 12:07:28 -07:00
Harshal Sheth
0d677e4992
fix(ingest/snowflake): fix column batcher (#10781) 2024-06-25 22:21:54 -07:00
Harshal Sheth
724907b8f4
feat(ingest): add async batch mode to the rest sink (#10733) 2024-06-25 15:49:00 -07:00
Harshal Sheth
0dc0bc5761
feat(ingest/snowflake): performance improvements (#10746) 2024-06-25 14:46:55 -07:00
Eric L (CCCS)
79ba0b1720
fix(ingest/iceberg): add support for nested dictionaries when configuring pyiceberg (#10762) 2024-06-21 14:38:01 -07:00
Harshal Sheth
2d727a960b
feat(ingest/snowflake): support more than 10k views in a db (#10718) 2024-06-18 07:37:39 +02:00
ethan-cartwright
c58be155f3
feat(ingest/bigquery): Support for View Labels (#10648)
Co-authored-by: Ethan Cartwright <ethan.cartwright@acryl.io>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2024-06-17 18:36:41 +05:30
Rajasekhar-Vuppala
a2d8a099d8
feat(ingest/vertica): use 3 part naming (#10636) 2024-06-14 17:04:55 -07:00
Harshal Sheth
62c6704f69
feat(ingest/snowflake): refactor + parallel schema extraction (#10653) 2024-06-14 13:23:07 -07:00
Shubham Jagtap
3479d070fe
fix(ingestion/sigma): Fix multiple requests http errors (#10616) 2024-06-14 11:30:45 -07:00
sagar-salvi-apptware
cc8389e431
fix(ingest/kafka-connect): Add lineage extraction for BigQuery Sink Connector in Kafka Connect source (#10647) 2024-06-14 18:51:04 +05:30
sagar-salvi-apptware
d69966074a
fix(ingest/bigquery): Map BigQuery policy tags to datahub column-level tags (#10669) 2024-06-14 16:43:12 +05:30
Harshal Sheth
6329153e36
fix(ingest): fix redshift query urns + reduce memory usage (#10691) 2024-06-13 11:27:06 -07:00
Harshal Sheth
25d48d2d09
fix(ingest/fivetran): fix fivetran bigquery support (#10693) 2024-06-13 11:26:47 -07:00
Harshal Sheth
3a72d92493
feat(ingest/dbt): include package_name in dbt custom props (#10652) 2024-06-12 15:07:42 +02:00
Shubham Jagtap
05aee03f3f
perf(ingestion/fivetran): Connector performance optimization (#10556) 2024-06-11 20:19:57 -07:00
skrydal
b9e71a61b1
feat(ingest/glue): database parameters extraction (#10665) 2024-06-11 11:50:46 -07:00
aabharti-visa
8a905774f7
feat(ingestion/kafka)-Add support for ingesting schemas from schema registry (#10612) 2024-06-11 14:00:12 +02:00