3089 Commits

Author SHA1 Message Date
Andrew Sikowitz
2751a09284
fix(ingest): pass platform correctly to browse path v2 helper (#8244) 2023-06-15 20:10:15 -07:00
Shubham Jagtap
35a4434490
fix(sdk/dataflow): deprecate cluster and use env and platform_instance instead (#8201)
Co-authored-by: mohdsiddique <mohdsiddiquebagwan@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-06-15 14:06:28 -07:00
mohdsiddique
e7e07a73b4
feat(ingestion/powerbi): Ingest datasets not used in PowerBI visualization(tiles/pages) (#8212)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-06-15 14:04:40 -07:00
Tamas Nemeth
a3ceee46d0
fix(ingest/druid) Handling gracefully if no table returned in a schema (#8203) 2023-06-15 18:40:21 +05:30
mohdsiddique
25fa4ebbef
fix(ingestion/tableau): Remove unused field documentViewId (#8225)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-06-15 16:38:30 +05:30
Andrew Sikowitz
66806a805e
feat(ingest/unity): Set external url for containers and datasets (#8238) 2023-06-15 09:05:49 +02:00
RyanHolstien
f78f7f9609
feat(patch): support datajob input output (#8190)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
2023-06-14 22:21:16 -07:00
Harshal Sheth
2d7692a245
feat(sdk): support patches as MCPs in file source (#8220)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2023-06-14 14:56:27 -07:00
Andrew Sikowitz
c5cc53b99a
feat(ingest/bigquery_v2): enable platform instance using project id (#8216)
Co-authored-by: Adrián Pertíñez <khurzak92@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-06-14 09:50:21 -07:00
Harshal Sheth
1528935d40
fix(cli): make quickstart robust to docker race conditions (#8233) 2023-06-13 23:28:03 -07:00
Harshal Sheth
e5f9ee178c
fix(cli/quickstart): handle docker hangs gracefully (#8211)
Co-authored-by: Pedro Silva <pedro@acryl.io>
2023-06-13 14:34:06 -05:00
Mayuri Nehate
8e8f2f5e65
fix(ingest/delta-lake): fix CI issues due to delta lake version bump (#8215) 2023-06-12 08:02:04 -07:00
Mayuri Nehate
f3cf9b7d5a
fix(glue): fix typo in reported warning, report with flow_urn (#8138) 2023-06-12 05:08:20 -07:00
Mayuri Nehate
ab3fe0da81
refractor(classification): datahub classifier init (#8193) 2023-06-12 05:07:03 -07:00
Andrew Sikowitz
f2c66fd8d3
feat(ingest): Produce browse paths v2 on demand and with platform instance (#8173)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
2023-06-09 10:35:54 -07:00
Andrew Sikowitz
369a04ae30
revert(ingest/bigquery): Do not emit DataPlatformInstance; remove references to platform_instance (#8196) 2023-06-09 13:44:24 +05:30
Tamas Nemeth
8d31a18e1f
fix(ingest/redshift): Adding env parameter where it was missing for urn generation (#8199) 2023-06-09 08:38:33 +02:00
Andrew Sikowitz
a04631bb5c
fix(ingest/bigquery): Swap log order to avoid confusion (#8197) 2023-06-08 21:05:06 +00:00
Harshal Sheth
2e1d31b6ae
fix(ingest/profiling): only apply monkeypatches once when profiling (#8160) 2023-06-08 12:59:17 +05:30
mohdsiddique
45e592b7c6
fix(ingestion/looker): ingest looks not part of dashboard (#8140)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-06-08 12:54:14 +05:30
Mayuri Nehate
ac06cf3d3f
feat(classification): configurable minimum values threshold (#8186) 2023-06-07 21:28:13 -07:00
Andrew Sikowitz
fd790e33d8
fix(ingest/kafka): Better error handling around topic and topic description extraction (#8183) 2023-06-07 12:52:38 -07:00
Tamas Nemeth
ea79a4b6af
feat(ingest/presto-on-hive): add support for extra properties and merge property capabilities (#8147)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2023-06-07 13:12:20 +02:00
Andrew Sikowitz
9fa8489cb8
feat(ingest/snowflake): Okta OAuth support; update docs (#8157) 2023-06-07 01:09:05 -07:00
Mayuri Nehate
983a8ca675
feat(classification): support for regex based custom infotypes (#8177) 2023-06-06 14:41:51 +02:00
Mayuri Nehate
9fc1ef0111
fix(ingest/snowflake): set use_quoted_name to profile lowercase tables (#8168)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-06-06 14:40:58 +02:00
Tamas Nemeth
279f9eb455
fix(ingest/kafka): Fixing error printing in Kafka properties get call (#8145) 2023-06-06 11:17:14 +02:00
Andrew Sikowitz
fcaaedd83f
feat(ingest/snowflake): Deprecate legacy lineage and optimize query history joins (#8176) 2023-06-06 10:14:28 +02:00
Adrián Pertíñez
743439c11d
feat(ingest/bigquery_v2): enable platform instance using project id (#8142)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-06-05 15:17:40 -07:00
Andrew Sikowitz
3022c2d12e
feat(ingest/unity): Add qualified name to dataset properties (#8164) 2023-06-05 11:20:13 -07:00
mohdsiddique
e7d1b900ec
fix(ingestion/looker): set project-name for imported_projects views (#8086) 2023-06-02 17:04:34 -07:00
Andrew Sikowitz
802c91a0a7
feat(ingest): Create Browse Paths V2 under flag (#8120) 2023-06-02 12:50:38 -07:00
Mayuri Nehate
fe1ff71318
fix(ingest/nifi): allow nifi site url with context path (#8156) 2023-06-02 15:43:33 +02:00
Mayuri Nehate
fb44cbdb15
feat(ingest/snowflake): support for more operation types (#8158) 2023-06-01 15:47:39 +02:00
Harshal Sheth
690ed083d9
feat(ingest): add more fail-safes to stateful ingestion (#8111) 2023-05-31 18:49:48 -07:00
Harshal Sheth
60dd9ef187
fix(ingest): remove original_table_name logic in sql source (#8130) 2023-05-31 15:58:09 -07:00
Harshal Sheth
a29b576daa
fix(ingest/json-schema): handle property inheritance in unions (#8121) 2023-05-30 22:59:28 -07:00
Tamas Nemeth
d50a99935b
fix(ingest/s3): Path spec aware folder traversal (#8095) 2023-05-30 16:20:49 +02:00
Gabe Lyons
ada6ea5a45
docs(csv-enricher): add example csv file & recipe (#8141) 2023-05-29 19:02:26 +05:30
Serhii Dimchenko
6adb496581
feat: add dbt-athena adapter support for column types mapping (#8116)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-05-27 09:22:55 -05:00
Pedro Silva
ea92b86e6a
feat(custom ownership): Adds Custom ownership types as a top level entity (#8045)
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
2023-05-26 17:54:09 +01:00
Aseem Bansal
96f364802b
feat(lineage source): add fine grained lineage support (#7904) 2023-05-26 17:09:32 +05:30
Mayuri Nehate
f2c53a3660
feat(ingest/glue): report glue job lineage failures, update doc (#8126) 2023-05-26 10:30:03 +02:00
Andrew Sikowitz
668a09ffd4
fix(ingest/bigquery): Deduplicate LogEntries based on insertId, timestamp, logName (#8132) 2023-05-25 17:53:47 -07:00
Tamas Nemeth
b216caa83b
fix(ingest/redshift): Fixing Redshift subtypes (#8125) 2023-05-25 19:45:07 +02:00
Vinícius Mello
7059874dec
feat(ingest/bigquery): Add BigQuery Views lineage extraction from Google Data Catalog API (#8100) 2023-05-25 08:37:46 -07:00
Andrew Sikowitz
d3cd4dbb0c
feat(ingest/unity): Allow ingestion without metastore admin role (#8091)
- Adds more detailed docs and connection test
- Fixes empty username queries
2023-05-24 15:36:22 -07:00
Tamas Nemeth
fb087c5e35
fix(ingest/redshift):fixing schema filter (#8119) 2023-05-24 22:35:28 +00:00
Mayuri Nehate
84270bcac8
feat(ingest/nifi): kerberos authentication (#8097)
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
Co-authored-by: Indy Prentice <iprentic@users.noreply.github.com>
2023-05-24 15:09:01 -07:00
Andrew Sikowitz
fdbc4de695
refactor(ingest): Call source_helpers via new WorkUnitProcessors in base Source (#8101) 2023-05-24 13:36:19 -07:00