Andrew Sikowitz
2751a09284
fix(ingest): pass platform correctly to browse path v2 helper ( #8244 )
2023-06-15 20:10:15 -07:00
mohdsiddique
e7e07a73b4
feat(ingestion/powerbi): Ingest datasets not used in PowerBI visualization(tiles/pages) ( #8212 )
...
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-06-15 14:04:40 -07:00
Andrew Sikowitz
66806a805e
feat(ingest/unity): Set external url for containers and datasets ( #8238 )
2023-06-15 09:05:49 +02:00
Harshal Sheth
2d7692a245
feat(sdk): support patches as MCPs in file source ( #8220 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2023-06-14 14:56:27 -07:00
Andrew Sikowitz
c5cc53b99a
feat(ingest/bigquery_v2): enable platform instance using project id ( #8216 )
...
Co-authored-by: Adrián Pertíñez <khurzak92@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-06-14 09:50:21 -07:00
Mayuri Nehate
ab3fe0da81
refractor(classification): datahub classifier init ( #8193 )
2023-06-12 05:07:03 -07:00
Andrew Sikowitz
f2c66fd8d3
feat(ingest): Produce browse paths v2 on demand and with platform instance ( #8173 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
2023-06-09 10:35:54 -07:00
Andrew Sikowitz
369a04ae30
revert(ingest/bigquery): Do not emit DataPlatformInstance; remove references to platform_instance ( #8196 )
2023-06-09 13:44:24 +05:30
mohdsiddique
45e592b7c6
fix(ingestion/looker): ingest looks not part of dashboard ( #8140 )
...
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-06-08 12:54:14 +05:30
Mayuri Nehate
ac06cf3d3f
feat(classification): configurable minimum values threshold ( #8186 )
2023-06-07 21:28:13 -07:00
Andrew Sikowitz
9fa8489cb8
feat(ingest/snowflake): Okta OAuth support; update docs ( #8157 )
2023-06-07 01:09:05 -07:00
Andrew Sikowitz
6bad15be5c
fix(ingest): Fix modeldocgen; bump feast to relax pyarrow constraint ( #8178 )
2023-06-06 13:12:10 -07:00
Mayuri Nehate
983a8ca675
feat(classification): support for regex based custom infotypes ( #8177 )
2023-06-06 14:41:51 +02:00
Adrián Pertíñez
743439c11d
feat(ingest/bigquery_v2): enable platform instance using project id ( #8142 )
...
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-06-05 15:17:40 -07:00
Andrew Sikowitz
3022c2d12e
feat(ingest/unity): Add qualified name to dataset properties ( #8164 )
2023-06-05 11:20:13 -07:00
mohdsiddique
e7d1b900ec
fix(ingestion/looker): set project-name for imported_projects views ( #8086 )
2023-06-02 17:04:34 -07:00
Andrew Sikowitz
802c91a0a7
feat(ingest): Create Browse Paths V2 under flag ( #8120 )
2023-06-02 12:50:38 -07:00
Mayuri Nehate
fe1ff71318
fix(ingest/nifi): allow nifi site url with context path ( #8156 )
2023-06-02 15:43:33 +02:00
Harshal Sheth
690ed083d9
feat(ingest): add more fail-safes to stateful ingestion ( #8111 )
2023-05-31 18:49:48 -07:00
Harshal Sheth
a29b576daa
fix(ingest/json-schema): handle property inheritance in unions ( #8121 )
2023-05-30 22:59:28 -07:00
Tamas Nemeth
d50a99935b
fix(ingest/s3): Path spec aware folder traversal ( #8095 )
2023-05-30 16:20:49 +02:00
Serhii Dimchenko
6adb496581
feat: add dbt-athena adapter support for column types mapping ( #8116 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-05-27 09:22:55 -05:00
Aseem Bansal
96f364802b
feat(lineage source): add fine grained lineage support ( #7904 )
2023-05-26 17:09:32 +05:30
Harshal Sheth
2d442161c4
ci(ingest/kafka): improve kafka integration test reliability ( #8085 )
2023-05-25 15:40:56 -07:00
Andrew Sikowitz
d3cd4dbb0c
feat(ingest/unity): Allow ingestion without metastore admin role ( #8091 )
...
- Adds more detailed docs and connection test
- Fixes empty username queries
2023-05-24 15:36:22 -07:00
Mayuri Nehate
84270bcac8
feat(ingest/nifi): kerberos authentication ( #8097 )
...
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
Co-authored-by: Indy Prentice <iprentic@users.noreply.github.com>
2023-05-24 15:09:01 -07:00
Andrew Sikowitz
fdbc4de695
refactor(ingest): Call source_helpers via new WorkUnitProcessors in base Source ( #8101 )
2023-05-24 13:36:19 -07:00
Amanda Hernando
0e0d8934ea
feat(ingest): Add GenericAspectTransformer ( #7994 )
...
Co-authored-by: Adrián Pertíñez <khurzak92@gmail.com>
2023-05-24 13:31:33 -07:00
Mayuri Nehate
b3d80e57e8
feat(ingest/bigquery): usage for views ( #8046 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-05-24 09:48:58 -07:00
Andrew Sikowitz
8357fc8d64
feat(ingest): Browse Path v2 helper ( #8012 )
2023-05-23 23:46:46 -07:00
Harshal Sheth
b0f8c3de1e
refactor(ingest): simplify stateful ingestion provider interface ( #8104 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-05-23 12:57:57 -07:00
Harshal Sheth
afd65e16fb
feat(cli): delete cli v2 ( #8068 )
2023-05-23 14:43:44 -05:00
Harshal Sheth
4873a32e4a
fix(ingest): emitter bug fixes ( #8093 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-05-23 12:04:16 -07:00
Tamas Nemeth
f8be9f6aee
feat(ingest/s3): type aware directory sorting ( #8089 )
2023-05-23 08:59:46 +02:00
Andrew Sikowitz
a43903bf6d
refactor(ingest): Auto report workunits ( #8061 )
2023-05-22 17:06:31 -07:00
Harshal Sheth
4e9c652707
feat(ingest): add env to container properties ( #8027 )
2023-05-22 12:07:16 -07:00
Shubham Jagtap
e6371c8e94
fix(ingestion/powerbi): skip erroneous pages of a report ( #8021 )
...
Co-authored-by: mohdsiddique <mohdsiddiquebagwan@gmail.com>
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-05-19 18:02:55 -07:00
Harshal Sheth
98bba52c20
test(sdk): move cli tests into the unit dir ( #8028 )
2023-05-19 16:13:39 +02:00
Tamas Nemeth
bdd4bc7b92
feat(ingest/s3) - Stateful ingestion and last-updated support ( #8022 )
2023-05-19 13:10:15 +02:00
Harshal Sheth
00470acc02
test(sdk): better error messages in registry codegen test ( #8081 )
2023-05-19 11:18:50 +02:00
Harshal Sheth
1902e7d4db
ci(ingest/clickhouse): don't use kernel ephemeral ports ( #8060 )
2023-05-19 11:17:41 +02:00
Andrew Sikowitz
2e1c3981aa
refactor(ingest): Move source_helpers.py from datahub/utilities -> datahub/api ( #8052 )
2023-05-17 20:51:06 -07:00
mohdsiddique
ae30be9c25
fix(ingestion/tableau): ingest parent project name in container properties ( #8030 )
...
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-05-17 14:19:41 -07:00
Shubham Jagtap
8cc6606e68
feat(ingestion/kafka): add description in dataset properties ( #7974 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: mohdsiddique <mohdsiddiquebagwan@gmail.com>
2023-05-17 11:03:08 -07:00
Shirshanka Das
151eab3628
fix(build): fix lint issue ( #8066 )
2023-05-17 08:17:52 -07:00
Shirshanka Das
b3c790aab6
feat: Add support for Data Products ( #8039 )
...
Co-authored-by: Chris Collins <chriscollins3456@gmail.com>
2023-05-17 07:17:25 +00:00
Andrew Sikowitz
7ba2d13087
refactor(ingest): Make get_workunits() return MetadataWorkUnits ( #8051 )
...
- Deprecates UsageAggregationClass, /usageStats?action=batchIngest, UsageStatsWorkUnit
- Removes parsing of UsageAggregationClass in file source, all sinks, and WorkUnitRecordExtractor
2023-05-17 00:01:57 -04:00
Mayuri Nehate
a06c5aee2c
fix(ingest/bigquery): update usage audit log query to include create/drop operations ( #7995 )
2023-05-16 11:58:20 -07:00
Harshal Sheth
e70d0b3859
fix(ingest/dbt): fix dbt subtypes for sources ( #8048 )
2023-05-16 11:11:26 -07:00
xiphl
d619cc6b3f
feat(ingest): Allow csv-enricher to update more types ( #7932 )
...
Co-authored-by: xiphl <xiphlerl9@gmail.com>
2023-05-15 10:38:19 -07:00
