1604 Commits

Author SHA1 Message Date
Andrew Sikowitz
759d0c86af
test(ingest/airflow): Fix test for airflow 2.6.3 (#8393) 2023-07-11 12:55:32 -04:00
Andrew Sikowitz
2261531e31
test(ingest): Aspect level golden file comparison (#8310) 2023-07-11 10:39:47 -04:00
Tamas Nemeth
e4736d7b46
fix(ingest/s3): Deequ import rearragement (#8389) 2023-07-10 14:40:06 -04:00
Andrew Sikowitz
d92cbe99e0
test(ingest/kafka): Configure kafka connect tests for arm64 (#8362) 2023-07-10 12:27:29 -04:00
Harshal Sheth
3e47b3d228
feat(ingest): schema-aware SQL parsing for column-level lineage (#8334) 2023-07-07 16:24:35 -07:00
Andrew Sikowitz
1f84bf5b2b
fix(ingest/sql-common): Fix profile_table_level_only (#8331)
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-07-07 19:05:50 -04:00
Andrew Sikowitz
74301ecb60
test(ingest/nifi): Configure nifi tests for arm64 (#8363)
Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-07-07 12:43:01 -04:00
Andrew Sikowitz
3a21c27f06
feat(ingest): Turn on browse path v2 creation (#8342)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-07-06 16:43:42 -04:00
Mayuri Nehate
8cf778dc9b
fix(ingest): update pydantic helpers to address unique name issue (#8324)
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2023-07-06 13:16:07 -07:00
Andrew Sikowitz
afbd52bdf0
test(ingest/mysql): Configure sql_server tests for arm64 (#8360)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-07-05 13:41:24 -04:00
Tamas Nemeth
74ab1bea06
fix(ingest/s3): Fix for flaky s3 test - uploading s3 files in consistent order (#8367) 2023-07-04 19:19:39 +02:00
Andrew Sikowitz
72a41ef9f6
test(ingest/trino): xfail test to unblock CI (#8340) 2023-06-30 17:51:50 +05:30
Harshal Sheth
08d4e904a8
feat(ingest): add YamlFileUpdater utility (#8266) 2023-06-29 13:15:34 -07:00
Mayuri Nehate
75d67b97bc
fix(ingest/postgres): fix profiling errors, skip json type column (#8291) 2023-06-28 10:59:31 -04:00
Mayuri Nehate
711efde2c0
feat(ingest/snowflake): snowpipe s3 lineage (#8262)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-06-28 10:59:01 -04:00
Gabe Lyons
d075bb4824
fix(embed): set embed url to false for tableau config (#8308) 2023-06-27 07:55:16 -07:00
Andrew Sikowitz
584366771d
refactor(unity): Remove databricks_cli and cleanup (#8249)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-06-23 18:01:05 +05:30
Andrew Sikowitz
aa5e02d0ec
feat(ingest): Create zero usage aspects (#8205)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-06-22 17:07:50 -04:00
Serhii Dimchenko
5b9fd977eb
fix(ingest/dbt-athena): dbt-athena types mapping for complex types (#8264)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-06-22 11:40:27 +02:00
Aseem Bansal
af6973ebe0
fix(ingest/okta): Set default of okta connector to match OIDC defaults (#8272) 2023-06-21 19:15:31 +05:30
Mayuri Nehate
88ceac316b
fix(ingest/tableau): split table columns query from datasources query (#8217)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-06-21 11:50:35 +02:00
Andrew Sikowitz
02bc71dd84
fix(ingest/okta): Set default of okta_profile_to_username_attr to email (#8263) 2023-06-21 13:38:59 +05:30
Andrew Sikowitz
2751a09284
fix(ingest): pass platform correctly to browse path v2 helper (#8244) 2023-06-15 20:10:15 -07:00
mohdsiddique
e7e07a73b4
feat(ingestion/powerbi): Ingest datasets not used in PowerBI visualization(tiles/pages) (#8212)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-06-15 14:04:40 -07:00
Andrew Sikowitz
66806a805e
feat(ingest/unity): Set external url for containers and datasets (#8238) 2023-06-15 09:05:49 +02:00
Harshal Sheth
2d7692a245
feat(sdk): support patches as MCPs in file source (#8220)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2023-06-14 14:56:27 -07:00
Andrew Sikowitz
c5cc53b99a
feat(ingest/bigquery_v2): enable platform instance using project id (#8216)
Co-authored-by: Adrián Pertíñez <khurzak92@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-06-14 09:50:21 -07:00
Mayuri Nehate
ab3fe0da81
refractor(classification): datahub classifier init (#8193) 2023-06-12 05:07:03 -07:00
Andrew Sikowitz
f2c66fd8d3
feat(ingest): Produce browse paths v2 on demand and with platform instance (#8173)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
2023-06-09 10:35:54 -07:00
Andrew Sikowitz
369a04ae30
revert(ingest/bigquery): Do not emit DataPlatformInstance; remove references to platform_instance (#8196) 2023-06-09 13:44:24 +05:30
mohdsiddique
45e592b7c6
fix(ingestion/looker): ingest looks not part of dashboard (#8140)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-06-08 12:54:14 +05:30
Mayuri Nehate
ac06cf3d3f
feat(classification): configurable minimum values threshold (#8186) 2023-06-07 21:28:13 -07:00
Andrew Sikowitz
9fa8489cb8
feat(ingest/snowflake): Okta OAuth support; update docs (#8157) 2023-06-07 01:09:05 -07:00
Andrew Sikowitz
6bad15be5c
fix(ingest): Fix modeldocgen; bump feast to relax pyarrow constraint (#8178) 2023-06-06 13:12:10 -07:00
Mayuri Nehate
983a8ca675
feat(classification): support for regex based custom infotypes (#8177) 2023-06-06 14:41:51 +02:00
Adrián Pertíñez
743439c11d
feat(ingest/bigquery_v2): enable platform instance using project id (#8142)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-06-05 15:17:40 -07:00
Andrew Sikowitz
3022c2d12e
feat(ingest/unity): Add qualified name to dataset properties (#8164) 2023-06-05 11:20:13 -07:00
mohdsiddique
e7d1b900ec
fix(ingestion/looker): set project-name for imported_projects views (#8086) 2023-06-02 17:04:34 -07:00
Andrew Sikowitz
802c91a0a7
feat(ingest): Create Browse Paths V2 under flag (#8120) 2023-06-02 12:50:38 -07:00
Mayuri Nehate
fe1ff71318
fix(ingest/nifi): allow nifi site url with context path (#8156) 2023-06-02 15:43:33 +02:00
Harshal Sheth
690ed083d9
feat(ingest): add more fail-safes to stateful ingestion (#8111) 2023-05-31 18:49:48 -07:00
Harshal Sheth
a29b576daa
fix(ingest/json-schema): handle property inheritance in unions (#8121) 2023-05-30 22:59:28 -07:00
Tamas Nemeth
d50a99935b
fix(ingest/s3): Path spec aware folder traversal (#8095) 2023-05-30 16:20:49 +02:00
Serhii Dimchenko
6adb496581
feat: add dbt-athena adapter support for column types mapping (#8116)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-05-27 09:22:55 -05:00
Aseem Bansal
96f364802b
feat(lineage source): add fine grained lineage support (#7904) 2023-05-26 17:09:32 +05:30
Harshal Sheth
2d442161c4
ci(ingest/kafka): improve kafka integration test reliability (#8085) 2023-05-25 15:40:56 -07:00
Andrew Sikowitz
d3cd4dbb0c
feat(ingest/unity): Allow ingestion without metastore admin role (#8091)
- Adds more detailed docs and connection test
- Fixes empty username queries
2023-05-24 15:36:22 -07:00
Mayuri Nehate
84270bcac8
feat(ingest/nifi): kerberos authentication (#8097)
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
Co-authored-by: Indy Prentice <iprentic@users.noreply.github.com>
2023-05-24 15:09:01 -07:00
Andrew Sikowitz
fdbc4de695
refactor(ingest): Call source_helpers via new WorkUnitProcessors in base Source (#8101) 2023-05-24 13:36:19 -07:00
Amanda Hernando
0e0d8934ea
feat(ingest): Add GenericAspectTransformer (#7994)
Co-authored-by: Adrián Pertíñez <khurzak92@gmail.com>
2023-05-24 13:31:33 -07:00