yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-14 04:31:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
11,753 Commits 192 Branches 118 Tags
Commit Graph

905 Commits

Author SHA1 Message Date
Andrew Sikowitz
2261531e31
test(ingest): Aspect level golden file comparison (#8310) 2023-07-11 10:39:47 -04:00
Harshal Sheth
3e47b3d228
feat(ingest): schema-aware SQL parsing for column-level lineage (#8334) 2023-07-07 16:24:35 -07:00
Andrew Sikowitz
1f84bf5b2b
fix(ingest/sql-common): Fix profile_table_level_only (#8331) 
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-07-07 19:05:50 -04:00
Andrew Sikowitz
3a21c27f06
feat(ingest): Turn on browse path v2 creation (#8342) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-07-06 16:43:42 -04:00
Mayuri Nehate
8cf778dc9b
fix(ingest): update pydantic helpers to address unique name issue (#8324) 
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
 2023-07-06 13:16:07 -07:00
Harshal Sheth
08d4e904a8
feat(ingest): add YamlFileUpdater utility (#8266) 2023-06-29 13:15:34 -07:00
Mayuri Nehate
711efde2c0
feat(ingest/snowflake): snowpipe s3 lineage (#8262) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-06-28 10:59:01 -04:00
Andrew Sikowitz
aa5e02d0ec
feat(ingest): Create zero usage aspects (#8205) 
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
 2023-06-22 17:07:50 -04:00
Andrew Sikowitz
2751a09284
fix(ingest): pass platform correctly to browse path v2 helper (#8244) 2023-06-15 20:10:15 -07:00
Andrew Sikowitz
66806a805e
feat(ingest/unity): Set external url for containers and datasets (#8238) 2023-06-15 09:05:49 +02:00
Harshal Sheth
2d7692a245
feat(sdk): support patches as MCPs in file source (#8220) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2023-06-14 14:56:27 -07:00
Andrew Sikowitz
c5cc53b99a
feat(ingest/bigquery_v2): enable platform instance using project id (#8216) 
Co-authored-by: Adrián Pertíñez <khurzak92@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-06-14 09:50:21 -07:00
Andrew Sikowitz
f2c66fd8d3
feat(ingest): Produce browse paths v2 on demand and with platform instance (#8173) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
 2023-06-09 10:35:54 -07:00
Andrew Sikowitz
369a04ae30
revert(ingest/bigquery): Do not emit DataPlatformInstance; remove references to platform_instance (#8196) 2023-06-09 13:44:24 +05:30
Andrew Sikowitz
9fa8489cb8
feat(ingest/snowflake): Okta OAuth support; update docs (#8157) 2023-06-07 01:09:05 -07:00
Mayuri Nehate
983a8ca675
feat(classification): support for regex based custom infotypes (#8177) 2023-06-06 14:41:51 +02:00
Adrián Pertíñez
743439c11d
feat(ingest/bigquery_v2): enable platform instance using project id (#8142) 
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
 2023-06-05 15:17:40 -07:00
Andrew Sikowitz
802c91a0a7
feat(ingest): Create Browse Paths V2 under flag (#8120) 2023-06-02 12:50:38 -07:00
Mayuri Nehate
fe1ff71318
fix(ingest/nifi): allow nifi site url with context path (#8156) 2023-06-02 15:43:33 +02:00
Harshal Sheth
690ed083d9
feat(ingest): add more fail-safes to stateful ingestion (#8111) 2023-05-31 18:49:48 -07:00
Harshal Sheth
a29b576daa
fix(ingest/json-schema): handle property inheritance in unions (#8121) 2023-05-30 22:59:28 -07:00
Tamas Nemeth
d50a99935b
fix(ingest/s3): Path spec aware folder traversal (#8095) 2023-05-30 16:20:49 +02:00
Aseem Bansal
96f364802b
feat(lineage source): add fine grained lineage support (#7904) 2023-05-26 17:09:32 +05:30
Harshal Sheth
2d442161c4
ci(ingest/kafka): improve kafka integration test reliability (#8085) 2023-05-25 15:40:56 -07:00
Andrew Sikowitz
d3cd4dbb0c
feat(ingest/unity): Allow ingestion without metastore admin role (#8091) 
- Adds more detailed docs and connection test
- Fixes empty username queries
 2023-05-24 15:36:22 -07:00
Mayuri Nehate
84270bcac8
feat(ingest/nifi): kerberos authentication (#8097) 
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
Co-authored-by: Indy Prentice <iprentic@users.noreply.github.com>
 2023-05-24 15:09:01 -07:00
Andrew Sikowitz
fdbc4de695
refactor(ingest): Call source_helpers via new WorkUnitProcessors in base Source (#8101) 2023-05-24 13:36:19 -07:00
Amanda Hernando
0e0d8934ea
feat(ingest): Add GenericAspectTransformer (#7994) 
Co-authored-by: Adrián Pertíñez <khurzak92@gmail.com>
 2023-05-24 13:31:33 -07:00
Mayuri Nehate
b3d80e57e8
feat(ingest/bigquery): usage for views (#8046) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
 2023-05-24 09:48:58 -07:00
Andrew Sikowitz
8357fc8d64
feat(ingest): Browse Path v2 helper (#8012) 2023-05-23 23:46:46 -07:00
Harshal Sheth
b0f8c3de1e
refactor(ingest): simplify stateful ingestion provider interface (#8104) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
 2023-05-23 12:57:57 -07:00
Harshal Sheth
afd65e16fb
feat(cli): delete cli v2 (#8068) 2023-05-23 14:43:44 -05:00
Harshal Sheth
4873a32e4a
fix(ingest): emitter bug fixes (#8093) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
 2023-05-23 12:04:16 -07:00
Tamas Nemeth
f8be9f6aee
feat(ingest/s3): type aware directory sorting (#8089) 2023-05-23 08:59:46 +02:00
Harshal Sheth
4e9c652707
feat(ingest): add env to container properties (#8027) 2023-05-22 12:07:16 -07:00
Harshal Sheth
98bba52c20
test(sdk): move cli tests into the unit dir (#8028) 2023-05-19 16:13:39 +02:00
Harshal Sheth
00470acc02
test(sdk): better error messages in registry codegen test (#8081) 2023-05-19 11:18:50 +02:00
Andrew Sikowitz
2e1c3981aa
refactor(ingest): Move source_helpers.py from datahub/utilities -> datahub/api (#8052) 2023-05-17 20:51:06 -07:00
Shubham Jagtap
8cc6606e68
feat(ingestion/kafka): add description in dataset properties (#7974) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: mohdsiddique <mohdsiddiquebagwan@gmail.com>
 2023-05-17 11:03:08 -07:00
Shirshanka Das
b3c790aab6
feat: Add support for Data Products (#8039) 
Co-authored-by: Chris Collins <chriscollins3456@gmail.com>
 2023-05-17 07:17:25 +00:00
Andrew Sikowitz
7ba2d13087
refactor(ingest): Make get_workunits() return MetadataWorkUnits (#8051) 
- Deprecates UsageAggregationClass, /usageStats?action=batchIngest, UsageStatsWorkUnit
- Removes parsing of UsageAggregationClass in file source, all sinks, and WorkUnitRecordExtractor
 2023-05-17 00:01:57 -04:00
Mayuri Nehate
a06c5aee2c
fix(ingest/bigquery): update usage audit log query to include create/drop operations (#7995) 2023-05-16 11:58:20 -07:00
Andrew Sikowitz
afcf462cb1
feat(ingest/unity): Add profiling support (#7976) 
- Also adds a new databricks sdk
 2023-05-11 10:00:50 -07:00
Andrew Sikowitz
44406f7adf
fix(ingest/postgres): Allow specification of initial engine database; set default database to postgres (#7915) 
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
 2023-05-09 11:11:43 -07:00
Mayuri Nehate
c845c75a2d
feat(ingest/snowflake): add config option to specify deny patterns for upstreams (#7962) 
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
 2023-05-08 14:13:57 -07:00
Mayuri Nehate
13b1d66170
fix(ingest/bigquery): remove incorrectly used table_pattern filter (#7810) 
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
 2023-05-08 10:33:42 -07:00
Mayuri Nehate
0131aeefb1
fix(ingest/unity): improve error message if no scheme in workspace_url (#7951) 
Co-authored-by: John Joyce <john@acryl.io>
 2023-05-08 10:13:53 -07:00
Tamas Nemeth
0e69e5a810
fix(ingest/redshift): Enabling autocommit for Redshift connection (#7983) 2023-05-08 10:24:40 +02:00
Andrew Sikowitz
8019d17aa6
fix(ingest/bigquery): Filter projects for lineage and usage (#7954) 2023-05-04 18:14:48 +02:00
Harshal Sheth
ca5dffa54d
refactor(ingest/biz-glossary): simplify business glossary source (#7912) 2023-05-03 17:01:58 -07:00