45 Commits

Author SHA1 Message Date
Harshal Sheth
f3b17d4d4c
feat(ingest): improve error messages for unknown metadata objects (#12745) 2025-02-28 12:36:58 -08:00
Harshal Sheth
7dbb3e60cb
chore(ingest): start using explicit exports (#11899) 2024-11-20 13:33:30 -08:00
Mayuri Nehate
84c677629d
feat(ingest): add stateful ingestion support for file source (#11804) 2024-11-08 16:11:30 +05:30
Andrew Sikowitz
16fe22aafa
feat(ui/last-updated): Calculate last updated time as max(properties time, operation time) (#9242) 2023-12-06 15:21:56 -05:00
skrydal
378d84a346
fix(ui): Fixes handling of resources filters in UI (#9087) 2023-10-24 21:12:11 +01:00
Harshal Sheth
2d7692a245
feat(sdk): support patches as MCPs in file source (#8220)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2023-06-14 14:56:27 -07:00
Harshal Sheth
89734587f7
feat(ingest): add urn modification helper (#7440)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-03-16 13:27:08 -07:00
Harshal Sheth
35bd73a28b
feat(ingest): fix handling of unions with aliases in post restli conversion (#7058) 2023-01-18 09:29:46 -08:00
Harshal Sheth
cb12910b6b
feat(ingest): add entity registry in codegen (#6984)
Co-authored-by: Pedro Silva <pedro@acryl.io>
2023-01-17 19:41:43 -08:00
Tamas Nemeth
e41b455e14
fix(ingest): bigquery - sharded table support improvements (#6789) 2022-12-19 18:57:37 +01:00
Harshal Sheth
7d63399d00
fix(ingest): fix serde for empty dicts in unions with null (#6745)
The code changes in https://github.com/acryldata/avro_gen/pull/16, but tests are written here.
2022-12-13 08:17:24 +01:00
Harshal Sheth
0ca3383d30
feat(ingest): support reserved keywords in model codegen (#6351) 2022-11-02 22:26:56 -07:00
Harshal Sheth
6063484714
fix(ingest): avrogen handling for missing fields with default values (#5844) 2022-09-08 14:05:28 -07:00
Harshal Sheth
08622f25ef
feat(ingest): add utility for converting MCEs to MCPs (#5812) 2022-09-06 15:25:48 -07:00
Harshal Sheth
c05f3970fd
feat(ingest): cli - add rewrite option for metadata file check (#5763) 2022-09-01 14:30:00 -07:00
Harshal Sheth
eb87db9813
fix(ingest): proper null skip logic in serialization (#5749) 2022-08-29 16:34:58 -07:00
Shirshanka Das
bb788ac317
feat(ingest): file - add support for folders, large files, improve co… (#5692) 2022-08-21 14:18:22 +05:30
Harshal Sheth
9790f3cefa
feat(ingest): infer aspectName from aspect type in MCP (#5566) 2022-08-07 07:52:58 -07:00
John Joyce
5c501fba2c
fix(cli): Bumping avro-gen3 to 0.7.4 (#5098) 2022-06-07 12:56:06 -07:00
Harshal Sheth
49a8ece02a
fix(ingestion): enable compat with avro 1.11 (#4205) 2022-02-22 22:13:50 -08:00
Harshal Sheth
22cef5f897
refactor(test): replace CliRunner with run_datahub_cmd method (#3746) 2021-12-16 20:07:38 -08:00
rslanka
8844240328
feat: Adding support for nested schemas in ingestion and visualization (#3079) 2021-08-11 15:47:18 -07:00
John Joyce
352a0abf8d
Introducing TimeSeries Aspects + Dataset Profile (Stats) Aspect (#2983)
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-07-30 17:41:03 -07:00
Gabe Lyons
aa253f5b3b
feat(deletes): add run commands (list, show, rollback) to datahub ingest (#2960) 2021-07-29 20:04:40 -07:00
Harshal Sheth
7ab6355b1c
feat(ingest): stricter deserialization for MCE JSONs (#2976) 2021-07-28 14:50:21 -07:00
Harshal Sheth
d66381451a
feat(ingest): refactor mce comparison and add pytest update golden files option (#2812) 2021-06-30 16:53:20 -07:00
Harshal Sheth
937f02c6bc
feat: usage stats (part 1) (#2750)
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-06-24 17:11:00 -07:00
Harshal Sheth
5d93f249b4
feat(ingest): expose additional types to Python via codegen (#2712) 2021-06-17 10:04:28 -07:00
Brian
a5f9b8dfe9
feat(entities): add markdown description update/viewer feature in dataset, datajob, dataflow, chart and dashboard, update ui/ux (#2707) 2021-06-16 15:48:27 -07:00
Harshal Sheth
9f4de4b20a
fix(ingest): remove datahub.metadata import shortcut (#2449) 2021-04-30 21:10:12 -07:00
Harshal Sheth
91a2f69310
fix(ingest): properly handle fieldDiscriminator with restli (#2408) 2021-04-16 09:42:52 -07:00
Harshal Sheth
ffe03e6758
fix(ingest): streamline codegen init methods (#2400) 2021-04-14 19:25:57 -07:00
Harshal Sheth
2af4603e49
fix(ingest): enable mypy disallow_incomplete_defs and disallow_untyped_decorators (#2393) 2021-04-14 13:40:24 -07:00
Harshal Sheth
a11329d5b8
refactor(ingest): update test harness to use a compose file per test (#2392) 2021-04-13 17:30:24 -07:00
Harshal Sheth
bfe345da42
fix(ingest): add test for avro serialization and deserialization (#2351) 2021-04-07 21:30:21 -07:00
Harshal Sheth
b8462028c3
feat(ingest): various minor fixes (#2246) 2021-03-17 23:05:05 -07:00
Harshal Sheth
aa6bc15cd7
fix(ingest): various avro codegen fixes (#2232) 2021-03-15 15:27:30 -07:00
Harshal Sheth
dfe00bfee8
feat(ingest): improve error reporting for pipelines (#2121) 2021-02-18 11:15:13 -08:00
Harshal Sheth
38f75be8ad gometa -> datahub 2021-02-15 18:29:27 -08:00
Harshal Sheth
f807bb44b8 Use type + config everywhere 2021-02-15 18:29:27 -08:00
Harshal Sheth
9332e6b878 Add isort to CI 2021-02-15 18:29:27 -08:00
Harshal Sheth
7ca018aaa4 Remove unused imports 2021-02-15 18:29:27 -08:00
Harshal Sheth
43d5fac494 Black 2021-02-15 18:29:27 -08:00
Harshal Sheth
1de2bacca4 Fix serde test 2021-02-15 18:29:27 -08:00
Harshal Sheth
b2449eedaa Add serde tests 2021-02-15 18:29:27 -08:00