Harshal Sheth
f3b17d4d4c
feat(ingest): improve error messages for unknown metadata objects ( #12745 )
2025-02-28 12:36:58 -08:00
Aseem Bansal
9c3bd34995
dev: enable ruff rule ( #12749 )
2025-02-28 17:49:52 +05:30
Aseem Bansal
262dd76518
dev: remove black in favor of ruff for formatting ( #12378 )
2025-01-18 15:06:20 +05:30
Harshal Sheth
d0423547ba
feat(ingest): add parse_ts_millis helper ( #12231 )
2024-12-27 10:50:28 -08:00
Harshal Sheth
76cfac3700
fix(cli/properties): allow structured properties without a graph instance ( #12144 )
...
Co-authored-by: Sergio Gómez Villamor <sgomezvillamor@gmail.com>
2024-12-18 09:04:51 +00:00
Harshal Sheth
7dbb3e60cb
chore(ingest): start using explicit exports ( #11899 )
2024-11-20 13:33:30 -08:00
Mayuri Nehate
84c677629d
feat(ingest): add stateful ingestion support for file source ( #11804 )
2024-11-08 16:11:30 +05:30
Harshal Sheth
b8144699fd
chore(ingest): reorganize unit tests ( #11636 )
2024-10-16 19:18:32 -07:00
Andrew Sikowitz
16fe22aafa
feat(ui/last-updated): Calculate last updated time as max(properties time, operation time) ( #9242 )
2023-12-06 15:21:56 -05:00
Harshal Sheth
bab9d1c931
fix(ingest): fully support MCPs in urn_iter primitive ( #9157 )
2023-11-01 20:47:18 -07:00
skrydal
378d84a346
fix(ui): Fixes handling of resources filters in UI ( #9087 )
2023-10-24 21:12:11 +01:00
Harshal Sheth
2d7692a245
feat(sdk): support patches as MCPs in file source ( #8220 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2023-06-14 14:56:27 -07:00
Harshal Sheth
89734587f7
feat(ingest): add urn modification helper ( #7440 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-03-16 13:27:08 -07:00
Harshal Sheth
35bd73a28b
feat(ingest): fix handling of unions with aliases in post restli conversion ( #7058 )
2023-01-18 09:29:46 -08:00
Harshal Sheth
cb12910b6b
feat(ingest): add entity registry in codegen ( #6984 )
...
Co-authored-by: Pedro Silva <pedro@acryl.io>
2023-01-17 19:41:43 -08:00
Harshal Sheth
667ca8632d
feat(ingest): avoid embedding serialized json in metadata files ( #6742 )
2022-12-28 19:28:38 -05:00
Tamas Nemeth
e41b455e14
fix(ingest): bigquery - sharded table support improvements ( #6789 )
2022-12-19 18:57:37 +01:00
Harshal Sheth
7d63399d00
fix(ingest): fix serde for empty dicts in unions with null ( #6745 )
...
The code changes in https://github.com/acryldata/avro_gen/pull/16 , but tests are written here.
2022-12-13 08:17:24 +01:00
Harshal Sheth
0ca3383d30
feat(ingest): support reserved keywords in model codegen ( #6351 )
2022-11-02 22:26:56 -07:00
Harshal Sheth
6063484714
fix(ingest): avrogen handling for missing fields with default values ( #5844 )
2022-09-08 14:05:28 -07:00
Harshal Sheth
08622f25ef
feat(ingest): add utility for converting MCEs to MCPs ( #5812 )
2022-09-06 15:25:48 -07:00
Harshal Sheth
c05f3970fd
feat(ingest): cli - add rewrite option for metadata file check ( #5763 )
2022-09-01 14:30:00 -07:00
Harshal Sheth
eb87db9813
fix(ingest): proper null skip logic in serialization ( #5749 )
2022-08-29 16:34:58 -07:00
Shirshanka Das
bb788ac317
feat(ingest): file - add support for folders, large files, improve co… ( #5692 )
2022-08-21 14:18:22 +05:30
Harshal Sheth
9790f3cefa
feat(ingest): infer aspectName from aspect type in MCP ( #5566 )
2022-08-07 07:52:58 -07:00
John Joyce
5c501fba2c
fix(cli): Bumping avro-gen3 to 0.7.4 ( #5098 )
2022-06-07 12:56:06 -07:00
Harshal Sheth
49a8ece02a
fix(ingestion): enable compat with avro 1.11 ( #4205 )
2022-02-22 22:13:50 -08:00
Swaroop Jagadish
7d986ec880
fix(ingest): populate system metadata for all metadata events (mcp, mcpw) ( #3900 )
2022-01-16 12:03:38 -08:00
Harshal Sheth
22cef5f897
refactor(test): replace CliRunner with run_datahub_cmd method ( #3746 )
2021-12-16 20:07:38 -08:00
Gabe Lyons
ff527f4bed
feat(foreign keys): add foreign key models ( #3275 )
2021-09-22 10:29:27 -07:00
rslanka
8844240328
feat: Adding support for nested schemas in ingestion and visualization ( #3079 )
2021-08-11 15:47:18 -07:00
John Joyce
352a0abf8d
Introducing TimeSeries Aspects + Dataset Profile (Stats) Aspect ( #2983 )
...
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-07-30 17:41:03 -07:00
Gabe Lyons
aa253f5b3b
feat(deletes): add run commands (list, show, rollback) to datahub ingest ( #2960 )
2021-07-29 20:04:40 -07:00
Harshal Sheth
7ab6355b1c
feat(ingest): stricter deserialization for MCE JSONs ( #2976 )
2021-07-28 14:50:21 -07:00
Harshal Sheth
d66381451a
feat(ingest): refactor mce comparison and add pytest update golden files option ( #2812 )
2021-06-30 16:53:20 -07:00
Harshal Sheth
19b2a42a00
feat: usage stats (part 2) ( #2762 )
...
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-06-24 19:44:59 -07:00
Harshal Sheth
937f02c6bc
feat: usage stats (part 1) ( #2750 )
...
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-06-24 17:11:00 -07:00
Harshal Sheth
5d93f249b4
feat(ingest): expose additional types to Python via codegen ( #2712 )
2021-06-17 10:04:28 -07:00
Brian
a5f9b8dfe9
feat(entities): add markdown description update/viewer feature in dataset, datajob, dataflow, chart and dashboard, update ui/ux ( #2707 )
2021-06-16 15:48:27 -07:00
shakti-garg
8ed14a62e2
feat(business_glossary): add new entity business term and its relationship with dataset and its fields ( #2228 )
...
Co-authored-by: shubham.garg <shubham.garg@thoughtworks.com>
2021-05-10 13:20:23 -07:00
Harshal Sheth
9f4de4b20a
fix(ingest): remove datahub.metadata import shortcut ( #2449 )
2021-04-30 21:10:12 -07:00
Gabe Lyons
851e00ba9f
feat(lineage): implement support for datasets, charts and dashboards downstream lineage fetching in a generic way ( #2397 )
...
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Brian <brianwebtek@gmail.com>
Co-authored-by: John Joyce <john@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2021-04-23 00:18:39 -07:00
Harshal Sheth
91a2f69310
fix(ingest): properly handle fieldDiscriminator with restli ( #2408 )
2021-04-16 09:42:52 -07:00
Harshal Sheth
ffe03e6758
fix(ingest): streamline codegen init methods ( #2400 )
2021-04-14 19:25:57 -07:00
Harshal Sheth
2af4603e49
fix(ingest): enable mypy disallow_incomplete_defs and disallow_untyped_decorators ( #2393 )
2021-04-14 13:40:24 -07:00
Harshal Sheth
a11329d5b8
refactor(ingest): update test harness to use a compose file per test ( #2392 )
2021-04-13 17:30:24 -07:00
Harshal Sheth
bfe345da42
fix(ingest): add test for avro serialization and deserialization ( #2351 )
2021-04-07 21:30:21 -07:00
Harshal Sheth
b8462028c3
feat(ingest): various minor fixes ( #2246 )
2021-03-17 23:05:05 -07:00
Harshal Sheth
aa6bc15cd7
fix(ingest): various avro codegen fixes ( #2232 )
2021-03-15 15:27:30 -07:00
Gabe Lyons
adfe60e97a
feat(tags): adding support for read/write of tags in gms & read-only in react datahub-frontend. ( #2164 )
2021-03-07 11:26:47 -08:00
