Andrew Sikowitz
802c91a0a7
feat(ingest): Create Browse Paths V2 under flag ( #8120 )
2023-06-02 12:50:38 -07:00
Andrew Sikowitz
7ba2d13087
refactor(ingest): Make get_workunits() return MetadataWorkUnits ( #8051 )
...
- Deprecates UsageAggregationClass, /usageStats?action=batchIngest, UsageStatsWorkUnit
- Removes parsing of UsageAggregationClass in file source, all sinks, and WorkUnitRecordExtractor
2023-05-17 00:01:57 -04:00
david-leifker
697e8e2647
fix(misc): misc fixes ( #7633 )
2023-03-21 19:42:50 +05:30
Harshal Sheth
e9d50ed992
refactor(ingest/stateful): remove IngestionJobStateProvider ( #6792 )
2022-12-19 17:03:54 -05:00
Shirshanka Das
056add128d
fix(ingest): datahub-api - move instantiation to the right config class ( #5878 )
2022-09-09 13:34:21 -07:00
Harshal Sheth
64e7da8a68
fix(ingest): use temp dir for file generated during test ( #5505 )
2022-07-27 14:29:11 -07:00
Aseem Bansal
d518b5a085
fix(cli): correct handling of env variables ( #5203 )
2022-06-20 20:53:47 +05:30
Shirshanka Das
d0eb772301
fix(ingest): fwk - datahub_api should be initialized by datahub-rest … ( #4786 )
2022-04-28 22:31:19 -07:00
Shirshanka Das
a518e3d13e
feat(cli): improve error reporting, make sink config optional ( #4718 )
2022-04-24 17:12:21 -07:00
Swaroop Jagadish
35b187a8d4
feat(ingest): transformers - add support for processing MCP-s ( #4337 )
2022-03-07 13:14:29 -08:00
Claudio Benfatto
f944a9ba05
fix(ingest): enforce correct behaviour for commit policy ( #4092 )
2022-02-08 23:21:23 -08:00
Gabe Lyons
aa253f5b3b
feat(deletes): add run commands (list, show, rollback) to datahub ingest ( #2960 )
2021-07-29 20:04:40 -07:00
Harshal Sheth
79f60d8b8a
refactor(ingest): remove deprecated methods and warn on deprecated import ( #2797 )
2021-06-29 11:43:43 -07:00
Harshal Sheth
6d875b8241
test(ingest): ensure transformer registry works for aliases ( #2572 )
2021-05-17 15:08:49 -07:00
Harshal Sheth
a47400f18e
build(ingest): use gradle in commands + docs ( #2531 )
2021-05-11 19:03:20 -07:00
Harshal Sheth
71933a9f31
test(ingest): rename TestSource -> FakeSource ( #2481 )
2021-04-30 20:54:07 -07:00
Harshal Sheth
d415234a8c
fix(ingest): fields with defaults should be optional ( #2461 )
2021-04-26 16:45:48 -07:00
Thomas Larsson
7869a8f142
feature(ingestion): Adding the concept of transformers ( #2411 )
...
Fixes : #2410
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-04-18 11:15:05 -07:00
Harshal Sheth
bd78b84bd3
feat(ingest): start airflow integration + metadata builders ( #2331 )
2021-04-05 19:11:28 -07:00
Harshal Sheth
ac064584ae
refactor(ingest): cleanup configuration models ( #2134 )
2021-02-23 15:55:31 -08:00
Harshal Sheth
dfe00bfee8
feat(ingest): improve error reporting for pipelines ( #2121 )
2021-02-18 11:15:13 -08:00
Harshal Sheth
38f75be8ad
gometa -> datahub
2021-02-15 18:29:27 -08:00
Harshal Sheth
f807bb44b8
Use type + config everywhere
2021-02-15 18:29:27 -08:00
Harshal Sheth
df3e3da45b
More autofixes
2021-02-15 18:29:27 -08:00
Harshal Sheth
7ca018aaa4
Remove unused imports
2021-02-15 18:29:27 -08:00
Harshal Sheth
43d5fac494
Black
2021-02-15 18:29:27 -08:00
Harshal Sheth
107cf043d9
Add a create method to pipeline
2021-02-15 18:29:27 -08:00
Harshal Sheth
a87161cad7
Run black formatting on tests
2021-02-15 18:29:27 -08:00
Harshal Sheth
8ca8ef2d23
Fix kafka tests
2021-02-15 18:29:27 -08:00
Harshal Sheth
c7892ada4c
Codegen avro + datahub kafka sink ( #3 )
...
* Add codegen
* New architecture + setup file -> console pipeline
* Cleanup source loader
* Basic Kafka metadata source
* Kafka source and extractor
* Add kwargs construct interface
* Fix kafka source unit test
* start working on pipeline test
* kafka datahub sink
* Make myself a profile
* Ingest to datahub from kafka
* Update codegen
* Add restli transport
* Fix bug in restli conversion
2021-02-15 18:29:27 -08:00
Shirshanka Das
9e61220132
checking in testing fixtures. docker still not working
2021-02-15 18:29:27 -08:00
