15 Commits

Author	SHA1	Message	Date
Harshal Sheth	d415234a8c	fix(ingest): fields with defaults should be optional (#2461)	2021-04-26 16:45:48 -07:00
Thomas Larsson	7869a8f142	feature(ingestion): Adding the concept of transformers (#2411) ... Fixes: #2410 Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>	2021-04-18 11:15:05 -07:00
Harshal Sheth	bd78b84bd3	feat(ingest): start airflow integration + metadata builders (#2331)	2021-04-05 19:11:28 -07:00
Harshal Sheth	ac064584ae	refactor(ingest): cleanup configuration models (#2134)	2021-02-23 15:55:31 -08:00
Harshal Sheth	dfe00bfee8	feat(ingest): improve error reporting for pipelines (#2121)	2021-02-18 11:15:13 -08:00
Harshal Sheth	38f75be8ad	gometa -> datahub	2021-02-15 18:29:27 -08:00
Harshal Sheth	f807bb44b8	Use type + config everywhere	2021-02-15 18:29:27 -08:00
Harshal Sheth	df3e3da45b	More autofixes	2021-02-15 18:29:27 -08:00
Harshal Sheth	7ca018aaa4	Remove unused imports	2021-02-15 18:29:27 -08:00
Harshal Sheth	43d5fac494	Black	2021-02-15 18:29:27 -08:00
Harshal Sheth	107cf043d9	Add a create method to pipeline	2021-02-15 18:29:27 -08:00
Harshal Sheth	a87161cad7	Run black formatting on tests	2021-02-15 18:29:27 -08:00
Harshal Sheth	8ca8ef2d23	Fix kafka tests	2021-02-15 18:29:27 -08:00
Harshal Sheth	c7892ada4c	Codegen avro + datahub kafka sink (#3) ... * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion	2021-02-15 18:29:27 -08:00
Shirshanka Das	9e61220132	checking in testing fixtures. docker still not working	2021-02-15 18:29:27 -08:00