780 Commits

Author	SHA1	Message	Date
Harshal Sheth	dfe00bfee8	feat(ingest): improve error reporting for pipelines (#2121)	2021-02-18 11:15:13 -08:00
Harshal Sheth	38f75be8ad	gometa -> datahub	2021-02-15 18:29:27 -08:00
Harshal Sheth	0063c04460	gometa-ingest -> datahub ingest	2021-02-15 18:29:27 -08:00
Harshal Sheth	d0bc3c55db	Setup CI	2021-02-15 18:29:27 -08:00
Harshal Sheth	f807bb44b8	Use type + config everywhere	2021-02-15 18:29:27 -08:00
Harshal Sheth	9332e6b878	Add isort to CI	2021-02-15 18:29:27 -08:00
Harshal Sheth	d483d23fd7	Allow/deny patterns for kafka source	2021-02-15 18:29:27 -08:00
Harshal Sheth	df3e3da45b	More autofixes	2021-02-15 18:29:27 -08:00
Harshal Sheth	7ca018aaa4	Remove unused imports	2021-02-15 18:29:27 -08:00
Harshal Sheth	43d5fac494	Black	2021-02-15 18:29:27 -08:00
Harshal Sheth	1de2bacca4	Fix serde test	2021-02-15 18:29:27 -08:00
Harshal Sheth	107cf043d9	Add a create method to pipeline	2021-02-15 18:29:27 -08:00
Harshal Sheth	b2449eedaa	Add serde tests	2021-02-15 18:29:27 -08:00
Shirshanka Das	7ba81d7087	remove old shell scripts	2021-02-15 18:29:27 -08:00
Harshal Sheth	a87161cad7	Run black formatting on tests	2021-02-15 18:29:27 -08:00
Harshal Sheth	2307c59296	Add support for rich kafka config	2021-02-15 18:29:27 -08:00
Harshal Sheth	b759205b52	Start checking files for mysql test	2021-02-15 18:29:27 -08:00
Harshal Sheth	36874970ce	Finish file sink	2021-02-15 18:29:27 -08:00
Harshal Sheth	23d675663b	Minor test runid renames	2021-02-15 18:29:27 -08:00
Harshal Sheth	8ca8ef2d23	Fix kafka tests	2021-02-15 18:29:27 -08:00
Harshal Sheth	db1ff07e6d	Add reporter everywhere	2021-02-15 18:29:27 -08:00
Harshal Sheth	0929c7cb77	Add test to check that classes are not abstract	2021-02-15 18:29:27 -08:00
Harshal Sheth	9ea03820c4	Enable bare pytest	2021-02-15 18:29:27 -08:00
Shirshanka Das	4e17a24dd4	Adding mysql integration test + source	2021-02-15 18:29:27 -08:00
Harshal Sheth	22a4dbb45f	Cleanup kafka sink schema setup	2021-02-15 18:29:27 -08:00
Harshal Sheth	4b83fc6591	adding allow deny patterns to sql config	2021-02-15 18:29:27 -08:00
Shirshanka Das	d73d8ff4c2	fixing unit tests	2021-02-15 18:29:27 -08:00
Harshal Sheth	c7892ada4c	Codegen avro + datahub kafka sink (#3) ... * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion	2021-02-15 18:29:27 -08:00
Shirshanka Das	6b5bbbdc5f	workaround for docker exec, waiting for 5 more seconds	2021-02-15 18:29:27 -08:00
Shirshanka Das	9e61220132	checking in testing fixtures. docker still not working	2021-02-15 18:29:27 -08:00