1604 Commits

Author	SHA1	Message	Date
Thomas Larsson	e02a17aecf	fix(ingestion): Support mapping from avro "boolean" and "map" types t… (#2364) ... Fixes: #2363 Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>	2021-04-08 14:23:12 -07:00
Thomas Larsson	4215dcd53c	fix(ingestion): properly detect optional fields in avro schemas (#2343) ... Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>	2021-04-08 14:00:01 -07:00
Harshal Sheth	bfe345da42	fix(ingest): add test for avro serialization and deserialization (#2351)	2021-04-07 21:30:21 -07:00
Harshal Sheth	cafdd9664a	fix(cli): add --verbose flag for datahub check plugins (#2345)	2021-04-06 15:41:15 -07:00
Harshal Sheth	bd78b84bd3	feat(ingest): start airflow integration + metadata builders (#2331)	2021-04-05 19:11:28 -07:00
Harshal Sheth	c1f3eaed35	fix(ingest): add support for database and table patterns to glue source (#2339)	2021-04-05 17:14:02 -07:00
amy m	759288161c	feat(ingest): adding support for AWS Glue (#2319) ... Co-authored-by: Harry Nash <harrywilliamnash@gmail.com>	2021-04-04 11:00:27 -07:00
Gary Lucas	9d00e8bb2e	fix(test): dbt-manifest files (#2330)	2021-04-02 10:59:20 -07:00
Harshal Sheth	cb24628886	feat(ingest): verify dynamic registry types at runtime (#2327)	2021-04-01 12:15:05 -07:00
Harshal Sheth	f57c954fc6	feat(ingest): support environment variables in recipes (#2306)	2021-03-26 21:57:05 -07:00
Harshal Sheth	a921d0deae	feat(ingest): MongoDB ingestion source (#2289)	2021-03-23 20:15:44 -07:00
Gary Lucas	922307f8ec	feat(ingest): Add DBT source to metadata-ingestion (#2247)	2021-03-22 23:11:29 -07:00
Harshal Sheth	b8462028c3	feat(ingest): various minor fixes (#2246)	2021-03-17 23:05:05 -07:00
Harshal Sheth	aa6bc15cd7	fix(ingest): various avro codegen fixes (#2232)	2021-03-15 15:27:30 -07:00
Harshal Sheth	6a8fca59f1	feat(ingest): use plugin system based on Python extras (#2224)	2021-03-11 13:41:05 -08:00
Harshal Sheth	4e15feebfd	fix(ingestion): nullable types and timestamp precision (#2212)	2021-03-10 23:29:24 -08:00
Harshal Sheth	6378347f81	feat(ingest): standalone metadata emitters (#2207)	2021-03-10 14:32:12 -08:00
Harshal Sheth	d220647094	feat: add date and time types to SQL model (#2201)	2021-03-09 23:07:20 -08:00
Gabe Lyons	adfe60e97a	feat(tags): adding support for read/write of tags in gms & read-only in react datahub-frontend. (#2164)	2021-03-07 11:26:47 -08:00
Harshal Sheth	e066991f54	fix(ingest): bigquery source and dataset naming fixes (#2161)	2021-03-03 19:49:46 -08:00
Harshal Sheth	0310e97b5b	feat(ingest): capture table descriptions (#2157)	2021-03-02 11:49:37 -08:00
Harshal Sheth	ac064584ae	refactor(ingest): cleanup configuration models (#2134)	2021-02-23 15:55:31 -08:00
Harshal Sheth	76e0594b8b	feat(ingest): add support for LDAP ingestion (#2122)	2021-02-18 20:05:39 -08:00
Harshal Sheth	9336cbc743	test(ingest): verify the output of mssql (#2120)	2021-02-18 14:47:49 -08:00
Harshal Sheth	dfe00bfee8	feat(ingest): improve error reporting for pipelines (#2121)	2021-02-18 11:15:13 -08:00
Harshal Sheth	38f75be8ad	gometa -> datahub	2021-02-15 18:29:27 -08:00
Harshal Sheth	0063c04460	gometa-ingest -> datahub ingest	2021-02-15 18:29:27 -08:00
Harshal Sheth	d0bc3c55db	Setup CI	2021-02-15 18:29:27 -08:00
Harshal Sheth	f807bb44b8	Use type + config everywhere	2021-02-15 18:29:27 -08:00
Harshal Sheth	9332e6b878	Add isort to CI	2021-02-15 18:29:27 -08:00
Harshal Sheth	d483d23fd7	Allow/deny patterns for kafka source	2021-02-15 18:29:27 -08:00
Harshal Sheth	df3e3da45b	More autofixes	2021-02-15 18:29:27 -08:00
Harshal Sheth	7ca018aaa4	Remove unused imports	2021-02-15 18:29:27 -08:00
Harshal Sheth	43d5fac494	Black	2021-02-15 18:29:27 -08:00
Harshal Sheth	1de2bacca4	Fix serde test	2021-02-15 18:29:27 -08:00
Harshal Sheth	107cf043d9	Add a create method to pipeline	2021-02-15 18:29:27 -08:00
Harshal Sheth	b2449eedaa	Add serde tests	2021-02-15 18:29:27 -08:00
Shirshanka Das	7ba81d7087	remove old shell scripts	2021-02-15 18:29:27 -08:00
Harshal Sheth	a87161cad7	Run black formatting on tests	2021-02-15 18:29:27 -08:00
Harshal Sheth	2307c59296	Add support for rich kafka config	2021-02-15 18:29:27 -08:00
Harshal Sheth	b759205b52	Start checking files for mysql test	2021-02-15 18:29:27 -08:00
Harshal Sheth	36874970ce	Finish file sink	2021-02-15 18:29:27 -08:00
Harshal Sheth	23d675663b	Minor test runid renames	2021-02-15 18:29:27 -08:00
Harshal Sheth	8ca8ef2d23	Fix kafka tests	2021-02-15 18:29:27 -08:00
Harshal Sheth	db1ff07e6d	Add reporter everywhere	2021-02-15 18:29:27 -08:00
Harshal Sheth	0929c7cb77	Add test to check that classes are not abstract	2021-02-15 18:29:27 -08:00
Harshal Sheth	9ea03820c4	Enable bare pytest	2021-02-15 18:29:27 -08:00
Shirshanka Das	4e17a24dd4	Adding mysql integration test + source	2021-02-15 18:29:27 -08:00
Harshal Sheth	22a4dbb45f	Cleanup kafka sink schema setup	2021-02-15 18:29:27 -08:00
Harshal Sheth	4b83fc6591	adding allow deny patterns to sql config	2021-02-15 18:29:27 -08:00