935 Commits

Author SHA1 Message Date
aseembansal-gogo
9dff1b7a1c
fix(ingest): allow database alias, remove extra removal from connect_… (#3352) 2021-10-19 22:29:38 -07:00
mayurinehate
f5d37c0621
feat(ingest): add complex types support in hive and trino source (#3375) 2021-10-18 22:53:51 -07:00
Harshal Sheth
6e5d1fe42f
fix(ingest): switch to avro from deprecated avro-python3 (#3412) 2021-10-18 15:35:27 -07:00
aseembansal-gogo
ebe0b3f7c1
feat(ingest): add mariadb as possible source (#3245) 2021-10-12 23:27:47 -07:00
Harshal Sheth
b43945ce55
test(airflow): remove unneeded execution_date parameter from test (#3368) 2021-10-12 23:25:27 -07:00
Swaroop Jagadish
7dab06519d
fix(ci): fix task instance constructor in airflow tests (#3366) 2021-10-11 20:58:37 -07:00
taufiqibrahim
ed38168f54
feat(ingest): add support for parsing lineage for redash connector (#3267)
Co-authored-by: tibrahim <taufiq.ibrahim@bizzy.co.id>
2021-10-05 23:08:02 -07:00
Sergio Gómez Villamor
696ad2ccc4
feat(ingest): adds replace_existing config to browse path transformer (#3313) 2021-10-01 12:35:16 -07:00
adriaanslechten
185f7e2f5f
feat(ingest): optional custom headers REST emitter (#3290)
Co-authored-by: Adriaan Slechten <adriaan.slechten@klarna.com>
2021-09-23 12:53:39 -07:00
Gabe Lyons
ff527f4bed
feat(foreign keys): add foreign key models (#3275) 2021-09-22 10:29:27 -07:00
aseembansal-gogo
97bed71e1c
feat(ingest): support for env variable in cli (#3215) 2021-09-16 23:18:42 -07:00
aseembansal-gogo
903348e1db
fix(ingest): add missing partition keys in schema for glue sources (#3238) 2021-09-15 21:39:49 -07:00
Sergio Gómez Villamor
9e28423c6d
feat(ingest): add optional config for ownership type in ownership transformers (#3159)
Co-authored-by: Sergio Gómez <sergio.gomez.villamor@adevinta.com>
2021-09-01 22:44:03 -07:00
Fredrik Sannholm
f73725fdf6
feat(ingestion): bring your own SQL parser (#3110) 2021-08-23 20:21:48 -07:00
taufiqibrahim
dd1b81b727
feat(ingest): adding redash source (#2918)
Co-authored-by: tibrahim <taufiq.ibrahim@bizzy.co.id>
2021-08-18 12:03:03 -07:00
Gabe Lyons
b73d0ebb8b
feat: Adding clarity around qualified unions and removing extra lines for structs (#3091)
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2021-08-12 16:25:44 -07:00
rslanka
8844240328
feat: Adding support for nested schemas in ingestion and visualization (#3079) 2021-08-11 15:47:18 -07:00
Rahul Jain
0e0198f385
feat(ingest): Added support for "add dataset ownership by regex match" (#2990) 2021-08-10 11:03:16 -07:00
Kevin Hu
5891482114
fix(ingest): apply case insensitive regex matching by default (#3067) 2021-08-10 10:35:57 -07:00
aseembansal-gogo
9ee091f518
feat(ingest): add transformers to clear dataset ownership, mark status, add browse paths (#2986) 2021-08-09 09:43:46 -07:00
Dexter Lee
8316316daa
fix(ingest): add default configurable timeout for rest emitter (#3055) 2021-08-08 22:30:55 -07:00
aseembansal-gogo
ac345e0ade
feat(ingest): add underlying platform for glue (#3035) 2021-08-06 10:19:21 -07:00
Dexter Lee
51a2669332
feat(ingest): add kafka emitters for MetadataChangeProposal format (#3018) 2021-08-03 11:17:16 -07:00
John Joyce
352a0abf8d
Introducing TimeSeries Aspects + Dataset Profile (Stats) Aspect (#2983)
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-07-30 17:41:03 -07:00
Gabe Lyons
aa253f5b3b
feat(deletes): add run commands (list, show, rollback) to datahub ingest (#2960) 2021-07-29 20:04:40 -07:00
Gabe Lyons
d9d03c9bd9
feat(ml-model): adding ml models and ml model groups (#2961) 2021-07-28 20:39:05 -07:00
Kevin Hu
bf2775f6cf
feat(ingest): type stubs for boto3 (#2975) 2021-07-28 20:35:35 -07:00
Chinmay Bhat
a33770b022
fix(ingest): fix hive ingestion to respect database configuration (#2978) 2021-07-28 20:20:18 -07:00
Harshal Sheth
7ab6355b1c
feat(ingest): stricter deserialization for MCE JSONs (#2976) 2021-07-28 14:50:21 -07:00
Harshal Sheth
14bca4c2f6
fix(ingestion): isolate dependency requirements of airflow hooks (#2977) 2021-07-28 14:23:06 -07:00
Harshal Sheth
fa22b8e17b
feat(ingest): add Airflow TaskFlow example (#2958) 2021-07-26 13:09:25 -07:00
Kevin Hu
662017ef17
fix(ingest): patch lookml types and refactor ingestion sources layout (#2950) 2021-07-26 13:06:52 -07:00
Harshal Sheth
6135eed757
test(ingestion): fix flaky package discovery test (#2949) 2021-07-25 21:15:22 -07:00
Harshal Sheth
7a1ce19d2b
fix(ingestion): resolve test bugs for 3.6 (#2953) 2021-07-23 17:07:13 -07:00
Kevin Hu
43e61d9628
feat(models): remove versions from metrics and hyperparams (#2938) 2021-07-22 22:18:45 -07:00
Kevin Hu
757abfc6f6
fix(ingest): fix browsepaths and ownership urns (#2935) 2021-07-22 15:26:10 -07:00
Kevin Hu
6dbc59940a
feat(ingest): refactor mlModel grouping and add browsepaths (#2929) 2021-07-22 13:33:15 -07:00
Harshal Sheth
7dc85a478e
feat(ingest): add make_data_platform_urn method to builder (#2926) 2021-07-22 13:25:07 -07:00
Kevin Hu
736249f0c7
feat(ingest): extract SageMaker metrics, hyperparameters, and external URLs (#2910) 2021-07-21 21:30:07 -07:00
Harshal Sheth
5f0b4464f5
fix(ingest): pin snowflake sqlalchemy connector (#2923) 2021-07-20 19:28:40 -07:00
Harshal Sheth
416f2a95df
feat(ingest): add support for Oracle spatial types (#2909) 2021-07-20 19:13:49 -07:00
aseembansal-gogo
6e1b2cf4f7
feat(ingest): Add option to change name of database for postgres (#2898) 2021-07-20 07:01:42 -07:00
Kevin Hu
6abd5e191a
feat(ingest): lineage for SageMaker model endpoints and groups (#2894) 2021-07-19 11:30:43 -07:00
Harshal Sheth
45e931dbac
feat(ingest): add can_add_aspect method for MCEs (#2905) 2021-07-17 20:02:07 -07:00
Kevin Hu
44ed2f3684
feat(ingest): extract lineage between SageMaker jobs and models (#2868) 2021-07-15 18:56:13 -07:00
Harshal Sheth
8e573fdb31
fix(ingest): fix druid misconfiguration bug (#2882) 2021-07-14 20:29:23 -07:00
Harshal Sheth
fe6bfc9685
fix(ingest): default to unlimited query log delay in bigquery-usage (#2881) 2021-07-14 20:05:31 -07:00
Harshal Sheth
be39037b10
build(ingest): reduce dependencies for dev install (#2872) 2021-07-14 20:02:48 -07:00
Harshal Sheth
220dfe728c
feat(ingest): support dynamic imports for transfomer methods (#2858) 2021-07-12 11:03:53 -07:00
Kevin Hu
a2106ca9e8
feat(ingest): SageMaker jobs and models (#2830) 2021-07-08 16:16:16 -07:00