18 Commits

Author SHA1 Message Date
Harshal Sheth
cf3db168ac
feat(ingest): start simplifying stateful ingestion state (#6740) 2022-12-13 10:05:57 +01:00
Ravindra Lanka
b8941ab190
feat(ingestion): Add fail-safe stale entity removal via configurable 'fail_safe_threshold' param. (#6027) 2022-09-22 16:09:22 -07:00
skrydal
f61a040555
feat(ingestion) Add more info to glue entities (#5874)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-09-14 12:25:09 -07:00
Ravindra Lanka
ee68f09624
feat(ingestion): Refactor standard state-handling tasks into a common handler that are common across all stateful ingestion sources. (#5766) 2022-09-14 09:30:42 -07:00
Amanda Hernando
337087cac0
feat(ingest): glue - add stateful ingestion (#5553) 2022-08-15 20:50:45 -07:00
Jordan Wolinsky
bbac4a7a11
feat(ingestion): glue/s3 - Ingest Tags from s3 bucket on an AWS Glue job and S3 Data Lake Ingest Job (#4689) 2022-04-29 10:09:06 +02:00
Sergio Gómez Villamor
bdf17f551e
feat(ingest): glue - adds platform instance capability (#4130) 2022-03-30 18:50:26 -07:00
Ravindra Lanka
1efe04f88a
feat(ingest): glue - support for nested structs (#3895) 2022-01-17 14:21:53 -08:00
aseembansal-gogo
ac345e0ade
feat(ingest): add underlying platform for glue (#3035) 2021-08-06 10:19:21 -07:00
John Joyce
352a0abf8d
Introducing TimeSeries Aspects + Dataset Profile (Stats) Aspect (#2983)
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-07-30 17:41:03 -07:00
Kevin Hu
662017ef17
fix(ingest): patch lookml types and refactor ingestion sources layout (#2950) 2021-07-26 13:06:52 -07:00
Harshal Sheth
d66381451a
feat(ingest): refactor mce comparison and add pytest update golden files option (#2812) 2021-06-30 16:53:20 -07:00
Kevin Hu
a89094da5b
feat(ingest): add support for Glue ETL jobs (#2687) 2021-06-22 11:33:22 -07:00
Thomas Larsson
b512920022
fix(ingestion): improve robustness of glue ingestion source (#2626)
fixes: #2625

Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-06-01 11:02:52 -07:00
Harshal Sheth
d415234a8c
fix(ingest): fields with defaults should be optional (#2461) 2021-04-26 16:45:48 -07:00
Harshal Sheth
ffe03e6758
fix(ingest): streamline codegen init methods (#2400) 2021-04-14 19:25:57 -07:00
Harshal Sheth
bd78b84bd3
feat(ingest): start airflow integration + metadata builders (#2331) 2021-04-05 19:11:28 -07:00
amy m
759288161c
feat(ingest): adding support for AWS Glue (#2319)
Co-authored-by: Harry Nash <harrywilliamnash@gmail.com>
2021-04-04 11:00:27 -07:00