15 Commits

Author SHA1 Message Date
Mayuri Nehate
e79b4e8c2b
feat(ingest): s3 - add status aspect for detected s3 datasets (#6402) 2022-11-13 17:29:42 -08:00
Harshal Sheth
09616ee2b3
feat(ingest): include instance in container dataPlatform when provided (#6083) 2022-10-13 11:29:54 -07:00
Harshal Sheth
e70c0ac4b6
feat(ingest): include raw s3 paths if s3 source (#6168) 2022-10-11 15:55:00 -07:00
Shirshanka Das
e9c4c823d8
fix(ingest): bigquery-beta - ensure that status aspect is emitted for… (#6154)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-10-08 16:00:45 -07:00
Harshal Sheth
3c0f63c50a
fix(ingest): hide deprecated path_spec option from config (#5944) 2022-10-04 12:14:00 -07:00
Mayuri Nehate
a14617b6a4
fix(ingest): continue validation of s3 path_specs even if platform is set (#5951) 2022-09-16 12:03:57 -07:00
Ravindra Lanka
228f3b50ea
feat(ingestion): send reports of ingestion runs to datahub (#5639)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-08-19 09:08:17 -07:00
Jordan Wolinsky
3a86ff3485
Fix profiling when using {table}. (#5531)
* profiling fix for when using {table}

Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-08-08 13:16:59 -07:00
Mayuri Nehate
2c48329810
feat(model): dashboard usage model, is_null condition added (#5397) 2022-07-15 15:37:06 +05:30
Aseem Bansal
4541379024
feat(build): changes to decrease build time, cancel runs in case of multiple commits (#5187) 2022-06-17 18:05:10 +05:30
Tamas Nemeth
be91e2341f
feat(ingest): s3 - speeding up ingestion with sampling (#4927) 2022-05-24 22:17:10 -07:00
Tamas Nemeth
56ee4d9651
feat(ingest): s3 - add support for multiple pathspecs in one recipe (#4777) 2022-05-05 10:09:47 -07:00
mayurinehate
c34a1ba735
fix(s3): improved handling for corner cases (#4774) 2022-04-29 12:25:41 -07:00
Jordan Wolinsky
bbac4a7a11
feat(ingestion): glue/s3 - Ingest Tags from s3 bucket on an AWS Glue job and S3 Data Lake Ingest Job (#4689) 2022-04-29 10:09:06 +02:00
MugdhaHardikar-GSLab
37aedfc87c
feat(s3): add s3 source (#4490)
* feat(data-lake): add containers and folder level dataset support

* docs(data-lake): Update readme for data lake

* doc(data-lake): fix examples, update doc

* lint fix

* feat(s3): add s3 source, restore old data-lake source

Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
2022-03-29 11:52:57 +02:00