3904 Commits

Author SHA1 Message Date
Harshal Sheth
fe6bfc9685
fix(ingest): default to unlimited query log delay in bigquery-usage (#2881) 2021-07-14 20:05:31 -07:00
Harshal Sheth
be39037b10
build(ingest): reduce dependencies for dev install (#2872) 2021-07-14 20:02:48 -07:00
Harshal Sheth
f01bb8bcd0
build(ingestion): add version prompt to release script (#2866) 2021-07-13 15:02:14 -07:00
Kevin Hu
904d4410fe
feat(ingest): update golden files only when diff fails (#2869) 2021-07-13 14:59:22 -07:00
Kevin Hu
bc84c82a68
feat(ingest): extract dbt meta fields (#2876) 2021-07-13 14:58:25 -07:00
Kevin Hu
c4e2b9afa2
feat(ingest): add browse paths + dataplatform for Feast features (#2849) 2021-07-12 11:05:18 -07:00
Harshal Sheth
220dfe728c
feat(ingest): support dynamic imports for transfomer methods (#2858) 2021-07-12 11:03:53 -07:00
Harshal Sheth
fd02f711cd
fix(ingest): handle 'fields' list missing in bigquery-usage (#2844) 2021-07-08 17:10:11 -07:00
Harshal Sheth
f6e72291c3
fix(ingest): delete pycache files when running clean (#2852) 2021-07-08 17:08:35 -07:00
Harshal Sheth
83fd69d46d
docs(ingest): remove hanging sentence from docs (#2853) 2021-07-08 16:37:32 -07:00
Kevin Hu
a2106ca9e8
feat(ingest): SageMaker jobs and models (#2830) 2021-07-08 16:16:16 -07:00
Kevin Hu
799b0634e1
fix(ingest): check for dbt materialization before proceeding (#2842) 2021-07-08 15:12:37 -07:00
Harshal Sheth
74e34dddfc
feat(ingest): prettify stack traces in CLI (#2845) 2021-07-08 13:29:34 -07:00
Fredrik Sannholm
c2f2973c1b
fix(ingest): Fix glob pattern and handle possible recursion in lookml (#2851) 2021-07-08 12:26:11 -07:00
Harshal Sheth
2f921d15e8
fix(ingest): avoid setting timestamps unless source system provides it (#2843) 2021-07-08 12:11:06 -07:00
Kevin Hu
739e0efa61
fix(ingest): do not fail dbt ingestion when encountering missing nodes (#2833) 2021-07-06 19:32:00 -07:00
Harshal Sheth
2d1dd95a84
docs(ingest): clarify that the Kafka options are pass-through (#2837) 2021-07-06 19:22:35 -07:00
Harshal Sheth
523fa32eb6
fix(ingest): various BigQuery source fixes (#2836) 2021-07-06 19:18:18 -07:00
Harshal Sheth
6b59cdeb82
fix(ingest): mask password in info-level logs (#2835) 2021-07-06 16:41:54 -07:00
Harshal Sheth
288d17f07e
docs(ingest): update links to Kafka docs (#2834) 2021-07-06 15:33:52 -07:00
Harshal Sheth
2a9fdfc181
fix(cli): change docker nuke to also remove stopped containers (#2825) 2021-07-02 20:27:28 -07:00
Harshal Sheth
1b4788537d
feat(ci): separate metadata-ingestion into a separate workflow (#2828) 2021-07-02 20:26:00 -07:00
Harshal Sheth
6ef0bf0dfd
fix(ingest): convert superset timestamps to micros (#2827)
* Previous values were in seconds, which rendered incorrectly in the UI.
2021-07-02 20:21:56 -07:00
Gabe Lyons
8d6362419f
docs(elastic-for-graph): Add migrating from neo4j to elastic instructions (#2826) 2021-07-02 20:19:33 -07:00
Harshal Sheth
4c39d86f63
docs(ingest): add extra info for Redshift behind a proxy (#2817) 2021-07-02 10:31:14 -07:00
Harshal Sheth
d6c3043bc4
fix(ingest): view handling resilience for redshift (#2816) 2021-07-02 10:26:50 -07:00
Harshal Sheth
6fe663bf6a
feat(ingest): basic support for complex hive types (#2804) 2021-06-30 22:57:13 -07:00
Harshal Sheth
d66381451a
feat(ingest): refactor mce comparison and add pytest update golden files option (#2812) 2021-06-30 16:53:20 -07:00
Harshal Sheth
e51f86a9de
feat(ingest): support ingesting from multiple snowflake dbs (#2793) 2021-06-30 15:54:17 -07:00
Harshal Sheth
906ad67a37
refactor(ingest): extract sqlalchemy uri generation logic (#2786) 2021-06-30 09:33:41 -07:00
Harshal Sheth
ef75bc6e5c
fix(ingest): better warnings and error handling for rest sink (#2800) 2021-06-29 22:37:44 -07:00
Kevin Hu
4da76726d3
feat(ingest): SageMaker feature store ingestion (#2758) 2021-06-29 19:43:31 -07:00
Remi
2aa95ec750
feat(ingest): Improve lookml sql derived tables detection, add cascading derived tables to lineage (#2770) 2021-06-29 19:41:34 -07:00
Harshal Sheth
c8fe8d4026
fix(ingest): quote table names in hive (#2801) 2021-06-29 17:51:01 -07:00
Harshal Sheth
57ddb41086
fix(ingest): handle case when view definition handler is not implemented (#2796) 2021-06-29 17:49:57 -07:00
Harshal Sheth
79f60d8b8a
refactor(ingest): remove deprecated methods and warn on deprecated import (#2797) 2021-06-29 11:43:43 -07:00
Kevin Hu
7083de9b4f
fix(ingest): use correct platform for MongoDB ingestion (#2783) 2021-06-29 10:16:58 -07:00
Kevin Hu
14294e8f89
fix(docs): links to Feast entities (#2780) 2021-06-29 08:12:11 -07:00
Harshal Sheth
5e69a4355e
refactor(ingest): use common get_sys_time method (#2782) 2021-06-28 20:40:10 -07:00
Kevin Hu
09bbcea0a8
feat(ingest): add non-random sampling for mongo (#2778) 2021-06-27 23:40:17 -07:00
Harshal Sheth
c05459b446
docs: upgrade docusaurus, minor ingestion updates (#2774) 2021-06-27 23:38:38 -07:00
Harshal Sheth
424139145b
docs(ingest): move usage stats docs into the "sources" section (#2766) 2021-06-24 23:03:26 -07:00
Harshal Sheth
19b2a42a00
feat: usage stats (part 2) (#2762)
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-06-24 19:44:59 -07:00
Harshal Sheth
937f02c6bc
feat: usage stats (part 1) (#2750)
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-06-24 17:11:00 -07:00
Gabe Lyons
62ba937bb7
feat(elastic-as-graph): defaulting to elastic in quickstart (#2753) 2021-06-24 15:44:03 -07:00
Harshal Sheth
82468016ae
fix(docker): use head tag for datahub-ingestion (#2760) 2021-06-24 13:16:49 -07:00
Remi
91f5d4f59a
feat(ingest): add option to specify source platform database in lookml ingestion (#2749) 2021-06-23 16:16:20 -07:00
Kevin Hu
22a2ed81e4
feat(ingest): ingest last-modified from dbt sources.json (#2729) 2021-06-23 13:56:20 -07:00
John Joyce
8fc194717c
feat(quickstart): remove orphaned docker containers on quickstart through cli (#2748) 2021-06-23 13:46:19 -07:00
Kevin Hu
96dde2c734
fix(ci): increase wait-for-it timeout to fix flaky feast test (#2747) 2021-06-23 11:37:31 -07:00