12076 Commits

Author SHA1 Message Date
Kevin Hu
bf2775f6cf
feat(ingest): type stubs for boto3 (#2975) 2021-07-28 20:35:35 -07:00
James Lamb
b2dc4b96eb
perf(ingest): remove unused variable in Looker ingestion (#2979) 2021-07-28 20:33:27 -07:00
Harshal Sheth
1c334bb0e0
fix(ingest): infer bigquery project identifier (#2980)
Also refactor by moving get_identifier method into the source rather than leaving it in the config.
2021-07-28 20:31:33 -07:00
Chinmay Bhat
a33770b022
fix(ingest): fix hive ingestion to respect database configuration (#2978) 2021-07-28 20:20:18 -07:00
Harshal Sheth
7ab6355b1c
feat(ingest): stricter deserialization for MCE JSONs (#2976) 2021-07-28 14:50:21 -07:00
Kevin Hu
a1d1dd4269
feat(docs): tutorial for writing a custom transformer (#2959) 2021-07-28 14:38:13 -07:00
Harshal Sheth
14bca4c2f6
fix(ingestion): isolate dependency requirements of airflow hooks (#2977) 2021-07-28 14:23:06 -07:00
Harshal Sheth
c32ba3c5ef
feat(ingest): add timezone validation to bigquery usage (#2962) 2021-07-28 14:16:31 -07:00
Dexter Lee
e2d1acae9d
fix(elasticsearch-setup): fix elasticsearch setup for aws (#2974) 2021-07-28 13:55:15 -07:00
Harshal Sheth
328b098d01
text(ingestion): test multiple python versions in CI (#2952) 2021-07-26 13:25:58 -07:00
Harshal Sheth
fa22b8e17b
feat(ingest): add Airflow TaskFlow example (#2958) 2021-07-26 13:09:25 -07:00
Kevin Hu
662017ef17
fix(ingest): patch lookml types and refactor ingestion sources layout (#2950) 2021-07-26 13:06:52 -07:00
James Lamb
5d396b1487
fix(ingest): match nested LookML files mentioned in 'include' statements (#2957) 2021-07-26 12:38:49 -07:00
Gabe Lyons
79c956abe7
Revert "fix(gql): removing data platform caching in gql (#2947)" (#2956)
This reverts commit 49093ea1cee5e2563d9390ae529af08d287000aa.
2021-07-26 12:17:21 -07:00
Kevin Hu
f82ea3abef
feat(ingest): ingest descriptions from dbt models (#2955) 2021-07-26 10:47:25 -07:00
Harshal Sheth
62d5306a28
fix(ingestion): add more mypy annotations (#2948) 2021-07-25 21:19:25 -07:00
Harshal Sheth
f08cf11a4d
feat(ingestion): test GMS connections before ingestion (#2946) 2021-07-25 21:17:51 -07:00
Gabe Lyons
49093ea1ce
fix(gql): removing data platform caching in gql (#2947) 2021-07-25 21:16:49 -07:00
Harshal Sheth
6135eed757
test(ingestion): fix flaky package discovery test (#2949) 2021-07-25 21:15:22 -07:00
Kevin Hu
5b4178408c
feat(docs): update videos and integration logos (#2951) 2021-07-23 17:08:16 -07:00
Harshal Sheth
7a1ce19d2b
fix(ingestion): resolve test bugs for 3.6 (#2953) 2021-07-23 17:07:13 -07:00
Kevin Hu
59f5d8f8c8
feat(ingest): add logo and platform entry for Glue (#2943) 2021-07-22 22:20:00 -07:00
Harshal Sheth
4bcfe92df0
fix(ingest): handle quotes in lookml properly (#2940) 2021-07-22 22:19:04 -07:00
Kevin Hu
43e61d9628
feat(models): remove versions from metrics and hyperparams (#2938) 2021-07-22 22:18:45 -07:00
Harshal Sheth
b064e51a84
fix(ingestion): make snowflake database names lowercase (#2942) 2021-07-22 21:52:03 -07:00
Harshal Sheth
01982310be
feat(ingest): use urn builders in looker and validate data platforms (#2939) 2021-07-22 21:50:44 -07:00
aseembansal-gogo
d39030dab2
refactor(ingest): make code pythonic (#2941) 2021-07-22 21:47:08 -07:00
Kevin Hu
51bd2c6a81
fix(ingest): allow custom Glue scripts (#2937) 2021-07-22 21:45:48 -07:00
Wei Hou
bac1ae42fc
refactor(datahub-web): removing frontend Ember app (i.e. datahub-web folder) (#2921) 2021-07-22 15:58:30 -07:00
Harshal Sheth
90e05df791
fix(ingest): refactor + fix recursion in lookml file loading logic (#2913) 2021-07-22 15:54:01 -07:00
Harshal Sheth
6e833e761e
feat(ingest): improve bigquery-usage robustness and docs (#2925) 2021-07-22 15:37:55 -07:00
aseembansal-gogo
7dfe61a542
fix(ingest): fix workunit name to be consistent with other sources (#2931) 2021-07-22 15:26:47 -07:00
Kevin Hu
757abfc6f6
fix(ingest): fix browsepaths and ownership urns (#2935) 2021-07-22 15:26:10 -07:00
aseembansal-gogo
4e0da7c319
fix(ingest): glue add support for mapping varchar, decimal types (#2930) 2021-07-22 13:47:26 -07:00
Kevin Hu
6dbc59940a
feat(ingest): refactor mlModel grouping and add browsepaths (#2929) 2021-07-22 13:33:15 -07:00
Harshal Sheth
bc6fdfa2d4
docs(ingest): update looker + docker script docs (#2934) 2021-07-22 13:30:20 -07:00
Harshal Sheth
7dc85a478e
feat(ingest): add make_data_platform_urn method to builder (#2926) 2021-07-22 13:25:07 -07:00
Brian
dd43e6b258
feat(react): surface edited descriptions on search preview for dataset, datajob, dataflow, chart, dashboard (#2932) 2021-07-22 13:20:40 -07:00
Harshal Sheth
3e328d9511
fix(ingest): add quotes to secured kafka yaml config example (#2911) 2021-07-22 13:20:00 -07:00
Kevin Hu
84a1486b89
feat(ingest): dbt aliases (#2927) 2021-07-22 07:57:14 -07:00
Lal Rishav
99790d9052
fix(react): enable relation between glossary term and datasets searchable (#2806)
* 2805 | glossary term linked to dataset is not displayed
2021-07-22 07:25:29 -07:00
Kevin Hu
736249f0c7
feat(ingest): extract SageMaker metrics, hyperparameters, and external URLs (#2910) 2021-07-21 21:30:07 -07:00
aseembansal-gogo
256c82f365
docs: update docs for consistency in naming (#2915) 2021-07-21 12:57:08 -07:00
Kevin Hu
d6875b4f4f
feat(ingest): test dbt ingestion with and without schemas (#2922) 2021-07-21 12:56:31 -07:00
Harshal Sheth
7535cf2b85
fix(ingest): note that views are not supported for Athena (#2924) 2021-07-21 12:48:40 -07:00
Harshal Sheth
ad30f2b8ec
feat(ingestion): support multiple project IDs in bigquery usage stats (#2920) 2021-07-21 12:42:06 -07:00
Harshal Sheth
5f0b4464f5
fix(ingest): pin snowflake sqlalchemy connector (#2923) 2021-07-20 19:28:40 -07:00
Harshal Sheth
416f2a95df
feat(ingest): add support for Oracle spatial types (#2909) 2021-07-20 19:13:49 -07:00
Kevin Hu
859fab8bd3
docs(ingest): update sample recipe and test input for dbt (#2917) 2021-07-20 13:28:37 -07:00
Brian
7e2c4d6e44
feat(mlFeatureTable): add graphql, ui/ux for mlFeatureTable, mlFeature, mlPrimaryKey entities (#2887) 2021-07-20 11:42:21 -07:00