780 Commits

Author SHA1 Message Date
Harshal Sheth
8e573fdb31
fix(ingest): fix druid misconfiguration bug (#2882) 2021-07-14 20:29:23 -07:00
Harshal Sheth
fe6bfc9685
fix(ingest): default to unlimited query log delay in bigquery-usage (#2881) 2021-07-14 20:05:31 -07:00
Harshal Sheth
be39037b10
build(ingest): reduce dependencies for dev install (#2872) 2021-07-14 20:02:48 -07:00
Kevin Hu
904d4410fe
feat(ingest): update golden files only when diff fails (#2869) 2021-07-13 14:59:22 -07:00
Kevin Hu
bc84c82a68
feat(ingest): extract dbt meta fields (#2876) 2021-07-13 14:58:25 -07:00
Kevin Hu
c4e2b9afa2
feat(ingest): add browse paths + dataplatform for Feast features (#2849) 2021-07-12 11:05:18 -07:00
Harshal Sheth
220dfe728c
feat(ingest): support dynamic imports for transfomer methods (#2858) 2021-07-12 11:03:53 -07:00
Kevin Hu
a2106ca9e8
feat(ingest): SageMaker jobs and models (#2830) 2021-07-08 16:16:16 -07:00
Kevin Hu
799b0634e1
fix(ingest): check for dbt materialization before proceeding (#2842) 2021-07-08 15:12:37 -07:00
Harshal Sheth
2f921d15e8
fix(ingest): avoid setting timestamps unless source system provides it (#2843) 2021-07-08 12:11:06 -07:00
Harshal Sheth
6fe663bf6a
feat(ingest): basic support for complex hive types (#2804) 2021-06-30 22:57:13 -07:00
Harshal Sheth
d66381451a
feat(ingest): refactor mce comparison and add pytest update golden files option (#2812) 2021-06-30 16:53:20 -07:00
Harshal Sheth
e51f86a9de
feat(ingest): support ingesting from multiple snowflake dbs (#2793) 2021-06-30 15:54:17 -07:00
Kevin Hu
4da76726d3
feat(ingest): SageMaker feature store ingestion (#2758) 2021-06-29 19:43:31 -07:00
Harshal Sheth
79f60d8b8a
refactor(ingest): remove deprecated methods and warn on deprecated import (#2797) 2021-06-29 11:43:43 -07:00
Harshal Sheth
5e69a4355e
refactor(ingest): use common get_sys_time method (#2782) 2021-06-28 20:40:10 -07:00
Harshal Sheth
19b2a42a00
feat: usage stats (part 2) (#2762)
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-06-24 19:44:59 -07:00
Harshal Sheth
937f02c6bc
feat: usage stats (part 1) (#2750)
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-06-24 17:11:00 -07:00
Kevin Hu
22a2ed81e4
feat(ingest): ingest last-modified from dbt sources.json (#2729) 2021-06-23 13:56:20 -07:00
Kevin Hu
96dde2c734
fix(ci): increase wait-for-it timeout to fix flaky feast test (#2747) 2021-06-23 11:37:31 -07:00
Kevin Hu
fffc61794c
feat(ingest): print docker logs on timeout (#2743) 2021-06-22 17:05:08 -07:00
Kevin Hu
4ddc4c28be
fix(ingest): fix lookml platform URN (#2742) 2021-06-22 13:55:29 -07:00
Kevin Hu
a89094da5b
feat(ingest): add support for Glue ETL jobs (#2687) 2021-06-22 11:33:22 -07:00
Kevin Hu
554e1637c5
fix(ingest): types for dbt (#2716) 2021-06-22 10:37:08 -07:00
Kevin Hu
2c20bce79d
fix(ci): increase Feast docker setup timeout (#2722) 2021-06-22 10:34:01 -07:00
Harshal Sheth
5d93f249b4
feat(ingest): expose additional types to Python via codegen (#2712) 2021-06-17 10:04:28 -07:00
Harshal Sheth
7e9a04479b
test(ingest): simplify docker cleanup commands (#2699) 2021-06-16 16:59:28 -07:00
Harshal Sheth
26dcece8ec
fix(ingest): use looker data platform (#2708) 2021-06-16 16:58:13 -07:00
Kevin Hu
63fe82995b
feat(ingest): Add test case and docs for SQL view ingestion (#2709) 2021-06-16 16:51:57 -07:00
Brian
a5f9b8dfe9
feat(entities): add markdown description update/viewer feature in dataset, datajob, dataflow, chart and dashboard, update ui/ux (#2707) 2021-06-16 15:48:27 -07:00
Harshal Sheth
1b539220d5
feat(ingest): support Oracle service names (#2676) 2021-06-11 17:27:34 -07:00
zack3241
91eb3cc57e
Add get_identifier to hive source in metadata ingestion (#2667) 2021-06-09 15:12:17 -07:00
Kevin Hu
ebdaa0e359
feat(ingest): Feast ingestion integration (#2605)
* Add feast testing setup

* Init Feast test script

* Add feast to dependencies

* Update feast descriptors

* Sort integrations

* Working feast pytest

* Clean up feast docker-compose file

* Expand Feast tests

* Setup feast classes

* Add continuous and bytes data to feature types

* Update field type mapping

* Add PDLs

* Add MLFeatureSetUrn.java

* Comment out feast setup

* Add snapshot file and update inits

* Init Feast golden files generation

* Clean up Feast ingest

* Feast testing comments

* Yield Feature snapshots

* Fix Feature URN naming

* Update feast MCE

* Update Feature URN prefix

* Add MLEntity

* Update golden files with entities

* Specify feast sources

* Add feast source configs

* Working feast docker ingestion

* List entities and features before adding tables

* Add featureset names

* Remove unused

* Rename feast image

* Update README

* Add env to feast URNs

* Fix URN naming

* Remove redundant URN names

* Fix enum backcompatibility

* Move feast testing to docker

* Move URN generators to mce_builder

* Add source for features

* Switch TypeClass -> enum_type

* Rename source -> sourceDataset

* Add local Feast ingest image builds

* Rename Entity -> MLPrimaryKey

* Restore features and keys for each featureset

* Do not json encode source configs

* Remove old source properties from feature sets

* Regenerate golden file

* Fix race condition with Feast tests

* Exclude unknown source

* Update feature datatype enum

* Update README and fix typos

* Fix Entity typo

* Fix path to local docker image

* Specify feast config and version

* Fix feast env variables

* PR fixes

* Refactor feast ingest constants

* Make feature sources optional for back-compatibility

* Remove unused GCP files

* adding docker publish workflow

* Simplify name+namespace in PrimaryKeys

* adding docker publish workflow

* debug

* final attempt

* final final attempt

* final final final commit

* Switch to published ingestion image

* Update name and namespace in java files

* Rename FeatureSet -> FeatureTable

* Regenerate codegen

* Fix initial generation errors

* Update snapshot jsons

* Regenerated schemas

* Fix URN formats

* Revise builds

* Clean up feast URN builders

* Fix naming typos

* Fix Feature Set -> Feature Table

* Fix comments

* PR fixes

* All you need is Urn

* Regenerate snapshots and update validation

* Add UNKNOWN data type

* URNs for source types

* Add note on docker requirement

* Fix typo

* Reorder aspect unions

* Refactor feast ingest functions

* Update snapshot jsons

* Rebuild

Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-06-09 15:07:04 -07:00
Harshal Sheth
31eae24300
fix(ingest): support mssql encryption via ODBC (#2657) 2021-06-04 18:19:11 -07:00
John Joyce
97e9660037
feat: No Code Metadata Modeling (#2629)
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-06-03 13:24:33 -07:00
Harshal Sheth
6b9d0d0129
fix(ingest): include urn as key for kafka emitter (#2634) 2021-06-03 11:04:40 -07:00
Thomas Larsson
b512920022
fix(ingestion): improve robustness of glue ingestion source (#2626)
fixes: #2625

Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-06-01 11:02:52 -07:00
Harshal Sheth
958fe8ea83
feat(ingest): populate inputDatajobs field in airflow integration (#2606) 2021-05-25 22:47:00 -07:00
Fredrik Sannholm
1e0b67ce56
feat(ingestion): Fix looker test (#2601) 2021-05-25 11:15:47 -07:00
Harshal Sheth
1d4bcbe4fb
feat(ingest): add dataset tag transformer (#2580) 2021-05-18 14:43:43 -07:00
Harshal Sheth
f310ff9a4a
test(ingest): use different mysql test port (#2573) 2021-05-17 19:45:34 -07:00
Harshal Sheth
6d875b8241
test(ingest): ensure transformer registry works for aliases (#2572) 2021-05-17 15:08:49 -07:00
Gary Lucas
af4f3b9683
fix(dbt): set target platform and load schema (#2483) 2021-05-17 12:22:52 -07:00
Harshal Sheth
f590f11ff3
fix(ingest): check mypy types for test helpers (#2561) 2021-05-17 11:42:12 -07:00
Harshal Sheth
3dfe3d375b
feat(ingest): add options for Airflow lineage backend (#2557) 2021-05-13 20:02:47 -07:00
Kevin Hu
5ab1cbbbb2
feat(ingest): MongoDB schema inference (#2546) 2021-05-13 19:44:33 -07:00
Fredrik Sannholm
133577557c
feat(ingest): Looker view and dashboard ingestion (#2493) 2021-05-13 11:42:53 -07:00
Harshal Sheth
a671001824
refactor(ingest): move Airflow into datahub_provider module (#2521) 2021-05-12 15:01:11 -07:00
Harshal Sheth
a47400f18e
build(ingest): use gradle in commands + docs (#2531) 2021-05-11 19:03:20 -07:00
Harshal Sheth
2811d23e45
feat(ingest): add a transformer for adding ownership (#2532) 2021-05-11 17:46:39 -07:00