John Joyce
8c9c696cdd
feat(ingest): Adding an Okta Integration to extract Users, Groups, Group Membership ( #3043 )
2021-08-11 18:49:16 -07:00
rslanka
8844240328
feat: Adding support for nested schemas in ingestion and visualization ( #3079 )
2021-08-11 15:47:18 -07:00
Dexter Lee
0956b79b4d
fix(ingest): stop looker source from unnecessarily filling out owners ( #3056 )
2021-08-08 23:07:05 -07:00
Kevin Hu
3701931363
feat(ingest): extract dbt versions into custom properties ( #3021 )
2021-08-03 12:13:49 -07:00
John Joyce
352a0abf8d
Introducing TimeSeries Aspects + Dataset Profile (Stats) Aspect ( #2983 )
...
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-07-30 17:41:03 -07:00
Gabe Lyons
aa253f5b3b
feat(deletes): add run commands (list, show, rollback) to datahub ingest ( #2960 )
2021-07-29 20:04:40 -07:00
Chinmay Bhat
a33770b022
fix(ingest): fix hive ingestion to respect database configuration ( #2978 )
2021-07-28 20:20:18 -07:00
Harshal Sheth
c32ba3c5ef
feat(ingest): add timezone validation to bigquery usage ( #2962 )
2021-07-28 14:16:31 -07:00
Kevin Hu
662017ef17
fix(ingest): patch lookml types and refactor ingestion sources layout ( #2950 )
2021-07-26 13:06:52 -07:00
Kevin Hu
f82ea3abef
feat(ingest): ingest descriptions from dbt models ( #2955 )
2021-07-26 10:47:25 -07:00
Harshal Sheth
90e05df791
fix(ingest): refactor + fix recursion in lookml file loading logic ( #2913 )
2021-07-22 15:54:01 -07:00
Harshal Sheth
6e833e761e
feat(ingest): improve bigquery-usage robustness and docs ( #2925 )
2021-07-22 15:37:55 -07:00
Kevin Hu
84a1486b89
feat(ingest): dbt aliases ( #2927 )
2021-07-22 07:57:14 -07:00
Kevin Hu
d6875b4f4f
feat(ingest): test dbt ingestion with and without schemas ( #2922 )
2021-07-21 12:56:31 -07:00
Harshal Sheth
ad30f2b8ec
feat(ingestion): support multiple project IDs in bigquery usage stats ( #2920 )
2021-07-21 12:42:06 -07:00
Kevin Hu
859fab8bd3
docs(ingest): update sample recipe and test input for dbt ( #2917 )
2021-07-20 13:28:37 -07:00
Harshal Sheth
89bdfdf1a8
fix(ingest): correct globs in lookml model discovery ( #2891 )
2021-07-16 11:59:50 -07:00
Harshal Sheth
be39037b10
build(ingest): reduce dependencies for dev install ( #2872 )
2021-07-14 20:02:48 -07:00
Kevin Hu
bc84c82a68
feat(ingest): extract dbt meta fields ( #2876 )
2021-07-13 14:58:25 -07:00
Kevin Hu
c4e2b9afa2
feat(ingest): add browse paths + dataplatform for Feast features ( #2849 )
2021-07-12 11:05:18 -07:00
Kevin Hu
799b0634e1
fix(ingest): check for dbt materialization before proceeding ( #2842 )
2021-07-08 15:12:37 -07:00
Harshal Sheth
2f921d15e8
fix(ingest): avoid setting timestamps unless source system provides it ( #2843 )
2021-07-08 12:11:06 -07:00
Harshal Sheth
6fe663bf6a
feat(ingest): basic support for complex hive types ( #2804 )
2021-06-30 22:57:13 -07:00
Harshal Sheth
d66381451a
feat(ingest): refactor mce comparison and add pytest update golden files option ( #2812 )
2021-06-30 16:53:20 -07:00
Kevin Hu
4da76726d3
feat(ingest): SageMaker feature store ingestion ( #2758 )
2021-06-29 19:43:31 -07:00
Harshal Sheth
19b2a42a00
feat: usage stats (part 2) ( #2762 )
...
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-06-24 19:44:59 -07:00
Harshal Sheth
937f02c6bc
feat: usage stats (part 1) ( #2750 )
...
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-06-24 17:11:00 -07:00
Kevin Hu
22a2ed81e4
feat(ingest): ingest last-modified from dbt sources.json ( #2729 )
2021-06-23 13:56:20 -07:00
Kevin Hu
96dde2c734
fix(ci): increase wait-for-it timeout to fix flaky feast test ( #2747 )
2021-06-23 11:37:31 -07:00
Kevin Hu
4ddc4c28be
fix(ingest): fix lookml platform URN ( #2742 )
2021-06-22 13:55:29 -07:00
Kevin Hu
a89094da5b
feat(ingest): add support for Glue ETL jobs ( #2687 )
2021-06-22 11:33:22 -07:00
Kevin Hu
554e1637c5
fix(ingest): types for dbt ( #2716 )
2021-06-22 10:37:08 -07:00
Kevin Hu
2c20bce79d
fix(ci): increase Feast docker setup timeout ( #2722 )
2021-06-22 10:34:01 -07:00
Harshal Sheth
26dcece8ec
fix(ingest): use looker data platform ( #2708 )
2021-06-16 16:58:13 -07:00
Kevin Hu
63fe82995b
feat(ingest): Add test case and docs for SQL view ingestion ( #2709 )
2021-06-16 16:51:57 -07:00
Kevin Hu
ebdaa0e359
feat(ingest): Feast ingestion integration ( #2605 )
...
* Add feast testing setup
* Init Feast test script
* Add feast to dependencies
* Update feast descriptors
* Sort integrations
* Working feast pytest
* Clean up feast docker-compose file
* Expand Feast tests
* Setup feast classes
* Add continuous and bytes data to feature types
* Update field type mapping
* Add PDLs
* Add MLFeatureSetUrn.java
* Comment out feast setup
* Add snapshot file and update inits
* Init Feast golden files generation
* Clean up Feast ingest
* Feast testing comments
* Yield Feature snapshots
* Fix Feature URN naming
* Update feast MCE
* Update Feature URN prefix
* Add MLEntity
* Update golden files with entities
* Specify feast sources
* Add feast source configs
* Working feast docker ingestion
* List entities and features before adding tables
* Add featureset names
* Remove unused
* Rename feast image
* Update README
* Add env to feast URNs
* Fix URN naming
* Remove redundant URN names
* Fix enum backcompatibility
* Move feast testing to docker
* Move URN generators to mce_builder
* Add source for features
* Switch TypeClass -> enum_type
* Rename source -> sourceDataset
* Add local Feast ingest image builds
* Rename Entity -> MLPrimaryKey
* Restore features and keys for each featureset
* Do not json encode source configs
* Remove old source properties from feature sets
* Regenerate golden file
* Fix race condition with Feast tests
* Exclude unknown source
* Update feature datatype enum
* Update README and fix typos
* Fix Entity typo
* Fix path to local docker image
* Specify feast config and version
* Fix feast env variables
* PR fixes
* Refactor feast ingest constants
* Make feature sources optional for back-compatibility
* Remove unused GCP files
* adding docker publish workflow
* Simplify name+namespace in PrimaryKeys
* adding docker publish workflow
* debug
* final attempt
* final final attempt
* final final final commit
* Switch to published ingestion image
* Update name and namespace in java files
* Rename FeatureSet -> FeatureTable
* Regenerate codegen
* Fix initial generation errors
* Update snapshot jsons
* Regenerated schemas
* Fix URN formats
* Revise builds
* Clean up feast URN builders
* Fix naming typos
* Fix Feature Set -> Feature Table
* Fix comments
* PR fixes
* All you need is Urn
* Regenerate snapshots and update validation
* Add UNKNOWN data type
* URNs for source types
* Add note on docker requirement
* Fix typo
* Reorder aspect unions
* Refactor feast ingest functions
* Update snapshot jsons
* Rebuild
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-06-09 15:07:04 -07:00
Harshal Sheth
31eae24300
fix(ingest): support mssql encryption via ODBC ( #2657 )
2021-06-04 18:19:11 -07:00
Fredrik Sannholm
1e0b67ce56
feat(ingestion): Fix looker test ( #2601 )
2021-05-25 11:15:47 -07:00
Harshal Sheth
f310ff9a4a
test(ingest): use different mysql test port ( #2573 )
2021-05-17 19:45:34 -07:00
Gary Lucas
af4f3b9683
fix(dbt): set target platform and load schema ( #2483 )
2021-05-17 12:22:52 -07:00
Kevin Hu
5ab1cbbbb2
feat(ingest): MongoDB schema inference ( #2546 )
2021-05-13 19:44:33 -07:00
Fredrik Sannholm
133577557c
feat(ingest): Looker view and dashboard ingestion ( #2493 )
2021-05-13 11:42:53 -07:00
shakti-garg
8ed14a62e2
feat(business_glossary): add new entity business term and its relationship with dataset and its fields ( #2228 )
...
Co-authored-by: shubham.garg <shubham.garg@thoughtworks.com>
2021-05-10 13:20:23 -07:00
Gabe Lyons
851e00ba9f
feat(lineage): implement support for datasets, charts and dashboards downstream lineage fetching in a generic way ( #2397 )
...
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Brian <brianwebtek@gmail.com>
Co-authored-by: John Joyce <john@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2021-04-23 00:18:39 -07:00
adriaanslechten
1295c44615
feat(ingest) LDAP groups ingestion ( #2434 )
2021-04-22 13:56:30 -07:00
Harshal Sheth
ffe03e6758
fix(ingest): streamline codegen init methods ( #2400 )
2021-04-14 19:25:57 -07:00
Harshal Sheth
a11329d5b8
refactor(ingest): update test harness to use a compose file per test ( #2392 )
2021-04-13 17:30:24 -07:00
Gary Lucas
9d00e8bb2e
fix(test): dbt-manifest files ( #2330 )
2021-04-02 10:59:20 -07:00
Harshal Sheth
cb24628886
feat(ingest): verify dynamic registry types at runtime ( #2327 )
2021-04-01 12:15:05 -07:00
Harshal Sheth
a921d0deae
feat(ingest): MongoDB ingestion source ( #2289 )
2021-03-23 20:15:44 -07:00
