Kevin Hu
2c20bce79d
fix(ci): increase Feast docker setup timeout ( #2722 )
2021-06-22 10:34:01 -07:00
Harshal Sheth
26dcece8ec
fix(ingest): use looker data platform ( #2708 )
2021-06-16 16:58:13 -07:00
Kevin Hu
63fe82995b
feat(ingest): Add test case and docs for SQL view ingestion ( #2709 )
2021-06-16 16:51:57 -07:00
Kevin Hu
ebdaa0e359
feat(ingest): Feast ingestion integration ( #2605 )
...
* Add feast testing setup
* Init Feast test script
* Add feast to dependencies
* Update feast descriptors
* Sort integrations
* Working feast pytest
* Clean up feast docker-compose file
* Expand Feast tests
* Setup feast classes
* Add continuous and bytes data to feature types
* Update field type mapping
* Add PDLs
* Add MLFeatureSetUrn.java
* Comment out feast setup
* Add snapshot file and update inits
* Init Feast golden files generation
* Clean up Feast ingest
* Feast testing comments
* Yield Feature snapshots
* Fix Feature URN naming
* Update feast MCE
* Update Feature URN prefix
* Add MLEntity
* Update golden files with entities
* Specify feast sources
* Add feast source configs
* Working feast docker ingestion
* List entities and features before adding tables
* Add featureset names
* Remove unused
* Rename feast image
* Update README
* Add env to feast URNs
* Fix URN naming
* Remove redundant URN names
* Fix enum backcompatibility
* Move feast testing to docker
* Move URN generators to mce_builder
* Add source for features
* Switch TypeClass -> enum_type
* Rename source -> sourceDataset
* Add local Feast ingest image builds
* Rename Entity -> MLPrimaryKey
* Restore features and keys for each featureset
* Do not json encode source configs
* Remove old source properties from feature sets
* Regenerate golden file
* Fix race condition with Feast tests
* Exclude unknown source
* Update feature datatype enum
* Update README and fix typos
* Fix Entity typo
* Fix path to local docker image
* Specify feast config and version
* Fix feast env variables
* PR fixes
* Refactor feast ingest constants
* Make feature sources optional for back-compatibility
* Remove unused GCP files
* adding docker publish workflow
* Simplify name+namespace in PrimaryKeys
* adding docker publish workflow
* debug
* final attempt
* final final attempt
* final final final commit
* Switch to published ingestion image
* Update name and namespace in java files
* Rename FeatureSet -> FeatureTable
* Regenerate codegen
* Fix initial generation errors
* Update snapshot jsons
* Regenerated schemas
* Fix URN formats
* Revise builds
* Clean up feast URN builders
* Fix naming typos
* Fix Feature Set -> Feature Table
* Fix comments
* PR fixes
* All you need is Urn
* Regenerate snapshots and update validation
* Add UNKNOWN data type
* URNs for source types
* Add note on docker requirement
* Fix typo
* Reorder aspect unions
* Refactor feast ingest functions
* Update snapshot jsons
* Rebuild
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-06-09 15:07:04 -07:00
Harshal Sheth
31eae24300
fix(ingest): support mssql encryption via ODBC ( #2657 )
2021-06-04 18:19:11 -07:00
Fredrik Sannholm
1e0b67ce56
feat(ingestion): Fix looker test ( #2601 )
2021-05-25 11:15:47 -07:00
Harshal Sheth
f310ff9a4a
test(ingest): use different mysql test port ( #2573 )
2021-05-17 19:45:34 -07:00
Gary Lucas
af4f3b9683
fix(dbt): set target platform and load schema ( #2483 )
2021-05-17 12:22:52 -07:00
Kevin Hu
5ab1cbbbb2
feat(ingest): MongoDB schema inference ( #2546 )
2021-05-13 19:44:33 -07:00
Fredrik Sannholm
133577557c
feat(ingest): Looker view and dashboard ingestion ( #2493 )
2021-05-13 11:42:53 -07:00
shakti-garg
8ed14a62e2
feat(business_glossary): add new entity business term and its relationship with dataset and its fields ( #2228 )
...
Co-authored-by: shubham.garg <shubham.garg@thoughtworks.com>
2021-05-10 13:20:23 -07:00
Gabe Lyons
851e00ba9f
feat(lineage): implement support for datasets, charts and dashboards downstream lineage fetching in a generic way ( #2397 )
...
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Brian <brianwebtek@gmail.com>
Co-authored-by: John Joyce <john@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2021-04-23 00:18:39 -07:00
adriaanslechten
1295c44615
feat(ingest) LDAP groups ingestion ( #2434 )
2021-04-22 13:56:30 -07:00
Harshal Sheth
ffe03e6758
fix(ingest): streamline codegen init methods ( #2400 )
2021-04-14 19:25:57 -07:00
Harshal Sheth
a11329d5b8
refactor(ingest): update test harness to use a compose file per test ( #2392 )
2021-04-13 17:30:24 -07:00
Gary Lucas
9d00e8bb2e
fix(test): dbt-manifest files ( #2330 )
2021-04-02 10:59:20 -07:00
Harshal Sheth
cb24628886
feat(ingest): verify dynamic registry types at runtime ( #2327 )
2021-04-01 12:15:05 -07:00
Harshal Sheth
a921d0deae
feat(ingest): MongoDB ingestion source ( #2289 )
2021-03-23 20:15:44 -07:00
Gary Lucas
922307f8ec
feat(ingest): Add DBT source to metadata-ingestion ( #2247 )
2021-03-22 23:11:29 -07:00
Harshal Sheth
aa6bc15cd7
fix(ingest): various avro codegen fixes ( #2232 )
2021-03-15 15:27:30 -07:00
Harshal Sheth
4e15feebfd
fix(ingestion): nullable types and timestamp precision ( #2212 )
2021-03-10 23:29:24 -08:00
Harshal Sheth
d220647094
feat: add date and time types to SQL model ( #2201 )
2021-03-09 23:07:20 -08:00
Gabe Lyons
adfe60e97a
feat(tags): adding support for read/write of tags in gms & read-only in react datahub-frontend. ( #2164 )
2021-03-07 11:26:47 -08:00
Harshal Sheth
e066991f54
fix(ingest): bigquery source and dataset naming fixes ( #2161 )
2021-03-03 19:49:46 -08:00
Harshal Sheth
0310e97b5b
feat(ingest): capture table descriptions ( #2157 )
2021-03-02 11:49:37 -08:00
Harshal Sheth
ac064584ae
refactor(ingest): cleanup configuration models ( #2134 )
2021-02-23 15:55:31 -08:00
Harshal Sheth
76e0594b8b
feat(ingest): add support for LDAP ingestion ( #2122 )
2021-02-18 20:05:39 -08:00
Harshal Sheth
9336cbc743
test(ingest): verify the output of mssql ( #2120 )
2021-02-18 14:47:49 -08:00
Harshal Sheth
0063c04460
gometa-ingest -> datahub ingest
2021-02-15 18:29:27 -08:00
Harshal Sheth
d0bc3c55db
Setup CI
2021-02-15 18:29:27 -08:00
Harshal Sheth
f807bb44b8
Use type + config everywhere
2021-02-15 18:29:27 -08:00
Harshal Sheth
9332e6b878
Add isort to CI
2021-02-15 18:29:27 -08:00
Harshal Sheth
df3e3da45b
More autofixes
2021-02-15 18:29:27 -08:00
Harshal Sheth
7ca018aaa4
Remove unused imports
2021-02-15 18:29:27 -08:00
Harshal Sheth
43d5fac494
Black
2021-02-15 18:29:27 -08:00
Shirshanka Das
7ba81d7087
remove old shell scripts
2021-02-15 18:29:27 -08:00
Harshal Sheth
a87161cad7
Run black formatting on tests
2021-02-15 18:29:27 -08:00
Harshal Sheth
b759205b52
Start checking files for mysql test
2021-02-15 18:29:27 -08:00
Harshal Sheth
36874970ce
Finish file sink
2021-02-15 18:29:27 -08:00
Harshal Sheth
23d675663b
Minor test runid renames
2021-02-15 18:29:27 -08:00
Harshal Sheth
9ea03820c4
Enable bare pytest
2021-02-15 18:29:27 -08:00
Shirshanka Das
4e17a24dd4
Adding mysql integration test + source
2021-02-15 18:29:27 -08:00
Harshal Sheth
4b83fc6591
adding allow deny patterns to sql config
2021-02-15 18:29:27 -08:00
Shirshanka Das
6b5bbbdc5f
workaround for docker exec, waiting for 5 more seconds
2021-02-15 18:29:27 -08:00
Shirshanka Das
9e61220132
checking in testing fixtures. docker still not working
2021-02-15 18:29:27 -08:00
