Kevin Hu
4da76726d3
feat(ingest): SageMaker feature store ingestion ( #2758 )
2021-06-29 19:43:31 -07:00
Remi
2aa95ec750
feat(ingest): Improve lookml sql derived tables detection, add cascading derived tables to lineage ( #2770 )
2021-06-29 19:41:34 -07:00
Harshal Sheth
c8fe8d4026
fix(ingest): quote table names in hive ( #2801 )
2021-06-29 17:51:01 -07:00
Harshal Sheth
57ddb41086
fix(ingest): handle case when view definition handler is not implemented ( #2796 )
2021-06-29 17:49:57 -07:00
Harshal Sheth
79f60d8b8a
refactor(ingest): remove deprecated methods and warn on deprecated import ( #2797 )
2021-06-29 11:43:43 -07:00
Kevin Hu
7083de9b4f
fix(ingest): use correct platform for MongoDB ingestion ( #2783 )
2021-06-29 10:16:58 -07:00
Harshal Sheth
5e69a4355e
refactor(ingest): use common get_sys_time method ( #2782 )
2021-06-28 20:40:10 -07:00
Kevin Hu
09bbcea0a8
feat(ingest): add non-random sampling for mongo ( #2778 )
2021-06-27 23:40:17 -07:00
Harshal Sheth
19b2a42a00
feat: usage stats (part 2) ( #2762 )
...
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-06-24 19:44:59 -07:00
Harshal Sheth
937f02c6bc
feat: usage stats (part 1) ( #2750 )
...
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-06-24 17:11:00 -07:00
Gabe Lyons
62ba937bb7
feat(elastic-as-graph): defaulting to elastic in quickstart ( #2753 )
2021-06-24 15:44:03 -07:00
Remi
91f5d4f59a
feat(ingest): add option to specify source platform database in lookml ingestion ( #2749 )
2021-06-23 16:16:20 -07:00
Kevin Hu
22a2ed81e4
feat(ingest): ingest last-modified from dbt sources.json ( #2729 )
2021-06-23 13:56:20 -07:00
John Joyce
8fc194717c
feat(quickstart): remove orphaned docker containers on quickstart through cli ( #2748 )
2021-06-23 13:46:19 -07:00
Kevin Hu
4ddc4c28be
fix(ingest): fix lookml platform URN ( #2742 )
2021-06-22 13:55:29 -07:00
Kevin Hu
a89094da5b
feat(ingest): add support for Glue ETL jobs ( #2687 )
2021-06-22 11:33:22 -07:00
Kevin Hu
554e1637c5
fix(ingest): types for dbt ( #2716 )
2021-06-22 10:37:08 -07:00
Remi
4944143042
fix(looker): fix invalid URN syntax error ( #2737 )
2021-06-22 10:05:17 -07:00
Gabe Lyons
0750332714
fix(editable descriptions): adding indexing for editable descriptions ( #2710 )
2021-06-17 10:55:26 -07:00
vijayan-nallasami-curve
943c54a835
Removed key not in catalog filter from dbt and added node filter using AllowDenyPattern ( #2702 )
2021-06-17 10:24:12 -07:00
Harshal Sheth
5d93f249b4
feat(ingest): expose additional types to Python via codegen ( #2712 )
2021-06-17 10:04:28 -07:00
Harshal Sheth
26dcece8ec
fix(ingest): use looker data platform ( #2708 )
2021-06-16 16:58:13 -07:00
Kevin Hu
63fe82995b
feat(ingest): Add test case and docs for SQL view ingestion ( #2709 )
2021-06-16 16:51:57 -07:00
Brian
a5f9b8dfe9
feat(entities): add markdown description update/viewer feature in dataset, datajob, dataflow, chart and dashboard, update ui/ux ( #2707 )
2021-06-16 15:48:27 -07:00
John Joyce
cc95916201
feat(gms): Merge MAE, MCE consumers into GMS ( #2690 )
2021-06-15 08:44:15 -07:00
John Joyce
7591c8994a
feat(datahub cli): DataHub CLI Quickstart ( #2689 )
2021-06-14 17:15:24 -07:00
Remi
9fb2e6aaa8
Only work with dbt catalog data if load_catalog is False ( #2686 )
2021-06-14 16:58:19 -07:00
Vincenzo Lavorini
a7fc76f590
feat(sql_views): added views as datasets for SQLAlchemy DBs ( #2663 )
2021-06-11 17:30:33 -07:00
Harshal Sheth
1b539220d5
feat(ingest): support Oracle service names ( #2676 )
2021-06-11 17:27:34 -07:00
Kevin Hu
24268c2021
feat(ingest): headers for codegen Python scripts ( #2637 )
2021-06-11 09:44:18 -07:00
zack3241
91eb3cc57e
Add get_identifier to hive source in metadata ingestion ( #2667 )
2021-06-09 15:12:17 -07:00
Gabe Lyons
28b5856c36
fix(no-code): Adding Chart input relationship annotations ( #2669 )
...
* chart input relationship pdl fix
* commiting schema.avsc changes
2021-06-09 15:09:50 -07:00
Kevin Hu
ebdaa0e359
feat(ingest): Feast ingestion integration ( #2605 )
...
* Add feast testing setup
* Init Feast test script
* Add feast to dependencies
* Update feast descriptors
* Sort integrations
* Working feast pytest
* Clean up feast docker-compose file
* Expand Feast tests
* Setup feast classes
* Add continuous and bytes data to feature types
* Update field type mapping
* Add PDLs
* Add MLFeatureSetUrn.java
* Comment out feast setup
* Add snapshot file and update inits
* Init Feast golden files generation
* Clean up Feast ingest
* Feast testing comments
* Yield Feature snapshots
* Fix Feature URN naming
* Update feast MCE
* Update Feature URN prefix
* Add MLEntity
* Update golden files with entities
* Specify feast sources
* Add feast source configs
* Working feast docker ingestion
* List entities and features before adding tables
* Add featureset names
* Remove unused
* Rename feast image
* Update README
* Add env to feast URNs
* Fix URN naming
* Remove redundant URN names
* Fix enum backcompatibility
* Move feast testing to docker
* Move URN generators to mce_builder
* Add source for features
* Switch TypeClass -> enum_type
* Rename source -> sourceDataset
* Add local Feast ingest image builds
* Rename Entity -> MLPrimaryKey
* Restore features and keys for each featureset
* Do not json encode source configs
* Remove old source properties from feature sets
* Regenerate golden file
* Fix race condition with Feast tests
* Exclude unknown source
* Update feature datatype enum
* Update README and fix typos
* Fix Entity typo
* Fix path to local docker image
* Specify feast config and version
* Fix feast env variables
* PR fixes
* Refactor feast ingest constants
* Make feature sources optional for back-compatibility
* Remove unused GCP files
* adding docker publish workflow
* Simplify name+namespace in PrimaryKeys
* adding docker publish workflow
* debug
* final attempt
* final final attempt
* final final final commit
* Switch to published ingestion image
* Update name and namespace in java files
* Rename FeatureSet -> FeatureTable
* Regenerate codegen
* Fix initial generation errors
* Update snapshot jsons
* Regenerated schemas
* Fix URN formats
* Revise builds
* Clean up feast URN builders
* Fix naming typos
* Fix Feature Set -> Feature Table
* Fix comments
* PR fixes
* All you need is Urn
* Regenerate snapshots and update validation
* Add UNKNOWN data type
* URNs for source types
* Add note on docker requirement
* Fix typo
* Reorder aspect unions
* Refactor feast ingest functions
* Update snapshot jsons
* Rebuild
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-06-09 15:07:04 -07:00
Harshal Sheth
31eae24300
fix(ingest): support mssql encryption via ODBC ( #2657 )
2021-06-04 18:19:11 -07:00
John Joyce
97e9660037
feat: No Code Metadata Modeling ( #2629 )
...
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-06-03 13:24:33 -07:00
Harshal Sheth
a0ad590b3f
fix(ingest): improve redshift ingestion performance ( #2635 )
2021-06-03 11:14:34 -07:00
Harshal Sheth
6b9d0d0129
fix(ingest): include urn as key for kafka emitter ( #2634 )
2021-06-03 11:04:40 -07:00
Thomas Larsson
b512920022
fix(ingestion): improve robustness of glue ingestion source ( #2626 )
...
fixes : #2625
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-06-01 11:02:52 -07:00
Harshal Sheth
acef397ece
fix(ingest): fail gracefully when lookml used on old python versions ( #2614 )
2021-05-26 17:16:17 -07:00
Harshal Sheth
958fe8ea83
feat(ingest): populate inputDatajobs field in airflow integration ( #2606 )
2021-05-25 22:47:00 -07:00
Remi
6aa133f99c
fix(ingest): fix lineage after dbt metadata ingestion when tables name and identifier differ ( #2596 )
2021-05-25 18:59:35 -07:00
Albert Franzi
4ad6d30509
fix(ingest): incorrect implementation of the allow pattern in looker dashboards ( #2585 )
2021-05-24 22:10:13 -07:00
Kevin Hu
48d2b94203
fix(ingest): default values for env ( #2598 )
2021-05-24 14:09:55 -07:00
Brian
aa8ba1b8e9
feat(dataflow): update dataflow to have datajobs in new tab ( #2579 )
2021-05-18 21:25:42 -07:00
taufiqibrahim
db78373427
feat(ingest): kafka connect metadata ingestion ( #2516 )
2021-05-18 14:45:38 -07:00
Harshal Sheth
1d4bcbe4fb
feat(ingest): add dataset tag transformer ( #2580 )
2021-05-18 14:43:43 -07:00
Harshal Sheth
7af1a13138
fix(ingest): better active directory LDAP support ( #2571 )
2021-05-17 14:42:54 -07:00
Gary Lucas
af4f3b9683
fix(dbt): set target platform and load schema ( #2483 )
2021-05-17 12:22:52 -07:00
Albert Franzi
38e3f6d4d0
feat(ingest): add AWS IAM Roles Support to the Glue Source ( #2563 )
2021-05-17 12:19:34 -07:00
David Koenitzer
4958febed5
docs: Add Sphinx Docstrings to Airflow Modules ( #2562 )
2021-05-17 12:10:35 -07:00
