Lal Rishav
99790d9052
fix(react): enable relation between glossary term and datasets searchable ( #2806 )
...
* 2805 | glossary term linked to dataset is not displayed
2021-07-22 07:25:29 -07:00
Kevin Hu
736249f0c7
feat(ingest): extract SageMaker metrics, hyperparameters, and external URLs ( #2910 )
2021-07-21 21:30:07 -07:00
Kevin Hu
6abd5e191a
feat(ingest): lineage for SageMaker model endpoints and groups ( #2894 )
2021-07-19 11:30:43 -07:00
Lal Rishav
ae014bf2e1
fix(glossary):default browse path for glossary term ( #2788 )
2021-07-16 09:15:24 -07:00
Kevin Hu
44ed2f3684
feat(ingest): extract lineage between SageMaker jobs and models ( #2868 )
2021-07-15 18:56:13 -07:00
Kevin Hu
a2106ca9e8
feat(ingest): SageMaker jobs and models ( #2830 )
2021-07-08 16:16:16 -07:00
Harshal Sheth
2f921d15e8
fix(ingest): avoid setting timestamps unless source system provides it ( #2843 )
2021-07-08 12:11:06 -07:00
Kevin Hu
4da76726d3
feat(ingest): SageMaker feature store ingestion ( #2758 )
2021-06-29 19:43:31 -07:00
Dexter Lee
ae8a8893b0
fix(nocode): removing service PDL ( #2772 )
2021-06-25 10:58:10 -07:00
Harshal Sheth
19b2a42a00
feat: usage stats (part 2) ( #2762 )
...
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-06-24 19:44:59 -07:00
Harshal Sheth
937f02c6bc
feat: usage stats (part 1) ( #2750 )
...
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2021-06-24 17:11:00 -07:00
Kevin Hu
a89094da5b
feat(ingest): add support for Glue ETL jobs ( #2687 )
2021-06-22 11:33:22 -07:00
Gabe Lyons
0750332714
fix(editable descriptions): adding indexing for editable descriptions ( #2710 )
2021-06-17 10:55:26 -07:00
Brian
a5f9b8dfe9
feat(entities): add markdown description update/viewer feature in dataset, datajob, dataflow, chart and dashboard, update ui/ux ( #2707 )
2021-06-16 15:48:27 -07:00
Gabe Lyons
523c3bf1d4
feat(aspects): support fetching of versioned aspects ( #2677 )
2021-06-16 10:03:21 -07:00
Gabe Lyons
0b75b4a96a
removing whitespace from service aspect ( #2688 )
2021-06-12 17:48:36 -07:00
Gabe Lyons
28b5856c36
fix(no-code): Adding Chart input relationship annotations ( #2669 )
...
* chart input relationship pdl fix
* commiting schema.avsc changes
2021-06-09 15:09:50 -07:00
Kevin Hu
ebdaa0e359
feat(ingest): Feast ingestion integration ( #2605 )
...
* Add feast testing setup
* Init Feast test script
* Add feast to dependencies
* Update feast descriptors
* Sort integrations
* Working feast pytest
* Clean up feast docker-compose file
* Expand Feast tests
* Setup feast classes
* Add continuous and bytes data to feature types
* Update field type mapping
* Add PDLs
* Add MLFeatureSetUrn.java
* Comment out feast setup
* Add snapshot file and update inits
* Init Feast golden files generation
* Clean up Feast ingest
* Feast testing comments
* Yield Feature snapshots
* Fix Feature URN naming
* Update feast MCE
* Update Feature URN prefix
* Add MLEntity
* Update golden files with entities
* Specify feast sources
* Add feast source configs
* Working feast docker ingestion
* List entities and features before adding tables
* Add featureset names
* Remove unused
* Rename feast image
* Update README
* Add env to feast URNs
* Fix URN naming
* Remove redundant URN names
* Fix enum backcompatibility
* Move feast testing to docker
* Move URN generators to mce_builder
* Add source for features
* Switch TypeClass -> enum_type
* Rename source -> sourceDataset
* Add local Feast ingest image builds
* Rename Entity -> MLPrimaryKey
* Restore features and keys for each featureset
* Do not json encode source configs
* Remove old source properties from feature sets
* Regenerate golden file
* Fix race condition with Feast tests
* Exclude unknown source
* Update feature datatype enum
* Update README and fix typos
* Fix Entity typo
* Fix path to local docker image
* Specify feast config and version
* Fix feast env variables
* PR fixes
* Refactor feast ingest constants
* Make feature sources optional for back-compatibility
* Remove unused GCP files
* adding docker publish workflow
* Simplify name+namespace in PrimaryKeys
* adding docker publish workflow
* debug
* final attempt
* final final attempt
* final final final commit
* Switch to published ingestion image
* Update name and namespace in java files
* Rename FeatureSet -> FeatureTable
* Regenerate codegen
* Fix initial generation errors
* Update snapshot jsons
* Regenerated schemas
* Fix URN formats
* Revise builds
* Clean up feast URN builders
* Fix naming typos
* Fix Feature Set -> Feature Table
* Fix comments
* PR fixes
* All you need is Urn
* Regenerate snapshots and update validation
* Add UNKNOWN data type
* URNs for source types
* Add note on docker requirement
* Fix typo
* Reorder aspect unions
* Refactor feast ingest functions
* Update snapshot jsons
* Rebuild
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-06-09 15:07:04 -07:00
John Joyce
97e9660037
feat: No Code Metadata Modeling ( #2629 )
...
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-06-03 13:24:33 -07:00
Brian
aa8ba1b8e9
feat(dataflow): update dataflow to have datajobs in new tab ( #2579 )
2021-05-18 21:25:42 -07:00
John Joyce
262108e3aa
fix(model): removing reference to go link in SchemaFieldPath model ( #2530 )
2021-05-11 15:03:04 -07:00
shakti-garg
8ed14a62e2
feat(business_glossary): add new entity business term and its relationship with dataset and its fields ( #2228 )
...
Co-authored-by: shubham.garg <shubham.garg@thoughtworks.com>
2021-05-10 13:20:23 -07:00
Dexter Lee
b4457afe30
feat(search): Add search for field level description and tags ( #2491 )
2021-05-05 14:04:02 -07:00
Gabe Lyons
851e00ba9f
feat(lineage): implement support for datasets, charts and dashboards downstream lineage fetching in a generic way ( #2397 )
...
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Brian <brianwebtek@gmail.com>
Co-authored-by: John Joyce <john@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2021-04-23 00:18:39 -07:00
Gabe Lyons
c7b49de67b
feat(ingest): adding superset ingestion source ( #2425 )
2021-04-22 00:11:54 -07:00
Fredrik Sannholm
e88a671959
Fix(search): fix datajob and dataflow search mappings ( #2418 )
2021-04-21 12:04:20 -07:00
Gabe Lyons
039fe597f7
feat(tags): editing tags from react client on datasets, schemas, charts & dashboards ( #2248 )
2021-03-18 11:52:14 -07:00
Harshal Sheth
b8462028c3
feat(ingest): various minor fixes ( #2246 )
2021-03-17 23:05:05 -07:00
Dexter Lee
c362cc4388
feat(tags): Enable search for datasets by tags ( #2240 )
2021-03-15 22:37:06 -07:00
Fredrik Sannholm
da6b3d111d
feat(datajob): Backend implementation ( #2197 )
2021-03-13 08:00:44 -08:00
Gabe Lyons
11e0cd66d4
feat(tag): adding search for tags in gms layer ( #2203 )
2021-03-10 00:02:58 -08:00
Gabe Lyons
92387019ff
feat(react): adding ability to support theming of datahub, with two themes included ( #2193 )
2021-03-09 23:14:52 -08:00
Harshal Sheth
d220647094
feat: add date and time types to SQL model ( #2201 )
2021-03-09 23:07:20 -08:00
Gabe Lyons
adfe60e97a
feat(tags): adding support for read/write of tags in gms & read-only in react datahub-frontend. ( #2164 )
2021-03-07 11:26:47 -08:00
John Joyce
4dcea8c1d3
feat(gms): Add optional data platform display name ( #2148 )
...
Co-authored-by: John Joyce <john@acryl.io>
2021-02-26 21:22:18 -08:00
Gabe Lyons
e6f7f39d15
docs(Ownership): making lack of support for ownergroups in frontend explicit in pdl ( #2139 )
2021-02-24 22:32:55 -08:00
John Joyce
4f8d8b31ac
feat: Introducing optional DataPlatform logo url ( #2127 )
...
Co-authored-by: John Joyce <john@acryl.io>
2021-02-19 19:51:49 -08:00
RyanHolstien
ea86ade29b
feat: ML Model Backend Implementation ( #1896 )
...
Co-authored-by: RyanHolstien <rholstien@expediagroup.com>
2021-02-17 13:28:13 -08:00
Nagarjuna Kanamarlapudi
f9d33f5519
(refactor): Convert dataPlatforms to GMA aspect models and associated resource to GMA resource. ( #2057 )
...
* (refactor): Convert dataPlatforms to GMA aspect and associated resource to GMA resource.
BREAKING CHANGE: /datasets/dataPlatforms API is now changed to become GMA resource.
* Change documentation style
2021-01-20 15:50:48 -08:00
Harsh Shah
06cca77a6b
feat(models): Add DataFlow and DataJob models ( #2008 )
...
Add DataFlow and DataJob urn and aspect models.
The aspects include metadata associated with Data processing flow and job entities,
associated inputs and outputs of a job.
2020-11-30 17:48:29 -08:00
Kerem Sahin
4d8320e4a0
feat(dashboard): Dashboards backend implementation ( #1884 )
2020-11-23 09:25:58 -08:00
Nagarjuna Kanamarlapudi
5d083143db
feat(dataset): Enable search of datasets by field names ( #2001 )
...
* feat(dataset): Enable search of datasets by field names
2020-11-20 12:01:07 -08:00
John Plaisted
6f59a91865
Bump GMA to latest ( #2007 )
...
Involves fixing some types. Bright side is that ModelValidation actually checks aspects now (it did not before because aspects are not records but unions!).
2020-11-18 14:07:48 -08:00
Kerem Sahin
733893f5f9
feat(dashboard): Dashboard models update ( #1932 )
...
* feat(dashboard): Dashboard models update
* Keep chartId/dashboardId fields in the URN definitions and add fields for chartURL/dashboardURL into info aspects
* Rebase and address some comments
2020-11-12 11:17:22 -08:00
Nagarjuna Kanamarlapudi
7d574d1094
feat(field-level-lineage): Add models for field level lineage ( #1936 )
...
* feat(field-level-lineage): adding models for field level lineage
adding models for field level lineage. Introduce DatasetFieldUrn as a unique identifier for dataset field
2020-11-09 14:08:48 -08:00
Mars Lan
609739179b
refactor: drop unused models to prevent drifts ( #1964 )
2020-10-27 10:02:21 -07:00
John Plaisted
25b663cc18
refactor: move code to linkedin/datahub-gma. ( #1955 )
...
Move code to linkedin/datahub-gma.
"GMA" (Generalized Metadata Architecture) is the backend of DataHub, and has been moved to its own repository.
This deletes the code that was moved and uses jars that GMA publishes to bintray to load it.
Note that not all of GMA was moved, but most of it. We may still move more things to the other repository in the future.
2020-10-23 15:14:57 -07:00
John Plaisted
5e70f3648c
Fix build after sync.
...
- Add build files for new module.
- Correctly edit TestUtils since it isn't synced.
- Reference new test utils.
- Delete duplicate pdl files.
SYNC=metadata-models_100.0.1
2020-09-24 16:02:12 -07:00
Kerem Sahin
7c7de50be8
feat(dashboard): Dashboard metadata models ( #1775 )
...
* Dashboard metadata models
* Update pairings in Contains and DownstreamOf relationships
* Adding ChartDataSourceType
* Extend chart types
* Make chart query an aspect & add chartIds to dashboard search document
* Add urn validation for chart & dashboard urns
2020-09-14 12:01:54 -07:00
John Plaisted
8223cdcbdb
Fix build after merge:
...
- Add commonsLang to build file.
- Add emails field to CorpUserInfoDocument (either this should be synced or the index builder not synced in the future).
- Fix EbeanLocalDAOTest which used internal Urn API.
- Fix BaseSearchableEntityResource "backfill" override return types (and regenerate snapshots).
- EbeanlocalDAO's constructor changed; now requires URN class.
- Add restli resource module as dependency of :gms:api as it now contains a needed PDL model.
2020-09-11 09:15:56 -07:00
