Pedro Silva
aa593c32d8
Flexible search on soft delete ( #4405 )
...
* Adds filter logic to correct DB
* Fix build
* Adds documentation & fixes flag typo
* apply review comments
* Adds test for filtered search
* Adds warning log for redundant parameter combo
2022-03-16 16:35:04 -07:00
Pedro Silva
e8f6c4cabd
feat(cli) Changes rollback behaviour to apply soft deletes by default ( #4358 )
...
* Changes rollback behaviour to apply soft deletes by default
Summary:
Addresses feature request: Flag in delete command to only delete aspects touched by an ingestion run; add flag to nuke everything by modifying the default behaviour of a rollback operation which will not by default delete an entity if a keyAspect is being rolled-back.
Instead the key aspect is kept and a StatusAspect is upserted with removed=true, effectively making a soft delete.
Another PR will follow to perform garbage collection on these soft deleted entities.
To keep old behaviour, a new parameter to the cli ingest rollback endpoint: --hard-delete was added.
* Adds restli specs
* Fixes deleteAspect endpoint & adds support for nested transactions
* Enable regression test & fix docker-compose for local development
* Add generated quickstart
* Fix quickstart generation script
* Adds missing var env to docker-compose-without-neo4j
* Sets status removed=true when ingesting resources
* Adds soft deletes for ElasticSearch + soft delete flags across ingestion sub-commands
* Makes elastic search consistent
* Update tests with new behaviour
* apply review comments
* apply review comment
* Forces Elastic search to add documents with status removed false when ingesting
* Reset gradle properties to default
* Fix tests
2022-03-15 12:05:52 -07:00
Dexter Lee
ab0b5162af
fix(lineage): Move lineage registry to entity-registry module ( #4412 )
...
* Move lineage registry
* Fix test
2022-03-15 11:59:59 -07:00
Dexter Lee
a12c9d2a30
Add flag to enable cache ( #4335 )
2022-03-14 16:40:10 -07:00
Dexter Lee
424d357cfd
Fix lineage ( #4326 )
2022-03-07 09:36:49 -08:00
Dexter Lee
18dd5b6c13
feat(Impact Analysis): Support impact analysis to check all downstreams of given entity ( #4322 )
2022-03-04 16:10:25 -08:00
Dexter Lee
daab31d6ef
refactor(extractor): Move extractors to entity-registry ( #4307 )
...
* Move extractors to entity-registry
* Fix test
* Fix checkstyle
2022-03-03 16:57:32 -08:00
Dexter Lee
5d26c86a30
fix(search): Update urn tokenizer to tokenize on periods and slashes ( #4085 )
...
* Add period
* Update urn tokenizer
* Add tests
2022-03-03 10:37:10 -08:00
Ravindra Lanka
1ac20ee764
feat(platform): add timeline api for dataset entity ( #4252 )
2022-02-24 19:40:55 -08:00
HuYueeer
3a0fe44344
feat(search): supporting chinese glossaryterm full text retrieval( #3914 ) ( #3956 )
...
* feat(search): supporting chinese glossaryterm full text retrieval(#3914 )
* refactor(search): modify mainTokenizer to appropriate position(#3914 )
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-24 17:11:43 -08:00
stephenp-gr
fb80e6b7bc
Common ElasticsearchContainer creation ( #4152 )
2022-02-16 16:10:10 -08:00
ksrinath
4c24f386a6
feat(model): data quality model ( #3787 )
...
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
2022-02-02 22:56:08 -08:00
senni0418
80c46f24ac
refactor(search): refactor NUM_RETRIES in esindexbuilder to be configurable ( #3870 )
2022-01-17 15:35:59 -08:00
Dexter Lee
b58609269d
fix(search): Fix issue with filters and autocomplete ( #3868 )
2022-01-12 13:01:36 -08:00
Aseem Bansal
a56f0661d5
perf(ingest): changes to improve ingest performance a bit ( #3837 )
2022-01-06 08:07:16 -08:00
RyanHolstien
9a237e9c92
fix(test): fix bug on verification for EbeanEntityService ( #3829 )
...
Co-authored-by: Ryan Holstien <ryan@acryl.io>
2022-01-05 14:05:54 -08:00
Dexter Lee
9d04aa1255
feat(mae/mcl): Make ingestAspect produce both MCLs and MAEs ( #3737 )
2022-01-04 18:24:30 -08:00
Dexter Lee
76f9423581
feat(nocode): Add get entities v2 endpoint that can get without snapshot ( #3738 )
2022-01-03 07:19:46 -08:00
Ravindra Lanka
4b81ad6262
feat(Stateful Ingestion): 1/3 Stateful ingestion server changes ( #3749 )
2021-12-15 10:32:58 -08:00
Gabe Lyons
3fd3313544
Revert "feat(graph): Make Dgraph a proper Neo4j alternative ( #3578 )" ( #3740 )
2021-12-14 10:49:03 -08:00
Dexter Lee
8e4769f40c
feat(retention): Add retention to Local DB ( #3715 )
2021-12-13 18:18:02 -08:00
Enrico Minack
a6deaabfcf
feat(graph): Make Dgraph a proper Neo4j alternative ( #3578 )
2021-12-13 12:37:59 -08:00
Dexter Lee
1f1467aac2
fix(home): Change docs count to not count removed datasets ( #3711 )
2021-12-09 22:51:06 -08:00
Dexter Lee
999e9e5b49
fix(search): Improve search and recs performance ( #3660 )
2021-12-03 18:13:11 -08:00
Pedro Silva
d5eddf52ab
fix(test): Adds docker engine configuration checks before running docker-based tests for metadata-io ( #3654 )
2021-12-02 12:42:43 -08:00
Shirshanka Das
842c05980b
feat(metadata-service): support for custom model extensions without forks ( #3630 )
2021-11-28 21:06:27 -08:00
Enrico Minack
031e0b9b91
feat(GraphService): Add Dgraph implementation of GraphService ( #3261 )
2021-11-15 10:33:30 -08:00
John Joyce
2f03ad854b
feat(recommendations): Recommendations infra P1 ( #3455 )
2021-10-26 21:23:08 -07:00
Dexter Lee
9b54242a6e
fix(datahub-upgrade): Update datahub upgrade to use MCL instead of MAE ( #3411 )
2021-10-18 22:52:48 -07:00
John Joyce
3d0c69b7d1
fix(metadata-service): add Rest.li Validation to ingestProposal ( #3409 )
2021-10-17 08:49:18 -07:00
John Joyce
61bd7fd080
feat(search): support Boolean OR Filters in Rest.li APIs ( #3344 )
2021-10-14 12:34:29 -07:00
Gabe Lyons
aeba0e9afe
Revert "revert "test(ElasticSearch): Retry ES requests" ( #3385 )" ( #3392 )
...
This reverts commit b16b090f2bf78e0a751490310b95db3dfc142bbf.
2021-10-13 18:53:47 -07:00
Gabe Lyons
b16b090f2b
revert "test(ElasticSearch): Retry ES requests" ( #3385 )
2021-10-13 14:37:51 -07:00
Enrico Minack
3753ff2298
test(elasticsearch): retry elasticsearch requests ( #3377 )
2021-10-12 15:41:34 -07:00
Dexter Lee
d0548408b4
feat(search): redesign search page - left filter pane ( #3337 )
2021-10-07 11:41:29 -07:00
Enrico Minack
c0aa3ecb4b
test(GraphService): Thorough graph service tests ( #3011 )
2021-09-17 13:54:24 -07:00
rslanka
c418bc845c
feat(Analytics): Support for Timeseries Aggregated Statistics ( #3207 )
...
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
Co-authored-by: Dexter Lee <dexter@acryl.io>
2021-09-14 18:35:10 -07:00
Enrico Minack
66c862fea9
test(metadata-io): Improve speed of ElasticSearch tests ( #3160 )
2021-09-13 14:37:29 -07:00
Dexter Lee
e30d7238c0
feat(search): Add searchable annotation to maps ( #3136 )
2021-09-07 22:58:44 -07:00
Dexter Lee
cbc449eba1
fix(search): Move filters to query instead of post query( #3208 )
2021-09-07 17:43:12 -07:00
John Joyce
ccb09a6966
feat(access control): Fine-Grained Access Control M1 ( #3182 )
2021-09-02 19:05:13 -07:00
John Joyce
f40bf1ce19
feat(group ui): Basic group search membership in UI ( #3094 )
2021-08-16 20:47:18 -07:00
Gabe Lyons
2fc71da58d
fix(restore-indices): add system metadata restoration to restore-indices ( #3029 )
2021-08-04 11:44:22 -07:00
Enrico Minack
8e9b5f4b7a
refactor(test): Refactor GraphService tests ( #2944 )
2021-08-02 11:34:45 -07:00
John Joyce
352a0abf8d
Introducing TimeSeries Aspects + Dataset Profile (Stats) Aspect ( #2983 )
...
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-07-30 17:41:03 -07:00
Dexter Lee
15c0c4dfb3
fix(browse): Fix browse pagination and multi-browse path issue ( #2984 )
2021-07-30 10:48:32 -07:00
Gabe Lyons
aa253f5b3b
feat(deletes): add run commands (list, show, rollback) to datahub ingest ( #2960 )
2021-07-29 20:04:40 -07:00
Dexter Lee
6fee59ebac
fix(search): Filter out "removed" entities from autocomplete and analytics ( #2781 )
2021-06-29 17:56:09 -07:00
Gabe Lyons
3f5e371851
feat(graph): support using elasticsearch as graph backend. ( #2726 )
2021-06-22 16:08:54 -07:00
John Joyce
550a9deab0
fix(noCode): Improving efficiency of EntityService "listLatestAspects" API ( #2711 )
2021-06-17 19:52:50 -07:00
