mirror of
https://github.com/Unstructured-IO/unstructured.git
synced 2025-07-14 12:36:01 +00:00
135aa65906
### Description * If the contents of a doc were updated by the process of reading/downloading it, this was not being persisted. To fix this, the data being passed around was updated to use a multiprocessing safe dict rather than the json string. Now that dict is updated after the `get_file` method is called. * Wikipedia connector was updated to use a static filename rather than one requiring a call to fetch data. * The read config param `re_download` was not being leveraged by the source node, this was fixed. * Added fix: chunking and embedding order reversed so chunking runs before embeddings --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
24 lines
629 B
JSON
24 lines
629 B
JSON
[
|
|
{
|
|
"type": "Title",
|
|
"element_id": "ab0e2c4143f4f6815306af9c90bcc21a",
|
|
"metadata": {
|
|
"data_source": {
|
|
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/2589704",
|
|
"version": "3",
|
|
"record_locator": {
|
|
"url": "https://unstructured-ingest-test.atlassian.net",
|
|
"page_id": "2589704"
|
|
},
|
|
"date_created": "2023-07-13T14:28:06.310000",
|
|
"date_modified": "2023-07-14T22:16:58.907000"
|
|
},
|
|
"filetype": "text/html",
|
|
"languages": [
|
|
"eng"
|
|
],
|
|
"page_number": 1
|
|
},
|
|
"text": "Test text"
|
|
}
|
|
] |