Roman Isecke 135aa65906

update ingest pipeline to share ingest docs via multiprocessing.manager.dict (#1814) ...

### Description
* If the contents of a doc were updated by the process of
reading/downloading it, this was not being persisted. To fix this, the
data being passed around was updated to use a multiprocessing safe dict
rather than the json string. Now that dict is updated after the
`get_file` method is called.
* Wikipedia connector was updated to use a static filename rather than
one requiring a call to fetch data.
* The read config param `re_download` was not being leveraged by the
source node, this was fixed.
* Added fix: chunking and embedding order reversed so chunking runs
before embeddings

---------

Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com>
Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>

2023-10-25 22:04:27 +00:00

..

test_cli.py

Add check for duplicate click options (#1775)

2023-10-20 14:00:19 +00:00

test_error.py

Roman/downstream connector cli subcommand (#1302)

2023-09-11 11:40:56 -04:00

test_interfaces.py

refactor: unstructured ingest as a pipeline (#1551)

2023-10-06 18:49:29 +00:00

test_paths.py

refactor: unstructured ingest as a pipeline (#1551)

2023-10-06 18:49:29 +00:00

test_serialization.py

update ingest pipeline to share ingest docs via multiprocessing.manager.dict (#1814)

2023-10-25 22:04:27 +00:00