Ben Heckmann
a492771b4d
feat: PreProcessor split by token (tiktoken & Hugging Face) ( #5276 ) ...
* #4983 implemented split by token for tiktoken tokenizer
* #4983 added unit test for tiktoken splitting
* #4983 implemented and added a test for splitting documents with HuggingFace tokenizer
* #4983 added support for passing HF model names (instead of objects) and added an example to the HF token splitting test
* mocked HTTP model loading in unit tests, fixed pylint error
* fix lossy tokenizers splitting, use LazyImport, ignore UnicodeEncodeError for tiktoken
* reno
* rename reno file
---------
Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-11-23 12:26:37 +01:00
2023-11-07 18:00:43 +01:00
2023-09-05 14:34:22 +02:00
2023-10-30 11:26:46 +01:00
2023-09-26 15:57:55 +02:00
2023-10-04 17:23:12 +02:00
2023-10-23 16:08:05 +02:00
2023-11-09 17:35:33 +01:00
2023-10-03 10:02:33 +02:00
2023-10-24 16:50:45 +02:00
2023-11-23 11:41:57 +01:00
2023-08-18 09:59:01 +02:00
2023-09-28 17:22:28 +02:00
2023-11-06 09:48:45 +01:00
2023-11-02 19:35:16 +01:00
2023-10-10 17:48:31 +02:00
2023-08-02 17:05:13 +02:00
2023-10-31 18:28:13 +01:00
2023-09-20 14:49:17 +02:00
2023-11-09 12:20:41 +01:00
2023-11-21 21:44:02 +01:00
2023-11-09 11:25:22 +01:00
2023-08-01 12:48:34 +02:00
2023-11-02 10:56:52 +01:00
2023-10-09 18:03:47 +02:00
2023-10-03 10:42:21 +02:00
2023-09-29 08:40:01 +02:00
2023-09-21 11:52:26 +02:00
2023-11-23 10:28:08 +01:00
2023-10-30 12:50:00 +01:00
2023-08-28 16:23:41 +02:00
2023-08-28 16:23:26 +02:00
2023-09-06 17:31:42 +02:00
2023-10-06 16:01:34 +02:00
2023-08-02 16:34:33 +02:00
2023-10-26 17:42:52 +02:00
2023-09-25 11:47:21 +02:00
2023-11-19 16:32:09 +01:00
2023-10-09 13:44:01 +02:00
2023-08-18 09:59:01 +02:00
2023-09-21 12:16:51 +02:00
2023-10-23 19:02:59 +02:00
2023-08-16 17:41:11 +02:00
2023-10-17 16:32:13 +02:00
2023-09-26 15:37:27 +02:00
2023-11-15 13:26:29 +01:00
2023-11-17 17:49:00 +01:00
2023-08-18 12:28:55 +02:00
2023-08-29 13:46:00 +02:00
2023-08-25 16:38:34 +02:00
2023-10-24 19:13:12 +02:00
2023-10-23 16:13:40 +02:00
2023-08-18 09:59:01 +02:00
2023-11-17 14:46:23 +01:00
2023-08-18 12:53:25 +02:00
2023-09-22 11:05:06 +02:00
2023-10-19 12:10:19 +02:00
2023-09-13 16:14:45 +02:00
2023-08-08 15:48:15 +02:00
2023-09-12 14:34:41 +02:00
2023-08-29 18:15:07 +02:00
2023-10-03 12:48:06 +02:00
2023-08-02 17:04:04 +02:00
2023-10-23 10:35:24 +02:00
2023-10-30 17:03:06 +01:00
2023-11-08 10:35:17 +01:00
2023-10-23 12:26:05 +02:00
2023-11-16 17:17:43 +01:00
2023-11-20 10:56:56 +01:00
2023-10-31 15:35:05 +01:00
2023-10-31 12:44:04 +01:00
2023-10-31 12:44:04 +01:00
2023-08-18 09:59:01 +02:00
2023-11-21 12:56:40 +01:00
2023-08-30 10:25:18 +02:00
2023-08-02 14:13:30 +02:00
2023-11-20 11:54:02 +01:00
2023-08-08 15:51:07 +02:00
2023-11-20 13:00:05 +01:00
2023-10-05 18:43:26 +02:00
2023-09-15 17:03:03 +02:00
2023-11-16 17:10:53 +01:00
2023-11-13 11:42:42 +01:00
2023-11-16 11:22:37 +01:00
2023-09-14 10:49:54 +02:00
2023-08-09 09:54:12 +02:00
2023-11-09 14:28:56 +01:00
2023-10-16 09:31:52 +02:00
2023-09-01 11:48:41 +02:00
2023-11-15 10:20:21 +01:00
2023-10-17 11:03:48 +02:00
2023-08-08 08:35:10 +02:00
2023-08-04 17:40:19 +02:00
2023-10-17 11:25:28 +02:00
2023-09-15 18:12:14 +02:00
2023-08-31 17:33:12 +02:00
2023-09-13 10:07:09 +02:00
2023-10-23 12:53:52 +02:00
2023-11-02 15:29:38 +01:00
2023-10-13 11:23:56 +02:00
2023-10-10 16:38:52 +02:00
2023-08-03 17:41:32 +02:00
2023-08-09 18:14:04 +02:00
2023-11-20 10:03:22 +01:00
2023-10-09 20:19:03 +02:00
2023-08-18 09:59:01 +02:00
2023-08-29 10:57:47 +02:00
2023-08-02 11:42:23 +02:00
2023-09-07 15:44:07 +02:00
2023-09-08 15:52:48 +02:00
2023-11-16 10:44:51 +01:00
2023-11-09 10:45:41 +01:00
2023-10-26 16:25:23 +02:00
2023-11-06 12:00:01 +01:00
2023-09-11 17:40:00 +02:00
2023-09-28 15:42:51 +02:00
2023-09-22 21:54:11 +02:00
2023-10-16 19:26:24 +02:00
2023-09-22 11:09:59 +02:00
2023-09-27 12:26:20 +02:00
2023-09-26 12:48:15 +02:00
2023-09-05 12:22:21 +02:00
2023-11-22 10:40:48 +01:00
2023-10-22 11:15:51 +02:00
2023-09-20 11:03:52 +02:00
2023-10-23 14:02:35 +02:00
2023-08-02 12:45:03 +02:00
2023-11-22 09:57:45 +01:00
2023-10-23 12:26:23 +02:00
2023-08-23 17:03:37 +02:00
2023-10-19 11:17:02 +02:00
2023-10-23 13:01:15 +02:00
2023-08-18 17:59:07 +02:00
2023-10-23 10:35:24 +02:00
2023-08-18 17:48:13 +02:00
2023-11-14 12:33:38 +01:00
2023-09-13 12:55:33 +02:00
2023-08-29 14:25:52 +02:00
2023-10-24 19:45:16 +02:00
2023-10-10 09:14:04 +02:00
2023-10-17 16:15:16 +02:00
2023-10-24 21:44:15 +02:00
2023-11-03 11:33:20 +01:00
2023-10-12 21:50:53 +02:00
2023-08-18 17:03:17 +02:00
2023-08-18 16:33:35 +02:00
2023-11-13 11:59:18 +01:00
2023-09-13 12:55:06 +02:00
2023-08-28 12:32:37 +02:00
2023-11-15 13:10:07 +01:00
2023-11-17 15:39:39 +01:00
2023-11-16 15:04:50 +01:00
2023-11-23 12:26:37 +01:00
2023-10-17 10:36:08 +02:00
2023-08-18 09:59:01 +02:00
2023-08-18 09:59:01 +02:00
2023-09-20 11:03:52 +02:00
2023-09-20 12:05:56 +02:00
2023-09-19 16:06:43 +02:00
2023-11-08 12:04:08 +01:00
2023-10-13 09:31:51 +02:00
2023-10-13 12:39:55 +02:00
2023-10-13 10:30:49 +02:00
2023-08-18 09:59:01 +02:00
2023-11-17 10:12:34 +01:00
2023-08-08 15:13:22 +02:00
2023-10-05 08:56:28 +02:00
2023-10-13 12:08:26 +02:00
2023-10-20 13:00:24 +02:00
2023-09-14 09:57:09 +02:00
2023-09-20 14:45:50 +02:00
2023-08-28 20:02:02 +02:00
2023-08-29 15:46:48 +02:00
a492771b4d