3 Commits

Author	SHA1	Message	Date
Wael Karkoub	372ac1e794	Text Compression Transform (#2225) ... * adds implementation * handles optional import * cleanup * updates github workflows * skip test if dependencies not installed * skip test if dependencies not installed * use cpu * skip openai * unskip openai * adds protocol * better docstr * minor fixes * updates optional dependencies docs * wip * update docstrings * wip * adds back llmlingua requirement * finalized protocol * improve docstr * guide complete * improve docstr * fix FAQ * added cache support * improve cache key * cache key fix + faq fix * improve docs * improve guide * args -> params * spelling	2024-05-06 14:16:49 +00:00
giorgossideris	11a43421e3	Min tokens in token limiter (#2400) ... * Add minimum token threshold in MessageHistoryLimiter * Update transforms tests for the threshold * Move min_threshold_tokens from Message to Token Limiter * Optimize _check_tokens_threshold Co-authored-by: Wael Karkoub <wael.karkoub96@gmail.com> * Apply requested changes (renaming, phrasing, validations) * Fix format * Fix _check_tokens_threshold logic * Update docs and notebook * Improve phrasing * Add min_tokens example in notebook * Add min_tokens example in website docs * Add min_tokens example in notebook * Update website docs to be in sync with get_logs change --------- Co-authored-by: Wael Karkoub <wael.karkoub96@gmail.com> Co-authored-by: Chi Wang <wang.chi@microsoft.com>	2024-04-29 02:11:16 +00:00
giorgossideris	90883904c5	Standardize printing of MessageTransforms (#2308) ... * Standardize printing of MessageTransforms * Fix pre-commit black failure * Add test for transform_messages printing * Return str instead of printing * Rename to_print_stats to verbose * Cleanup * t i# This is a combination of 3 commits. Update requirements * Remove lazy-fixture * Avoid calling apply_transform in two code paths * Format * Replace stats with logs * Handle no content messages in TokenLimiter get_logs() * Move tests from test_transform_messages to test_transforms --------- Co-authored-by: Wael Karkoub <wael.karkoub96@gmail.com>	2024-04-14 00:48:24 +00:00