dkirsche a34e4cc515
Refactor transform_messages (#1631)
* refactored code to simplify

* optimize function. Instead of iterating over each character, guess at size and then iterate by token.

* adding tests

* Add missing tests

* minor test fix

* simplified token truncation by using tiktoken to encode and decode

* updated truncated notification message

* Fix llm_config spec to use os.environ

* Add test case and fix bug in loop

---------

Co-authored-by: gagb <gagb@users.noreply.github.com>
2024-02-20 22:53:05 +00:00
..
2024-02-15 00:54:17 +00:00
2023-10-16 17:20:53 +00:00