Vladimir Blagojevic 8cafff0645
refactor: Extract HF stop words handling in hf_utils.py (#6745)
* Move StopWordsCriteria to hf_utils.py

* Raise ValueError for invalid StopWordsCriteria tokenizer

* StopWordsCriteria, make sure padding token exists

* Use proper torch types

* Update unit tests
2024-01-15 17:42:29 +01:00
..
2023-11-29 19:24:25 +01:00
2023-11-29 19:24:25 +01:00