yujunjun/haystack
1
0
Fork 0
mirror of https://github.com/deepset-ai/haystack.git synced 2025-09-25 16:15:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: Tiktoken does not support Azure gpt-35-turbo (#4739)

Browse Source 
* force support for gpt-35-turbo

Cos Tiktoken doesn't support it yet - see https://github.com/openai/tiktoken/pull/72

* Update openai_utils.py

* Appeasing the linting gods

Why hast thou forsaken me ?

* Remove trailing whitespace

* chg: remove redundant elif block
This commit is contained in:
recrudesce 2023-04-25 15:43:24 +01:00 committed by GitHub
parent 2be1a68fce
commit 38768bffdf
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
haystack/utils/openai_utils.py
View File

@ -99,6 +99,10 @@ def _openai_text_completion_tokenization_details(model_name: str):
max_tokens_limit = 2049 # Based on this ref: https://platform.openai.com/docs/models/gpt-3
model_tokenizer = MODEL_TO_ENCODING.get(model_name) if USE_TIKTOKEN else None
# covering the lack of support in Tiktoken. https://github.com/openai/tiktoken/pull/72
if model_name == "gpt-35-turbo" and USE_TIKTOKEN:
model_tokenizer = "cl100k_base"
if model_tokenizer:
# Based on OpenAI models page, 'davinci' considers have 2049 tokens,
## therefore, it is better to add `text-davinci` instead to the condition.