yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-10-10 15:40:08 +00:00
Code Issues Packages Projects Releases Wiki Activity

fixes

Browse Source
This commit is contained in:
rasbt 2024-05-26 14:28:30 -05:00
parent bd23d7e515
commit fa757e5e48
No known key found for this signature in database
GPG Key ID: 3C6E5C7C075611DB
2 changed files with 22 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
ch07/02_dataset-utilities/README.md
View File

@ -12,7 +12,7 @@ pip install -r requirements-extra.txt
### Finding near duplicates
### Finding Near Duplicates
The `find-near-duplicates.py` function can be used to identify duplicates and near-duplicates in an instruction dataset. For example,
@ -23,6 +23,9 @@ python find-near-duplicates.py --json_file instruction-examples.json
```
```
scikit-learn version: 1.3.1
==================================================
Searching 'instruction' for duplicates ...
==================================================
@ -56,6 +59,22 @@ Duplicate pair found with similarity 1.00:
```
&nbsp
 
You can use the `--threshold` setting with a value between 0 and 1 to decrease or increase the sensitivity.
The default threshold is 0.9.
 
## Creating Passive Voice Entries
- The [create-passive-voice-entries.ipynb](create-passive-voice-entries.ipynb) notebook uses OpenAI's GPT-4 to create "passive voice" entries for an instruction dataset, as shown in the example below
```python
{
'instruction': 'Identify the verb in the following sentence',
'input': 'The cat sleeps on the couch.',
'output': 'The verb in the sentence is "sleeps."',
'output_2': 'The sentence is "sleeps."' # <---- Newly created entry
}
```

2
ch07/02_dataset-utilities/find-near-duplicates.py
View File

@ -61,7 +61,7 @@ def find_near_duplicates(json_data, threshold=0.75, key="instruction"):
for i in range(len(cos_sim_matrix)):
for j in range(i+1, len(cos_sim_matrix)):
if cos_sim_matrix[i, j] > threshold:
if len(json_data[i][key]) <= 1 or len(json_data[j][key]) <=1:
if len(json_data[i][key]) <= 1 or len(json_data[j][key]) <= 1:
continue
near_duplicates.append((json_data[i], json_data[j], cos_sim_matrix[i, j]))
if key in ("input", "output"): # Don't remove duplicates based on the instruction