yujunjun/autogen
1
0
Fork 0
mirror of https://github.com/microsoft/autogen.git synced 2026-01-04 11:11:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update flaml/nlp/README.md (#404)

Browse Source
This commit is contained in:
Xueqing Liu 2022-01-14 20:55:38 -05:00 committed by GitHub
parent dda4ac90a1
commit 31645187f3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 8 additions and 56 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

64
flaml/nlp/README.md
View File

@ -1,65 +1,17 @@
# Hyperparameter Optimization for Huggingface Transformers
# AutoML for NLP
Fine-tuning pre-trained language models based on the transformers library.
This directory contains utility functions used by AutoNLP. Currently we support four NLP tasks: sequence classification, sequence regression, multiple choice and summarization.
An example:
Please refer to this [link](https://microsoft.github.io/FLAML/docs/Examples/AutoML-NLP) for examples.
```python
from flaml import AutoML
import pandas as pd
train_dataset = pd.read_csv("data/input/train.tsv", delimiter="\t", quoting=3)
dev_dataset = pd.read_csv("data/input/dev.tsv", delimiter="\t", quoting=3)
test_dataset = pd.read_csv("data/input/test.tsv", delimiter="\t", quoting=3)
# Troubleshooting fine-tuning HPO for pre-trained language models
custom_sent_keys = ["#1 String", "#2 String"]
label_key = "Quality"
The frequent updates of transformers may lead to fluctuations in the results of tuning. To help users quickly troubleshoot the result of AutoNLP when a tuning failure occurs (e.g., failing to reproduce previous results), we have provided the following jupyter notebook:
X_train = train_dataset[custom_sent_keys]
y_train = train_dataset[label_key]
* [Troubleshooting HPO for fine-tuning pre-trained language models](https://github.com/microsoft/FLAML/blob/main/notebook/research/acl2021.ipynb)
X_val = dev_dataset[custom_sent_keys]
y_val = dev_dataset[label_key]
X_test = test_dataset[custom_sent_keys]
automl = AutoML()
automl_settings = {
"gpu_per_trial": 0, # use a value larger than 0 for GPU training
"max_iter": 10,
"time_budget": 300,
"task": "seq-classification",
"metric": "accuracy",
}
automl_settings["custom_hpo_args"] = {
"model_path": "google/electra-small-discriminator",
"output_dir": "data/output/",
"ckpt_per_epoch": 1,
}
automl.fit(
X_train=X_train, y_train=y_train, X_val=X_val, y_val=y_val, **automl_settings
)
automl.predict(X_test)
```
The current use cases that are supported:
1. A simplified version of fine-tuning the GLUE dataset using HuggingFace;
2. For selecting better search space for fine-tuning the GLUE dataset;
3. Use the search algorithms in flaml for more efficient fine-tuning of HuggingFace.
The use cases that can be supported in future:
1. HPO fine-tuning for text generation;
2. HPO fine-tuning for question answering.
## Troubleshooting fine-tuning HPO for pre-trained language models
To reproduce the results for our ACL2021 paper:
Our findings on troubleshooting fine-tuning the Electra and RoBERTa model for the GLUE dataset can be seen in the following paper published in ACL 2021:
* [An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models](https://arxiv.org/abs/2106.09204). Xueqing Liu, Chi Wang. ACL-IJCNLP 2021.
@ -72,4 +24,4 @@ To reproduce the results for our ACL2021 paper:
}
```
Please refer to the following jupyter notebook: [Troubleshooting HPO for fine-tuning pre-trained language models](https://github.com/microsoft/FLAML/blob/main/notebook/research/acl2021.ipynb)