yujunjun/autogen
1
0
Fork 0
mirror of https://github.com/microsoft/autogen.git synced 2025-07-18 06:22:53 +00:00
Code Issues Packages Projects Releases Wiki Activity
autogen/notebook/flaml_automl.ipynb

1287 lines
98 KiB
Plaintext
Raw Normal View History

v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
{
"cells": [
{
"cell_type": "markdown",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"source": [
warning -> info for low cost partial config (#231) * warning -> info for low cost partial config #195, #110 * when n_estimators < 0, use trained_estimator's * log debug info * test random seed * remove "objective"; avoid ZeroDivisionError * hp config to estimator params * check type of searcher * default n_jobs * try import * Update searchalgo_auto.py * CLASSIFICATION * auto_augment flag * min_sample_size * make catboost optional
2021-10-08 16:09:43 -07:00
"Copyright (c) Microsoft Corporation. All rights reserved. \n",
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"\n",
"Licensed under the MIT License.\n",
"\n",
"# AutoML with FLAML Library\n",
"\n",
"\n",
"## 1. Introduction\n",
"\n",
"FLAML is a Python library (https://github.com/microsoft/FLAML) designed to automatically produce accurate machine learning models \n",
warning -> info for low cost partial config (#231) * warning -> info for low cost partial config #195, #110 * when n_estimators < 0, use trained_estimator's * log debug info * test random seed * remove "objective"; avoid ZeroDivisionError * hp config to estimator params * check type of searcher * default n_jobs * try import * Update searchalgo_auto.py * CLASSIFICATION * auto_augment flag * min_sample_size * make catboost optional
2021-10-08 16:09:43 -07:00
"with low computational cost. It is fast and cheap. The simple and lightweight design makes it easy to use and extend, such as adding new learners. FLAML can \n",
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"- serve as an economical AutoML engine,\n",
"- be used as a fast hyperparameter tuning tool, or \n",
"- be embedded in self-tuning software that requires low latency & resource in repetitive\n",
" tuning tasks.\n",
"\n",
"In this notebook, we use one real data example (binary classification) to showcase how to use FLAML library.\n",
"\n",
"FLAML requires `Python>=3.6`. To run this notebook example, please install flaml with the `notebook` option:\n",
"```bash\n",
"pip install flaml[notebook]\n",
"```"
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "code",
test -> val; docstr (#300) * rename test -> val in custom metric function * add an example in docstr resolve #299
2021-11-22 22:17:29 -08:00
"execution_count": 1,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
test -> val; docstr (#300) * rename test -> val in custom metric function * add an example in docstr resolve #299
2021-11-22 22:17:29 -08:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: flaml[notebook] in /usr/local/lib/python3.9/site-packages (0.7.1)\n",
"Requirement already satisfied: scipy>=1.4.1 in /usr/local/lib/python3.9/site-packages (from flaml[notebook]) (1.7.2)\n",
"Requirement already satisfied: lightgbm>=2.3.1 in /usr/local/lib/python3.9/site-packages (from flaml[notebook]) (3.3.1)\n",
"Requirement already satisfied: pandas>=1.1.4 in /usr/local/lib/python3.9/site-packages (from flaml[notebook]) (1.3.4)\n",
"Requirement already satisfied: NumPy>=1.16.2 in /usr/local/lib/python3.9/site-packages (from flaml[notebook]) (1.21.4)\n",
"Requirement already satisfied: xgboost<=1.3.3,>=0.90 in /usr/local/lib/python3.9/site-packages (from flaml[notebook]) (1.3.3)\n",
"Requirement already satisfied: scikit-learn>=0.24 in /usr/local/lib/python3.9/site-packages (from flaml[notebook]) (1.0.1)\n",
"Requirement already satisfied: catboost>=0.26 in /usr/local/lib/python3.9/site-packages (from flaml[notebook]) (1.0.3)\n",
"Requirement already satisfied: jupyter in /usr/local/lib/python3.9/site-packages (from flaml[notebook]) (1.0.0)\n",
"Requirement already satisfied: rgf-python in /usr/local/lib/python3.9/site-packages (from flaml[notebook]) (3.11.0)\n",
"Requirement already satisfied: matplotlib in /usr/local/lib/python3.9/site-packages (from flaml[notebook]) (3.5.0)\n",
"Requirement already satisfied: openml==0.10.2 in /usr/local/lib/python3.9/site-packages (from flaml[notebook]) (0.10.2)\n",
"Requirement already satisfied: xmltodict in /usr/local/lib/python3.9/site-packages (from openml==0.10.2->flaml[notebook]) (0.12.0)\n",
"Requirement already satisfied: requests in /usr/local/lib/python3.9/site-packages (from openml==0.10.2->flaml[notebook]) (2.26.0)\n",
"Requirement already satisfied: liac-arff>=2.4.0 in /usr/local/lib/python3.9/site-packages (from openml==0.10.2->flaml[notebook]) (2.5.0)\n",
"Requirement already satisfied: python-dateutil in /usr/local/lib/python3.9/site-packages (from openml==0.10.2->flaml[notebook]) (2.8.2)\n",
"Requirement already satisfied: plotly in /usr/local/lib/python3.9/site-packages (from catboost>=0.26->flaml[notebook]) (5.4.0)\n",
"Requirement already satisfied: six in /usr/local/lib/python3.9/site-packages (from catboost>=0.26->flaml[notebook]) (1.16.0)\n",
"Requirement already satisfied: graphviz in /usr/local/lib/python3.9/site-packages (from catboost>=0.26->flaml[notebook]) (0.18.2)\n",
"Requirement already satisfied: wheel in /usr/local/lib/python3.9/site-packages (from lightgbm>=2.3.1->flaml[notebook]) (0.37.0)\n",
"Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.9/site-packages (from pandas>=1.1.4->flaml[notebook]) (2021.3)\n",
"Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.9/site-packages (from scikit-learn>=0.24->flaml[notebook]) (1.1.0)\n",
"Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.9/site-packages (from scikit-learn>=0.24->flaml[notebook]) (3.0.0)\n",
"Requirement already satisfied: ipykernel in /usr/local/lib/python3.9/site-packages (from jupyter->flaml[notebook]) (6.5.1)\n",
"Requirement already satisfied: qtconsole in /usr/local/lib/python3.9/site-packages (from jupyter->flaml[notebook]) (5.2.0)\n",
"Requirement already satisfied: notebook in /usr/local/lib/python3.9/site-packages (from jupyter->flaml[notebook]) (6.4.6)\n",
"Requirement already satisfied: ipywidgets in /usr/local/lib/python3.9/site-packages (from jupyter->flaml[notebook]) (7.6.5)\n",
"Requirement already satisfied: jupyter-console in /usr/local/lib/python3.9/site-packages (from jupyter->flaml[notebook]) (6.4.0)\n",
"Requirement already satisfied: nbconvert in /usr/local/lib/python3.9/site-packages (from jupyter->flaml[notebook]) (6.3.0)\n",
"Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.9/site-packages (from matplotlib->flaml[notebook]) (3.0.6)\n",
"Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.9/site-packages (from matplotlib->flaml[notebook]) (8.4.0)\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.9/site-packages (from matplotlib->flaml[notebook]) (1.3.2)\n",
"Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.9/site-packages (from matplotlib->flaml[notebook]) (4.28.1)\n",
"Requirement already satisfied: setuptools-scm>=4 in /usr/local/lib/python3.9/site-packages (from matplotlib->flaml[notebook]) (6.3.2)\n",
"Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.9/site-packages (from matplotlib->flaml[notebook]) (0.11.0)\n",
"Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.9/site-packages (from matplotlib->flaml[notebook]) (21.3)\n",
"Requirement already satisfied: tomli>=1.0.0 in /usr/local/lib/python3.9/site-packages (from setuptools-scm>=4->matplotlib->flaml[notebook]) (1.2.2)\n",
"Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from setuptools-scm>=4->matplotlib->flaml[notebook]) (57.5.0)\n",
"Requirement already satisfied: traitlets<6.0,>=5.1.0 in /usr/local/lib/python3.9/site-packages (from ipykernel->jupyter->flaml[notebook]) (5.1.1)\n",
"Requirement already satisfied: tornado<7.0,>=4.2 in /usr/local/lib/python3.9/site-packages (from ipykernel->jupyter->flaml[notebook]) (6.1)\n",
"Requirement already satisfied: jupyter-client<8.0 in /usr/local/lib/python3.9/site-packages (from ipykernel->jupyter->flaml[notebook]) (7.0.6)\n",
"Requirement already satisfied: ipython>=7.23.1 in /usr/local/lib/python3.9/site-packages (from ipykernel->jupyter->flaml[notebook]) (7.29.0)\n",
"Requirement already satisfied: debugpy<2.0,>=1.0.0 in /usr/local/lib/python3.9/site-packages (from ipykernel->jupyter->flaml[notebook]) (1.5.1)\n",
"Requirement already satisfied: matplotlib-inline<0.2.0,>=0.1.0 in /usr/local/lib/python3.9/site-packages (from ipykernel->jupyter->flaml[notebook]) (0.1.3)\n",
"Requirement already satisfied: widgetsnbextension~=3.5.0 in /usr/local/lib/python3.9/site-packages (from ipywidgets->jupyter->flaml[notebook]) (3.5.2)\n",
"Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /usr/local/lib/python3.9/site-packages (from ipywidgets->jupyter->flaml[notebook]) (1.0.2)\n",
"Requirement already satisfied: nbformat>=4.2.0 in /usr/local/lib/python3.9/site-packages (from ipywidgets->jupyter->flaml[notebook]) (5.1.3)\n",
"Requirement already satisfied: ipython-genutils~=0.2.0 in /usr/local/lib/python3.9/site-packages (from ipywidgets->jupyter->flaml[notebook]) (0.2.0)\n",
"Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.9/site-packages (from jupyter-console->jupyter->flaml[notebook]) (3.0.22)\n",
"Requirement already satisfied: pygments in /usr/local/lib/python3.9/site-packages (from jupyter-console->jupyter->flaml[notebook]) (2.10.0)\n",
"Requirement already satisfied: jupyter-core in /usr/local/lib/python3.9/site-packages (from nbconvert->jupyter->flaml[notebook]) (4.9.1)\n",
"Requirement already satisfied: bleach in /usr/local/lib/python3.9/site-packages (from nbconvert->jupyter->flaml[notebook]) (4.1.0)\n",
"Requirement already satisfied: jinja2>=2.4 in /usr/local/lib/python3.9/site-packages (from nbconvert->jupyter->flaml[notebook]) (3.0.3)\n",
"Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.9/site-packages (from nbconvert->jupyter->flaml[notebook]) (0.3)\n",
"Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /usr/local/lib/python3.9/site-packages (from nbconvert->jupyter->flaml[notebook]) (0.5.9)\n",
"Requirement already satisfied: jupyterlab-pygments in /usr/local/lib/python3.9/site-packages (from nbconvert->jupyter->flaml[notebook]) (0.1.2)\n",
"Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.9/site-packages (from nbconvert->jupyter->flaml[notebook]) (1.5.0)\n",
"Requirement already satisfied: defusedxml in /usr/local/lib/python3.9/site-packages (from nbconvert->jupyter->flaml[notebook]) (0.7.1)\n",
"Requirement already satisfied: testpath in /usr/local/lib/python3.9/site-packages (from nbconvert->jupyter->flaml[notebook]) (0.5.0)\n",
"Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.9/site-packages (from nbconvert->jupyter->flaml[notebook]) (0.8.4)\n",
"Requirement already satisfied: Send2Trash>=1.8.0 in /usr/local/lib/python3.9/site-packages (from notebook->jupyter->flaml[notebook]) (1.8.0)\n",
"Requirement already satisfied: prometheus-client in /usr/local/lib/python3.9/site-packages (from notebook->jupyter->flaml[notebook]) (0.12.0)\n",
"Requirement already satisfied: pyzmq>=17 in /usr/local/lib/python3.9/site-packages (from notebook->jupyter->flaml[notebook]) (22.3.0)\n",
"Requirement already satisfied: nest-asyncio>=1.5 in /usr/local/lib/python3.9/site-packages (from notebook->jupyter->flaml[notebook]) (1.5.1)\n",
"Requirement already satisfied: argon2-cffi in /usr/local/lib/python3.9/site-packages (from notebook->jupyter->flaml[notebook]) (21.1.0)\n",
"Requirement already satisfied: terminado>=0.8.3 in /usr/local/lib/python3.9/site-packages (from notebook->jupyter->flaml[notebook]) (0.12.1)\n",
"Requirement already satisfied: tenacity>=6.2.0 in /usr/local/lib/python3.9/site-packages (from plotly->catboost>=0.26->flaml[notebook]) (8.0.1)\n",
"Requirement already satisfied: qtpy in /usr/local/lib/python3.9/site-packages (from qtconsole->jupyter->flaml[notebook]) (1.11.2)\n",
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/site-packages (from requests->openml==0.10.2->flaml[notebook]) (1.26.7)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests->openml==0.10.2->flaml[notebook]) (2021.10.8)\n",
"Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests->openml==0.10.2->flaml[notebook]) (3.3)\n",
"Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/site-packages (from requests->openml==0.10.2->flaml[notebook]) (2.0.7)\n",
"Requirement already satisfied: decorator in /usr/local/lib/python3.9/site-packages (from ipython>=7.23.1->ipykernel->jupyter->flaml[notebook]) (5.1.0)\n",
"Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.9/site-packages (from ipython>=7.23.1->ipykernel->jupyter->flaml[notebook]) (4.8.0)\n",
"Requirement already satisfied: jedi>=0.16 in /usr/local/lib/python3.9/site-packages (from ipython>=7.23.1->ipykernel->jupyter->flaml[notebook]) (0.18.1)\n",
"Requirement already satisfied: pickleshare in /usr/local/lib/python3.9/site-packages (from ipython>=7.23.1->ipykernel->jupyter->flaml[notebook]) (0.7.5)\n",
"Requirement already satisfied: backcall in /usr/local/lib/python3.9/site-packages (from ipython>=7.23.1->ipykernel->jupyter->flaml[notebook]) (0.2.0)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.9/site-packages (from jinja2>=2.4->nbconvert->jupyter->flaml[notebook]) (2.0.1)\n",
"Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /usr/local/lib/python3.9/site-packages (from nbformat>=4.2.0->ipywidgets->jupyter->flaml[notebook]) (4.2.1)\n",
"Requirement already satisfied: wcwidth in /usr/local/lib/python3.9/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->jupyter-console->jupyter->flaml[notebook]) (0.2.5)\n",
"Requirement already satisfied: ptyprocess in /usr/local/lib/python3.9/site-packages (from terminado>=0.8.3->notebook->jupyter->flaml[notebook]) (0.7.0)\n",
"Requirement already satisfied: cffi>=1.0.0 in /usr/local/lib/python3.9/site-packages (from argon2-cffi->notebook->jupyter->flaml[notebook]) (1.15.0)\n",
"Requirement already satisfied: webencodings in /usr/local/lib/python3.9/site-packages (from bleach->nbconvert->jupyter->flaml[notebook]) (0.5.1)\n",
"Requirement already satisfied: pycparser in /usr/local/lib/python3.9/site-packages (from cffi>=1.0.0->argon2-cffi->notebook->jupyter->flaml[notebook]) (2.21)\n",
"Requirement already satisfied: parso<0.9.0,>=0.8.0 in /usr/local/lib/python3.9/site-packages (from jedi>=0.16->ipython>=7.23.1->ipykernel->jupyter->flaml[notebook]) (0.8.2)\n",
"Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.9/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets->jupyter->flaml[notebook]) (21.2.0)\n",
"Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.9/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets->jupyter->flaml[notebook]) (0.18.0)\n",
"\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n",
"\u001b[33mWARNING: You are using pip version 21.3; however, version 21.3.1 is available.\n",
"You should consider upgrading via the '/usr/local/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n"
]
}
],
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"source": [
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"!pip install flaml[notebook];\n",
"# from v0.6.6, catboost is made an optional dependency to build conda package.\n",
"# to install catboost without installing the notebook option, you can run:\n",
warning -> info for low cost partial config (#231) * warning -> info for low cost partial config #195, #110 * when n_estimators < 0, use trained_estimator's * log debug info * test random seed * remove "objective"; avoid ZeroDivisionError * hp config to estimator params * check type of searcher * default n_jobs * try import * Update searchalgo_auto.py * CLASSIFICATION * auto_augment flag * min_sample_size * make catboost optional
2021-10-08 16:09:43 -07:00
"# !pip install flaml[catboost]"
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "markdown",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"source": [
"## 2. Classification Example\n",
"### Load data and preprocess\n",
"\n",
"Download [Airlines dataset](https://www.openml.org/d/1169) from OpenML. The task is to predict whether a given flight will be delayed, given the information of the scheduled departure."
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "code",
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"execution_count": 1,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"outputs": [
{
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"name": "stdout",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"output_type": "stream",
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"text": [
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"download dataset from openml\n",
coverage (#135) * coverage * readme * timeout
2021-07-20 17:00:44 -07:00
"Dataset name: airlines\n",
"X_train.shape: (404537, 7), y_train.shape: (404537,);\n",
"X_test.shape: (134846, 7), y_test.shape: (134846,)\n"
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
]
}
],
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"source": [
"from flaml.data import load_openml_dataset\n",
"X_train, X_test, y_train, y_test = load_openml_dataset(dataset_id=1169, data_dir='./')"
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
},
"source": [
"### Run FLAML\n",
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"In the FLAML automl run configuration, users can specify the task type, time budget, error metric, learner list, whether to subsample, resampling strategy type, and so on. All these arguments have default values which will be used if users do not provide them. For example, the default classifiers are `['lgbm', 'xgboost', 'xgb_limitdepth', 'catboost', 'rf', 'extra_tree', 'lrl1']`. "
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
},
"outputs": [],
"source": [
"''' import AutoML class from flaml package '''\n",
"from flaml import AutoML\n",
"automl = AutoML()"
]
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
},
"outputs": [],
"source": [
"settings = {\n",
" \"time_budget\": 240, # total running time in seconds\n",
" \"metric\": 'accuracy', # can be: 'r2', 'rmse', 'mae', 'mse', 'accuracy', 'roc_auc', 'roc_auc_ovr',\n",
" # 'roc_auc_ovo', 'log_loss', 'mape', 'f1', 'ap', 'ndcg', 'micro_f1', 'macro_f1'\n",
" \"task\": 'classification', # task type\n",
" \"log_file_name\": 'airlines_experiment.log', # flaml log file\n",
" \"seed\": 7654321, # random seed\n",
"}"
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "code",
"execution_count": 4,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {
"slideshow": {
"slide_type": "slide"
},
"tags": [
"outputPrepend"
]
},
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"outputs": [
{
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"name": "stderr",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"output_type": "stream",
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"text": [
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"[flaml.automl: 11-23 01:33:02] {1861} INFO - task = classification\n",
"[flaml.automl: 11-23 01:33:02] {1863} INFO - Data split method: stratified\n",
"[flaml.automl: 11-23 01:33:02] {1867} INFO - Evaluation method: holdout\n",
"[flaml.automl: 11-23 01:33:02] {1933} INFO - Minimizing error metric: 1-accuracy\n",
"[flaml.automl: 11-23 01:33:03] {1985} INFO - List of ML learners in AutoML Run: ['lgbm', 'rf', 'catboost', 'xgboost', 'extra_tree', 'xgb_limitdepth', 'lrl1']\n",
"[flaml.automl: 11-23 01:33:03] {2223} INFO - iteration 0, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:03] {2337} INFO - Estimated sufficient time budget=53519s. Estimated necessary time budget=1314s.\n",
"[flaml.automl: 11-23 01:33:03] {2417} INFO - at 1.3s,\testimator lgbm's best error=0.3777,\tbest estimator lgbm's best error=0.3777\n",
"[flaml.automl: 11-23 01:33:03] {2223} INFO - iteration 1, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:03] {2417} INFO - at 1.4s,\testimator lgbm's best error=0.3759,\tbest estimator lgbm's best error=0.3759\n",
"[flaml.automl: 11-23 01:33:03] {2223} INFO - iteration 2, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:03] {2417} INFO - at 1.4s,\testimator lgbm's best error=0.3759,\tbest estimator lgbm's best error=0.3759\n",
"[flaml.automl: 11-23 01:33:03] {2223} INFO - iteration 3, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:03] {2417} INFO - at 1.5s,\testimator lgbm's best error=0.3644,\tbest estimator lgbm's best error=0.3644\n",
"[flaml.automl: 11-23 01:33:03] {2223} INFO - iteration 4, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:03] {2417} INFO - at 1.6s,\testimator lgbm's best error=0.3588,\tbest estimator lgbm's best error=0.3588\n",
"[flaml.automl: 11-23 01:33:03] {2223} INFO - iteration 5, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:03] {2417} INFO - at 1.6s,\testimator lgbm's best error=0.3588,\tbest estimator lgbm's best error=0.3588\n",
"[flaml.automl: 11-23 01:33:03] {2223} INFO - iteration 6, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:03] {2417} INFO - at 1.8s,\testimator lgbm's best error=0.3555,\tbest estimator lgbm's best error=0.3555\n",
"[flaml.automl: 11-23 01:33:03] {2223} INFO - iteration 7, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:04] {2417} INFO - at 1.9s,\testimator lgbm's best error=0.3555,\tbest estimator lgbm's best error=0.3555\n",
"[flaml.automl: 11-23 01:33:04] {2223} INFO - iteration 8, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:04] {2417} INFO - at 2.2s,\testimator lgbm's best error=0.3555,\tbest estimator lgbm's best error=0.3555\n",
"[flaml.automl: 11-23 01:33:04] {2223} INFO - iteration 9, current learner xgboost\n",
"[flaml.automl: 11-23 01:33:04] {2417} INFO - at 2.3s,\testimator xgboost's best error=0.3787,\tbest estimator lgbm's best error=0.3555\n",
"[flaml.automl: 11-23 01:33:04] {2223} INFO - iteration 10, current learner xgboost\n",
"[flaml.automl: 11-23 01:33:04] {2417} INFO - at 2.3s,\testimator xgboost's best error=0.3649,\tbest estimator lgbm's best error=0.3555\n",
"[flaml.automl: 11-23 01:33:04] {2223} INFO - iteration 11, current learner xgboost\n",
"[flaml.automl: 11-23 01:33:04] {2417} INFO - at 2.3s,\testimator xgboost's best error=0.3649,\tbest estimator lgbm's best error=0.3555\n",
"[flaml.automl: 11-23 01:33:04] {2223} INFO - iteration 12, current learner xgboost\n",
"[flaml.automl: 11-23 01:33:04] {2417} INFO - at 2.4s,\testimator xgboost's best error=0.3649,\tbest estimator lgbm's best error=0.3555\n",
"[flaml.automl: 11-23 01:33:04] {2223} INFO - iteration 13, current learner xgboost\n",
"[flaml.automl: 11-23 01:33:04] {2417} INFO - at 2.4s,\testimator xgboost's best error=0.3629,\tbest estimator lgbm's best error=0.3555\n",
"[flaml.automl: 11-23 01:33:04] {2223} INFO - iteration 14, current learner extra_tree\n",
"[flaml.automl: 11-23 01:33:04] {2417} INFO - at 2.8s,\testimator extra_tree's best error=0.3773,\tbest estimator lgbm's best error=0.3555\n",
"[flaml.automl: 11-23 01:33:04] {2223} INFO - iteration 15, current learner rf\n",
"[flaml.automl: 11-23 01:33:05] {2417} INFO - at 3.1s,\testimator rf's best error=0.3787,\tbest estimator lgbm's best error=0.3555\n",
"[flaml.automl: 11-23 01:33:05] {2223} INFO - iteration 16, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:05] {2417} INFO - at 3.7s,\testimator lgbm's best error=0.3555,\tbest estimator lgbm's best error=0.3555\n",
"[flaml.automl: 11-23 01:33:05] {2223} INFO - iteration 17, current learner extra_tree\n",
"[flaml.automl: 11-23 01:33:06] {2417} INFO - at 4.2s,\testimator extra_tree's best error=0.3762,\tbest estimator lgbm's best error=0.3555\n",
"[flaml.automl: 11-23 01:33:06] {2223} INFO - iteration 18, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:06] {2417} INFO - at 4.4s,\testimator lgbm's best error=0.3542,\tbest estimator lgbm's best error=0.3542\n",
"[flaml.automl: 11-23 01:33:06] {2223} INFO - iteration 19, current learner rf\n",
"[flaml.automl: 11-23 01:33:07] {2417} INFO - at 4.9s,\testimator rf's best error=0.3736,\tbest estimator lgbm's best error=0.3542\n",
"[flaml.automl: 11-23 01:33:07] {2223} INFO - iteration 20, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:07] {2417} INFO - at 5.5s,\testimator lgbm's best error=0.3507,\tbest estimator lgbm's best error=0.3507\n",
"[flaml.automl: 11-23 01:33:07] {2223} INFO - iteration 21, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:07] {2417} INFO - at 5.7s,\testimator lgbm's best error=0.3507,\tbest estimator lgbm's best error=0.3507\n",
"[flaml.automl: 11-23 01:33:07] {2223} INFO - iteration 22, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:10] {2417} INFO - at 7.8s,\testimator lgbm's best error=0.3507,\tbest estimator lgbm's best error=0.3507\n",
"[flaml.automl: 11-23 01:33:10] {2223} INFO - iteration 23, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:11] {2417} INFO - at 9.3s,\testimator lgbm's best error=0.3404,\tbest estimator lgbm's best error=0.3404\n",
"[flaml.automl: 11-23 01:33:11] {2223} INFO - iteration 24, current learner catboost\n",
"[flaml.automl: 11-23 01:33:12] {2417} INFO - at 10.2s,\testimator catboost's best error=0.3587,\tbest estimator lgbm's best error=0.3404\n",
"[flaml.automl: 11-23 01:33:12] {2223} INFO - iteration 25, current learner catboost\n",
"[flaml.automl: 11-23 01:33:12] {2417} INFO - at 10.5s,\testimator catboost's best error=0.3587,\tbest estimator lgbm's best error=0.3404\n",
"[flaml.automl: 11-23 01:33:12] {2223} INFO - iteration 26, current learner xgboost\n",
"[flaml.automl: 11-23 01:33:12] {2417} INFO - at 10.5s,\testimator xgboost's best error=0.3629,\tbest estimator lgbm's best error=0.3404\n",
"[flaml.automl: 11-23 01:33:12] {2223} INFO - iteration 27, current learner catboost\n",
"[flaml.automl: 11-23 01:33:15] {2417} INFO - at 13.8s,\testimator catboost's best error=0.3587,\tbest estimator lgbm's best error=0.3404\n",
"[flaml.automl: 11-23 01:33:15] {2223} INFO - iteration 28, current learner catboost\n",
"[flaml.automl: 11-23 01:33:17] {2417} INFO - at 15.2s,\testimator catboost's best error=0.3496,\tbest estimator lgbm's best error=0.3404\n",
"[flaml.automl: 11-23 01:33:17] {2223} INFO - iteration 29, current learner xgboost\n",
"[flaml.automl: 11-23 01:33:19] {2417} INFO - at 17.5s,\testimator xgboost's best error=0.3625,\tbest estimator lgbm's best error=0.3404\n",
"[flaml.automl: 11-23 01:33:19] {2223} INFO - iteration 30, current learner catboost\n",
"[flaml.automl: 11-23 01:33:20] {2417} INFO - at 18.6s,\testimator catboost's best error=0.3471,\tbest estimator lgbm's best error=0.3404\n",
"[flaml.automl: 11-23 01:33:20] {2223} INFO - iteration 31, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:25] {2417} INFO - at 23.0s,\testimator lgbm's best error=0.3404,\tbest estimator lgbm's best error=0.3404\n",
"[flaml.automl: 11-23 01:33:25] {2223} INFO - iteration 32, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:28] {2417} INFO - at 26.4s,\testimator lgbm's best error=0.3312,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:33:28] {2223} INFO - iteration 33, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:40] {2417} INFO - at 37.8s,\testimator lgbm's best error=0.3312,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:33:40] {2223} INFO - iteration 34, current learner catboost\n",
"[flaml.automl: 11-23 01:33:51] {2417} INFO - at 49.0s,\testimator catboost's best error=0.3471,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:33:51] {2223} INFO - iteration 35, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:55] {2417} INFO - at 53.3s,\testimator lgbm's best error=0.3312,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:33:55] {2223} INFO - iteration 36, current learner rf\n",
"[flaml.automl: 11-23 01:33:55] {2417} INFO - at 53.8s,\testimator rf's best error=0.3706,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:33:55] {2223} INFO - iteration 37, current learner xgb_limitdepth\n",
"[flaml.automl: 11-23 01:33:56] {2417} INFO - at 53.9s,\testimator xgb_limitdepth's best error=0.3630,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:33:56] {2223} INFO - iteration 38, current learner xgb_limitdepth\n",
"[flaml.automl: 11-23 01:33:56] {2417} INFO - at 53.9s,\testimator xgb_limitdepth's best error=0.3630,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:33:56] {2223} INFO - iteration 39, current learner xgb_limitdepth\n",
"[flaml.automl: 11-23 01:33:56] {2417} INFO - at 54.0s,\testimator xgb_limitdepth's best error=0.3630,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:33:56] {2223} INFO - iteration 40, current learner xgb_limitdepth\n",
"[flaml.automl: 11-23 01:33:56] {2417} INFO - at 54.1s,\testimator xgb_limitdepth's best error=0.3611,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:33:56] {2223} INFO - iteration 41, current learner rf\n",
"[flaml.automl: 11-23 01:33:56] {2417} INFO - at 54.6s,\testimator rf's best error=0.3706,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:33:56] {2223} INFO - iteration 42, current learner xgb_limitdepth\n",
"[flaml.automl: 11-23 01:33:56] {2417} INFO - at 54.7s,\testimator xgb_limitdepth's best error=0.3611,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:33:56] {2223} INFO - iteration 43, current learner xgb_limitdepth\n",
"[flaml.automl: 11-23 01:33:56] {2417} INFO - at 54.7s,\testimator xgb_limitdepth's best error=0.3611,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:33:56] {2223} INFO - iteration 44, current learner lgbm\n",
"[flaml.automl: 11-23 01:33:59] {2417} INFO - at 57.8s,\testimator lgbm's best error=0.3312,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:33:59] {2223} INFO - iteration 45, current learner xgb_limitdepth\n",
"[flaml.automl: 11-23 01:34:00] {2417} INFO - at 58.0s,\testimator xgb_limitdepth's best error=0.3609,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:34:00] {2223} INFO - iteration 46, current learner xgb_limitdepth\n",
"[flaml.automl: 11-23 01:34:00] {2417} INFO - at 58.1s,\testimator xgb_limitdepth's best error=0.3603,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:34:00] {2223} INFO - iteration 47, current learner xgb_limitdepth\n",
"[flaml.automl: 11-23 01:34:00] {2417} INFO - at 58.3s,\testimator xgb_limitdepth's best error=0.3603,\tbest estimator lgbm's best error=0.3312\n",
"[flaml.automl: 11-23 01:34:00] {2223} INFO - iteration 48, current learner lgbm\n",
"[flaml.automl: 11-23 01:34:05] {2417} INFO - at 63.3s,\testimator lgbm's best error=0.3304,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:34:05] {2223} INFO - iteration 49, current learner lrl1\n",
"/usr/local/lib/python3.7/site-packages/sklearn/linear_model/_sag.py:354: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
" ConvergenceWarning,\n",
"[flaml.automl: 11-23 01:34:05] {2417} INFO - at 63.5s,\testimator lrl1's best error=0.4338,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:34:05] {2223} INFO - iteration 50, current learner lrl1\n",
"/usr/local/lib/python3.7/site-packages/sklearn/linear_model/_sag.py:354: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
" ConvergenceWarning,\n",
"[flaml.automl: 11-23 01:34:05] {2417} INFO - at 63.7s,\testimator lrl1's best error=0.4338,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:34:05] {2223} INFO - iteration 51, current learner lrl1\n",
"/usr/local/lib/python3.7/site-packages/sklearn/linear_model/_sag.py:354: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
" ConvergenceWarning,\n",
"[flaml.automl: 11-23 01:34:06] {2417} INFO - at 64.1s,\testimator lrl1's best error=0.4338,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:34:06] {2223} INFO - iteration 52, current learner lrl1\n",
"/usr/local/lib/python3.7/site-packages/sklearn/linear_model/_sag.py:354: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
" ConvergenceWarning,\n",
"[flaml.automl: 11-23 01:34:07] {2417} INFO - at 64.9s,\testimator lrl1's best error=0.4333,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:34:07] {2223} INFO - iteration 53, current learner xgb_limitdepth\n",
"[flaml.automl: 11-23 01:34:07] {2417} INFO - at 65.1s,\testimator xgb_limitdepth's best error=0.3603,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:34:07] {2223} INFO - iteration 54, current learner lgbm\n",
"[flaml.automl: 11-23 01:34:09] {2417} INFO - at 67.3s,\testimator lgbm's best error=0.3304,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:34:09] {2223} INFO - iteration 55, current learner lgbm\n",
"[flaml.automl: 11-23 01:34:21] {2417} INFO - at 79.2s,\testimator lgbm's best error=0.3304,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:34:21] {2223} INFO - iteration 56, current learner lgbm\n",
"[flaml.automl: 11-23 01:34:23] {2417} INFO - at 81.2s,\testimator lgbm's best error=0.3304,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:34:23] {2223} INFO - iteration 57, current learner lgbm\n",
"[flaml.automl: 11-23 01:34:42] {2417} INFO - at 100.2s,\testimator lgbm's best error=0.3304,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:34:42] {2223} INFO - iteration 58, current learner extra_tree\n",
"[flaml.automl: 11-23 01:34:42] {2417} INFO - at 100.7s,\testimator extra_tree's best error=0.3762,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:34:42] {2223} INFO - iteration 59, current learner lgbm\n",
"[flaml.automl: 11-23 01:35:24] {2417} INFO - at 142.3s,\testimator lgbm's best error=0.3304,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:24] {2223} INFO - iteration 60, current learner rf\n",
"[flaml.automl: 11-23 01:35:24] {2417} INFO - at 142.6s,\testimator rf's best error=0.3697,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:24] {2223} INFO - iteration 61, current learner xgboost\n",
"[flaml.automl: 11-23 01:35:24] {2417} INFO - at 142.7s,\testimator xgboost's best error=0.3625,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:24] {2223} INFO - iteration 62, current learner lgbm\n",
"[flaml.automl: 11-23 01:35:25] {2417} INFO - at 143.5s,\testimator lgbm's best error=0.3304,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:25] {2223} INFO - iteration 63, current learner extra_tree\n",
"[flaml.automl: 11-23 01:35:26] {2417} INFO - at 143.8s,\testimator extra_tree's best error=0.3762,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:26] {2223} INFO - iteration 64, current learner lgbm\n",
"[flaml.automl: 11-23 01:35:28] {2417} INFO - at 146.3s,\testimator lgbm's best error=0.3304,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:28] {2223} INFO - iteration 65, current learner extra_tree\n",
"[flaml.automl: 11-23 01:35:28] {2417} INFO - at 146.6s,\testimator extra_tree's best error=0.3762,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:28] {2223} INFO - iteration 66, current learner xgb_limitdepth\n",
"[flaml.automl: 11-23 01:35:28] {2417} INFO - at 146.8s,\testimator xgb_limitdepth's best error=0.3603,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:28] {2223} INFO - iteration 67, current learner lgbm\n",
"[flaml.automl: 11-23 01:35:42] {2417} INFO - at 159.9s,\testimator lgbm's best error=0.3304,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:42] {2223} INFO - iteration 68, current learner rf\n",
"[flaml.automl: 11-23 01:35:42] {2417} INFO - at 160.3s,\testimator rf's best error=0.3697,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:42] {2223} INFO - iteration 69, current learner xgb_limitdepth\n",
"[flaml.automl: 11-23 01:35:42] {2417} INFO - at 160.4s,\testimator xgb_limitdepth's best error=0.3602,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:42] {2223} INFO - iteration 70, current learner rf\n",
"[flaml.automl: 11-23 01:35:42] {2417} INFO - at 160.7s,\testimator rf's best error=0.3697,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:42] {2223} INFO - iteration 71, current learner lgbm\n",
"[flaml.automl: 11-23 01:35:45] {2417} INFO - at 162.8s,\testimator lgbm's best error=0.3304,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:45] {2223} INFO - iteration 72, current learner catboost\n",
"[flaml.automl: 11-23 01:35:49] {2417} INFO - at 167.6s,\testimator catboost's best error=0.3369,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:49] {2223} INFO - iteration 73, current learner catboost\n",
"[flaml.automl: 11-23 01:35:54] {2417} INFO - at 172.5s,\testimator catboost's best error=0.3369,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:35:54] {2223} INFO - iteration 74, current learner catboost\n",
"[flaml.automl: 11-23 01:37:02] {2417} INFO - at 240.1s,\testimator catboost's best error=0.3369,\tbest estimator lgbm's best error=0.3304\n",
"[flaml.automl: 11-23 01:37:11] {2629} INFO - retrain lgbm for 9.3s\n",
"[flaml.automl: 11-23 01:37:11] {2634} INFO - retrained model: LGBMClassifier(learning_rate=0.10320258241974468, max_bin=1023,\n",
" min_child_samples=36, n_estimators=1071, num_leaves=25,\n",
" reg_alpha=0.0009765625, reg_lambda=0.08547376339713011,\n",
warning -> info for low cost partial config (#231) * warning -> info for low cost partial config #195, #110 * when n_estimators < 0, use trained_estimator's * log debug info * test random seed * remove "objective"; avoid ZeroDivisionError * hp config to estimator params * check type of searcher * default n_jobs * try import * Update searchalgo_auto.py * CLASSIFICATION * auto_augment flag * min_sample_size * make catboost optional
2021-10-08 16:09:43 -07:00
" verbose=-1)\n",
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"[flaml.automl: 11-23 01:37:11] {2014} INFO - fit succeeded\n",
"[flaml.automl: 11-23 01:37:11] {2016} INFO - Time taken to find the best model: 63.26783061027527\n"
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
]
}
],
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"source": [
"'''The main flaml automl API'''\n",
"automl.fit(X_train=X_train, y_train=y_train, **settings)"
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
},
"source": [
"### Best model and metric"
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "code",
"execution_count": 5,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {
"slideshow": {
"slide_type": "slide"
},
"tags": []
},
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"outputs": [
{
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"name": "stdout",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"output_type": "stream",
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"text": [
coverage (#135) * coverage * readme * timeout
2021-07-20 17:00:44 -07:00
"Best ML leaner: lgbm\n",
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"Best hyperparmeter config: {'n_estimators': 1071, 'num_leaves': 25, 'min_child_samples': 36, 'learning_rate': 0.10320258241974468, 'log_max_bin': 10, 'colsample_bytree': 1.0, 'reg_alpha': 0.0009765625, 'reg_lambda': 0.08547376339713011, 'FLAML_sample_size': 364083}\n",
"Best accuracy on validation data: 0.6696\n",
"Training duration of best run: 9.274 s\n"
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
]
}
],
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"source": [
"'''retrieve best config and best learner'''\n",
"print('Best ML leaner:', automl.best_estimator)\n",
"print('Best hyperparmeter config:', automl.best_config)\n",
"print('Best accuracy on validation data: {0:.4g}'.format(1-automl.best_loss))\n",
"print('Training duration of best run: {0:.4g} s'.format(automl.best_config_train_time))"
]
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
},
{
"cell_type": "code",
"execution_count": 6,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"outputs": [
{
"data": {
"text/plain": [
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"LGBMClassifier(learning_rate=0.10320258241974468, max_bin=1023,\n",
" min_child_samples=36, n_estimators=1071, num_leaves=25,\n",
" reg_alpha=0.0009765625, reg_lambda=0.08547376339713011,\n",
warning -> info for low cost partial config (#231) * warning -> info for low cost partial config #195, #110 * when n_estimators < 0, use trained_estimator's * log debug info * test random seed * remove "objective"; avoid ZeroDivisionError * hp config to estimator params * check type of searcher * default n_jobs * try import * Update searchalgo_auto.py * CLASSIFICATION * auto_augment flag * min_sample_size * make catboost optional
2021-10-08 16:09:43 -07:00
" verbose=-1)"
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
]
},
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"execution_count": 6,
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"metadata": {},
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"output_type": "execute_result"
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
}
],
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"source": [
"automl.model.estimator"
]
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
},
{
"cell_type": "code",
"execution_count": 7,
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
},
"outputs": [],
"source": [
"'''pickle and save the automl object'''\n",
"import pickle\n",
"with open('automl.pkl', 'wb') as f:\n",
" pickle.dump(automl, f, pickle.HIGHEST_PROTOCOL)\n",
"'''load pickled automl object'''\n",
"with open('automl.pkl', 'rb') as f:\n",
" automl = pickle.load(f)"
]
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
},
{
"cell_type": "code",
"execution_count": 8,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {
"slideshow": {
"slide_type": "slide"
},
"tags": []
},
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"outputs": [
{
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"name": "stdout",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"output_type": "stream",
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"text": [
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"Predicted labels ['1' '0' '1' ... '1' '0' '0']\n",
"True labels 118331 0\n",
"328182 0\n",
"335454 0\n",
"520591 1\n",
"344651 0\n",
" ..\n",
"367080 0\n",
"203510 1\n",
"254894 0\n",
"296512 1\n",
"362444 0\n",
"Name: Delay, Length: 134846, dtype: category\n",
"Categories (2, object): ['0' < '1']\n"
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
]
}
],
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"source": [
"'''compute predictions of testing dataset''' \n",
"y_pred = automl.predict(X_test)\n",
"print('Predicted labels', y_pred)\n",
"print('True labels', y_test)\n",
"y_pred_proba = automl.predict_proba(X_test)[:,1]"
]
},
{
"cell_type": "code",
"execution_count": 9,
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"metadata": {
"slideshow": {
"slide_type": "slide"
},
"tags": []
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
},
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"outputs": [
{
"name": "stdout",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"output_type": "stream",
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"text": [
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"accuracy = 0.6720332824110467\n",
"roc_auc = 0.7253276908529442\n",
"log_loss = 0.6034449031876942\n"
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
]
}
],
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"source": [
"''' compute different metric values on testing dataset'''\n",
"from flaml.ml import sklearn_metric_loss_score\n",
"print('accuracy', '=', 1 - sklearn_metric_loss_score('accuracy', y_pred, y_test))\n",
"print('roc_auc', '=', 1 - sklearn_metric_loss_score('roc_auc', y_pred_proba, y_test))\n",
"print('log_loss', '=', sklearn_metric_loss_score('log_loss', y_pred_proba, y_test))"
]
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
},
"source": [
"See Section 4 for an accuracy comparison with default LightGBM and XGBoost.\n",
"\n",
"### Log history"
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "code",
"execution_count": 10,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"outputs": [
{
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"name": "stdout",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"output_type": "stream",
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"text": [
Support parallel and add random search (#167) * non hashable value out of signature * parallel trials * add random in _search_parallel * fix bug in retraining * check memory constraint before training * retrain_full * log custom metric * retraining budget check * sample size check before retrain * remove 'time2eval' from result * report 'total_search_time' in result * rename total_search_time to wall_clock_time * rename train_loss boolean to log_training_metric * set default train_loss to None * exclude oom result * log retrained model * no subsample * doc str * notebook * predicted value is NaN for sarimax * version Co-authored-by: Chi Wang <wang.chi@microsoft.com> Co-authored-by: Qingyun Wu <qxw5138@psu.edu>
2021-08-23 19:36:51 -04:00
"{'Current Learner': 'lgbm', 'Current Sample': 10000, 'Current Hyper-parameters': {'n_estimators': 4, 'num_leaves': 4, 'min_child_samples': 20, 'learning_rate': 0.09999999999999995, 'log_max_bin': 8, 'colsample_bytree': 1.0, 'reg_alpha': 0.0009765625, 'reg_lambda': 1.0, 'FLAML_sample_size': 10000}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 4, 'num_leaves': 4, 'min_child_samples': 20, 'learning_rate': 0.09999999999999995, 'log_max_bin': 8, 'colsample_bytree': 1.0, 'reg_alpha': 0.0009765625, 'reg_lambda': 1.0, 'FLAML_sample_size': 10000}}\n",
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"{'Current Learner': 'lgbm', 'Current Sample': 10000, 'Current Hyper-parameters': {'n_estimators': 4, 'num_leaves': 14, 'min_child_samples': 15, 'learning_rate': 0.22841390623808822, 'log_max_bin': 9, 'colsample_bytree': 1.0, 'reg_alpha': 0.0014700173967242716, 'reg_lambda': 7.624911621832711, 'FLAML_sample_size': 10000}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 4, 'num_leaves': 14, 'min_child_samples': 15, 'learning_rate': 0.22841390623808822, 'log_max_bin': 9, 'colsample_bytree': 1.0, 'reg_alpha': 0.0014700173967242716, 'reg_lambda': 7.624911621832711, 'FLAML_sample_size': 10000}}\n",
"{'Current Learner': 'lgbm', 'Current Sample': 10000, 'Current Hyper-parameters': {'n_estimators': 4, 'num_leaves': 25, 'min_child_samples': 12, 'learning_rate': 0.5082200481556802, 'log_max_bin': 8, 'colsample_bytree': 0.9696263001275751, 'reg_alpha': 0.0028107036379524425, 'reg_lambda': 3.716898117989413, 'FLAML_sample_size': 10000}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 4, 'num_leaves': 25, 'min_child_samples': 12, 'learning_rate': 0.5082200481556802, 'log_max_bin': 8, 'colsample_bytree': 0.9696263001275751, 'reg_alpha': 0.0028107036379524425, 'reg_lambda': 3.716898117989413, 'FLAML_sample_size': 10000}}\n",
"{'Current Learner': 'lgbm', 'Current Sample': 10000, 'Current Hyper-parameters': {'n_estimators': 23, 'num_leaves': 14, 'min_child_samples': 15, 'learning_rate': 0.22841390623808822, 'log_max_bin': 9, 'colsample_bytree': 1.0, 'reg_alpha': 0.0014700173967242718, 'reg_lambda': 7.624911621832699, 'FLAML_sample_size': 10000}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 23, 'num_leaves': 14, 'min_child_samples': 15, 'learning_rate': 0.22841390623808822, 'log_max_bin': 9, 'colsample_bytree': 1.0, 'reg_alpha': 0.0014700173967242718, 'reg_lambda': 7.624911621832699, 'FLAML_sample_size': 10000}}\n",
"{'Current Learner': 'lgbm', 'Current Sample': 10000, 'Current Hyper-parameters': {'n_estimators': 101, 'num_leaves': 12, 'min_child_samples': 24, 'learning_rate': 0.07647794276357095, 'log_max_bin': 10, 'colsample_bytree': 1.0, 'reg_alpha': 0.001749539645587163, 'reg_lambda': 4.373760956394571, 'FLAML_sample_size': 10000}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 101, 'num_leaves': 12, 'min_child_samples': 24, 'learning_rate': 0.07647794276357095, 'log_max_bin': 10, 'colsample_bytree': 1.0, 'reg_alpha': 0.001749539645587163, 'reg_lambda': 4.373760956394571, 'FLAML_sample_size': 10000}}\n",
"{'Current Learner': 'lgbm', 'Current Sample': 40000, 'Current Hyper-parameters': {'n_estimators': 101, 'num_leaves': 12, 'min_child_samples': 24, 'learning_rate': 0.07647794276357095, 'log_max_bin': 10, 'colsample_bytree': 1.0, 'reg_alpha': 0.001749539645587163, 'reg_lambda': 4.373760956394571, 'FLAML_sample_size': 40000}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 101, 'num_leaves': 12, 'min_child_samples': 24, 'learning_rate': 0.07647794276357095, 'log_max_bin': 10, 'colsample_bytree': 1.0, 'reg_alpha': 0.001749539645587163, 'reg_lambda': 4.373760956394571, 'FLAML_sample_size': 40000}}\n",
"{'Current Learner': 'lgbm', 'Current Sample': 40000, 'Current Hyper-parameters': {'n_estimators': 361, 'num_leaves': 11, 'min_child_samples': 32, 'learning_rate': 0.13528717598813866, 'log_max_bin': 9, 'colsample_bytree': 0.9851977789068981, 'reg_alpha': 0.0038372002422749616, 'reg_lambda': 0.25113531892556773, 'FLAML_sample_size': 40000}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 361, 'num_leaves': 11, 'min_child_samples': 32, 'learning_rate': 0.13528717598813866, 'log_max_bin': 9, 'colsample_bytree': 0.9851977789068981, 'reg_alpha': 0.0038372002422749616, 'reg_lambda': 0.25113531892556773, 'FLAML_sample_size': 40000}}\n",
"{'Current Learner': 'lgbm', 'Current Sample': 364083, 'Current Hyper-parameters': {'n_estimators': 361, 'num_leaves': 11, 'min_child_samples': 32, 'learning_rate': 0.13528717598813866, 'log_max_bin': 9, 'colsample_bytree': 0.9851977789068981, 'reg_alpha': 0.0038372002422749616, 'reg_lambda': 0.25113531892556773, 'FLAML_sample_size': 364083}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 361, 'num_leaves': 11, 'min_child_samples': 32, 'learning_rate': 0.13528717598813866, 'log_max_bin': 9, 'colsample_bytree': 0.9851977789068981, 'reg_alpha': 0.0038372002422749616, 'reg_lambda': 0.25113531892556773, 'FLAML_sample_size': 364083}}\n",
"{'Current Learner': 'lgbm', 'Current Sample': 364083, 'Current Hyper-parameters': {'n_estimators': 547, 'num_leaves': 46, 'min_child_samples': 60, 'learning_rate': 0.281323306091088, 'log_max_bin': 10, 'colsample_bytree': 1.0, 'reg_alpha': 0.001643352694266288, 'reg_lambda': 0.14719738747481906, 'FLAML_sample_size': 364083}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 547, 'num_leaves': 46, 'min_child_samples': 60, 'learning_rate': 0.281323306091088, 'log_max_bin': 10, 'colsample_bytree': 1.0, 'reg_alpha': 0.001643352694266288, 'reg_lambda': 0.14719738747481906, 'FLAML_sample_size': 364083}}\n",
"{'Current Learner': 'lgbm', 'Current Sample': 364083, 'Current Hyper-parameters': {'n_estimators': 1071, 'num_leaves': 25, 'min_child_samples': 36, 'learning_rate': 0.10320258241974468, 'log_max_bin': 10, 'colsample_bytree': 1.0, 'reg_alpha': 0.0009765625, 'reg_lambda': 0.08547376339713011, 'FLAML_sample_size': 364083}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 1071, 'num_leaves': 25, 'min_child_samples': 36, 'learning_rate': 0.10320258241974468, 'log_max_bin': 10, 'colsample_bytree': 1.0, 'reg_alpha': 0.0009765625, 'reg_lambda': 0.08547376339713011, 'FLAML_sample_size': 364083}}\n"
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
]
}
],
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"source": [
"from flaml.data import get_output_from_log\n",
"time_history, best_valid_loss_history, valid_loss_history, config_history, metric_history = \\\n",
" get_output_from_log(filename=settings['log_file_name'], time_budget=240)\n",
"for config in config_history:\n",
" print(config)"
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "code",
"execution_count": 11,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"outputs": [
{
"data": {
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAiN0lEQVR4nO3df5hdVX3v8feHIcAowoAJlEyAxEsSDUUTjFSMVKBqIo9CpIhAfyi2oLditdShpCqlWCo21V59TPUGixSvIJKGMSoyUkBo+ZUMDObH0GBIYpgJmgAZRRxJMvneP/Y+cHKyZ2YnzJ5zzpzP63nOM2evvfbZ3x0O85211t5rKSIwMzOrtF+1AzAzs9rkBGFmZpmcIMzMLJMThJmZZXKCMDOzTE4QZmaWyQnCbB9IOkXS2mrHYVYkJwirO5I2Snp7NWOIiP+KiOlFfb6kuZLulfScpK2S7pF0ZlHnM8viBGGWQVJTFc99DnALcAMwCTgSuAJ4zz58liT5/3PbJ/7i2JghaT9Jl0t6QtIzkr4j6fCy/bdI+rmkX6Z/nR9ftu96SV+VdJuk54HT0pbKJyWtTI+5WdJBaf1TJfWUHT9o3XT/ZZKekrRZ0p9LCknHZVyDgC8Cn42Ir0fELyNiV0TcExEXpXWulPT/yo6ZnH7e/un2jyVdLek+4DdAm6TOivP8laRl6fsDJf2zpE2SfiHpa5KaX+Z/DhsDnCBsLPkYMB94GzAR2AYsKtv/Q2AqcATwCPCtiuMvAK4GXgX8d1p2LjAPmAK8HvjgEOfPrCtpHnAp8HbgOODUIT5jOnA0sGSIOnn8CXAxybV8DZguaWrZ/guAG9P31wDTgJlpfK0kLRZrcE4QNpZ8BPhURPRExAvAlcA5pb+sI+K6iHiubN8bJB1advx3I+K+9C/236ZlX46IzRHxLPA9kl+igxms7rnANyJiTUT8Jj33YF6d/nwq3yUP6vr0fDsj4pfAd4HzAdJE8VpgWdpiuRj4q4h4NiKeA/4ROO9lnt/GACcIG0uOBW6V1CepD3gMGACOlNQk6Zq0++lXwMb0mPFlxz+Z8Zk/L3v/G+DgIc4/WN2JFZ+ddZ6SZ9KfRw1RJ4/Kc9xImiBIWg/tabKaALwCeLjs3+32tNwanBOEjSVPAu+KiJay10ER0UvyS/Eskm6eQ4HJ6TEqO76oqY2fIhlsLjl6iLprSa7jD4eo8zzJL/WS38moU3ktdwATJM0kSRSl7qWngX7g+LJ/s0MjYqhEaA3CCcLq1ThJB5W99ifpa79a0rEAkiZIOiut/yrgBZK/0F9B0o0yWr4DXCjpdZJeAXxmsIqRzL9/KfAZSRdKOiQdfH+rpMVptUeB35d0TNpFtmC4ACJiB8mdUQuBw0kSBhGxC7gW+BdJRwBIapU0d18v1sYOJwirV7eR/OVbel0JfAlYBvxI0nPAg8DvpfVvAH4G9ALd6b5RERE/BL4M3A2sKzv3C4PUXwK8H/gQsBn4BfAPJOMIRMQdwM3ASuBh4Ps5Q7mRpAV1S0TsLCv/m1Jcaffbf5IMlluDkxcMMhtdkl4HrAYOrPhFbVZT3IIwGwWS3ps+b3AY8Hnge04OVuucIMxGx4eBLcATJHdW/e/qhmM2PHcxmZlZJrcgzMws0/7VDmCkjB8/PiZPnlztMMzM6srDDz/8dERkPhg5ZhLE5MmT6ezsHL6imZm9SNLPBtvnLiYzM8vkBGFmZpmcIMzMLFOhCULSPElrJa2TdPkgdc6V1C1pjaQb07LTJD1a9vqtpPlFxmpmZrsrbJA6XbJxEfAOoAdYIWlZRHSX1ZlKMtHYnIjYVposLCLuJp1LP10RbB3wo6JiNTOzPRV5F9NJwLqIWA8g6dsk0y13l9W5CFgUEdsAImJLxuecA/wwnbvezMxS7V29LOxYy+a+fia2NNM2dzrzZ7WO2OcX2cXUyu6LlvSkZeWmAdMk3SfpwXRpxkrnATdlnUDSxZI6JXVu3bp1RII2M6sH7V29LFi6it6+fgLo7etnwdJVtHf1jtg5qj1IvT/JGsGnkixicq2kltJOSUcBJwAdWQdHxOKImB0RsydM8AJYZtY4FnaspX/HwG5l/TsGWNixdsTOUWQXUy+7r5w1KS0r1wM8lC5mskHS4yQJY0W6/1zg1nS/2ctSdHPcbDRt7uvfq/J9UWQLYgUwVdIUSQeQdBUtq6jTTtJ6QNJ4ki6n9WX7z2eQ7iWzvTEazXGz0TSxpXmvyvdFYS2IiNgp6RKS7qEm4LqIWCPpKqAzIpal+94pqZtkCuS2iHgGQNJkkhbIPUXFaI1jsOb4ZUtWctPyTVWKymzfHTRuP/YT7CqbkLt5XBNtc0duMcBC52KKiNtIloYsL7ui7H1p/d1LM47dyJ6D2mb7ZLBm9/aBXaMcidnIGH/wgQA8+Ww/2wd20VpAt+mYmazPbCgTW5rpzUgSrS3N3Pzhk6sQkVntq/ZdTGajom3udJrHNe1WNtLNcbOxxi0IawilZvdlS1YW1hw3G2ucIKxhzJ/V+uKAtLuVzIbnLiYzM8vkBGFmZpmcIMzMLJMThJmZZXKCMDOzTE4QZmaWyQnCzMwyOUGYmVkmJwgzM8vkBGFmZpmcIMzMLJMThJmZZXKCMDOzTE4QZmaWyQnCzMwyOUGYmVkmJwgzM8vkBGFmZpmcIMzMLJMThJmZZXKCMDOzTE4QZmaWyQnCzMwyOUGYmVkmJwgzM8vkBGFmZpmcIMzMLJMThJmZZXKCMDOzTIUmCEnzJK2VtE7S5YPUOVdSt6Q1km4sKz9G0o8kPZbun1xkrGZmtrv9i/pgSU3AIuAdQA+wQtKyiOguqzMVWADMiYhtko4o+4gbgKsj4g5JBwO7iorVzMz2VFiCAE4C1kXEegBJ3wbOArrL6lwELIqIbQARsSWtOwPYPyLuSMt/XWCcNau9q5eFHWvZ3NfPxJZm2uZOZ/6s1mqHZWYNosguplbgybLtnrSs3DRgmqT7JD0oaV5ZeZ+kpZK6JC1MWyS7kXSxpE5JnVu3bi3kIqqlvauXBUtX0dvXTwC9ff0sWLqK9q7eaodmZg2iyBZE3vNPBU4FJgH3SjohLT8FmAVsAm4GPgj8W/nBEbEYWAwwe/bsGK2gR8PCjrX07xjYrax/xwCXLVnJTcs3VSmq+tf91K+YcdQh1Q7DrC4U2YLoBY4u256UlpXrAZZFxI6I2AA8TpIweoBHI2J9ROwE2oETC4y15mzu688s3z7goZiXY8ZRh3DWTHfTmeVRZAtiBTBV0hSSxHAecEFFnXbgfOAbksaTdC2tB/qAFkkTImIrcDrQWWCsNWdiSzO9GUmitaWZmz98chUiMrNGU1gLIv3L/xKgA3gM+E5ErJF0laQz02odwDOSuoG7gbaIeCYiBoBPAndKWgUIuLaoWGtR29zpNI/bfdileVwTbXOnVykiM2s0ihgbXfezZ8+Ozs6x1cho7+rlsiUr2T6wi1bfxWRmBZD0cETMztpX7UFqG8L8Wa0vDki7W8nMRpun2jAzs0xOEGZmlskJwszMMjlBmJlZpmEThKRXj0YgZmZWW/K0IB6UdIukMySp8IjMzKwm5EkQ00jmO/oT4KeS/lHStGLDMjOzahs2QUTijog4n2R67g8AyyXdI8k355uZjVHDPiiXjkH8MUkL4hfAx4BlwEzgFmBKgfGZmVmV5HmS+gHgm8D8iOgpK++U9LViwjIzs2rLkyCmxyATNkXE50c4HjMzqxF5Bql/JKmltCHpMEkdxYVkZma1IE+CmBARfaWNdP3oIwqLyMzMakKeBDEg6ZjShqRjgbExR7iZmQ0qzxjEp4D/lnQPycI9pwAXFxqVmZlV3bAJIiJul3Qi8Oa06BMR8XSxYZmZWbXlXTBoANgCHATMkERE3FtcWGZmVm15HpT7c+DjwCTgUZKWxAPA6YVGZmZmVZVnkPrjwJuAn0XEacAsoK/IoMzMrPryJIjfRsRvASQdGBH/A0wvNiwzM6u2PGMQPemDcu3AHZK2AT8rMigzM6u+PHcxvTd9e6Wku4FDgdsLjcrMzKpuyAQhqQlYExGvBYiIe0YlKjMzq7ohxyA
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
warning -> info for low cost partial config (#231) * warning -> info for low cost partial config #195, #110 * when n_estimators < 0, use trained_estimator's * log debug info * test random seed * remove "objective"; avoid ZeroDivisionError * hp config to estimator params * check type of searcher * default n_jobs * try import * Update searchalgo_auto.py * CLASSIFICATION * auto_augment flag * min_sample_size * make catboost optional
2021-10-08 16:09:43 -07:00
]
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
},
"metadata": {
"needs_background": "light"
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
},
"output_type": "display_data"
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
}
],
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"\n",
"plt.title('Learning Curve')\n",
"plt.xlabel('Wall Clock Time (s)')\n",
"plt.ylabel('Validation Accuracy')\n",
"plt.scatter(time_history, 1 - np.array(valid_loss_history))\n",
"plt.step(time_history, 1 - np.array(best_valid_loss_history), where='post')\n",
"plt.show()"
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "markdown",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"source": [
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"## 3. Comparison with alternatives\n"
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
},
{
"cell_type": "markdown",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"source": [
"### Default LightGBM"
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
},
{
"cell_type": "code",
"execution_count": 12,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
"outputs": [],
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"source": [
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"from lightgbm import LGBMClassifier\n",
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"lgbm = LGBMClassifier()"
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
},
{
"cell_type": "code",
"execution_count": 13,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"outputs": [
{
"data": {
"text/plain": [
"LGBMClassifier()"
]
},
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"execution_count": 13,
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"metadata": {},
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"output_type": "execute_result"
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
}
],
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"source": [
"lgbm.fit(X_train, y_train)"
]
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
},
{
"cell_type": "code",
"execution_count": 14,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
"outputs": [],
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"source": [
"y_pred_lgbm = lgbm.predict(X_test)"
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
},
{
"cell_type": "markdown",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"source": [
"### Default XGBoost"
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
},
{
"cell_type": "code",
"execution_count": 15,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"outputs": [],
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"source": [
"from xgboost import XGBClassifier\n",
"xgb = XGBClassifier()\n",
"cat_columns = X_train.select_dtypes(include=['category']).columns\n",
"X = X_train.copy()\n",
"X[cat_columns] = X[cat_columns].apply(lambda x: x.cat.codes)\n"
]
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
},
{
"cell_type": "code",
"execution_count": 16,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"outputs": [
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/site-packages/xgboost/sklearn.py:888: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].\n",
" warnings.warn(label_encoder_deprecation_msg, UserWarning)\n"
]
},
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
{
"data": {
"text/plain": [
"XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
" colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
" importance_type='gain', interaction_constraints='',\n",
" learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n",
" min_child_weight=1, missing=nan, monotone_constraints='()',\n",
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
" n_estimators=100, n_jobs=32, num_parallel_tree=1, random_state=0,\n",
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
" reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=1,\n",
" tree_method='exact', validate_parameters=1, verbosity=None)"
]
},
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"execution_count": 16,
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"metadata": {},
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"output_type": "execute_result"
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
}
],
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"source": [
"xgb.fit(X, y_train)"
]
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
},
{
"cell_type": "code",
"execution_count": 17,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
"outputs": [],
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"source": [
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"X = X_test.copy()\n",
"X[cat_columns] = X[cat_columns].apply(lambda x: x.cat.codes)\n",
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"y_pred_xgb = xgb.predict(X)"
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
},
{
"cell_type": "code",
"execution_count": 18,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"outputs": [
{
"name": "stdout",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"output_type": "stream",
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"text": [
"default xgboost accuracy = 0.6676060098186078\n",
"default lgbm accuracy = 0.6602346380315323\n",
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"flaml (4min) accuracy = 0.6720332824110467\n"
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
]
}
],
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"source": [
"print('default xgboost accuracy', '=', 1 - sklearn_metric_loss_score('accuracy', y_pred_xgb, y_test))\n",
"print('default lgbm accuracy', '=', 1 - sklearn_metric_loss_score('accuracy', y_pred_lgbm, y_test))\n",
"print('flaml (4min) accuracy', '=', 1 - sklearn_metric_loss_score('accuracy', y_pred, y_test))"
]
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
},
{
"cell_type": "markdown",
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
},
"source": [
"## 4. Customized Learner"
]
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
},
{
"cell_type": "markdown",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"source": [
"Some experienced automl users may have a preferred model to tune or may already have a reasonably by-hand-tuned model before launching the automl experiment. They need to select optimal configurations for the customized model mixed with standard built-in learners. \n",
"\n",
"FLAML can easily incorporate customized/new learners (preferably with sklearn API) provided by users in a real-time manner, as demonstrated below."
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
},
{
"cell_type": "markdown",
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
},
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"source": [
"### Example of Regularized Greedy Forest\n",
"\n",
"[Regularized Greedy Forest](https://arxiv.org/abs/1109.0887) (RGF) is a machine learning method currently not included in FLAML. The RGF has many tuning parameters, the most critical of which are: `[max_leaf, n_iter, n_tree_search, opt_interval, min_samples_leaf]`. To run a customized/new learner, the user needs to provide the following information:\n",
"* an implementation of the customized/new learner\n",
"* a list of hyperparameter names and types\n",
"* rough ranges of hyperparameters (i.e., upper/lower bounds)\n",
"* choose initial value corresponding to low cost for cost-related hyperparameters (e.g., initial value for max_leaf and n_iter should be small)\n",
"\n",
"In this example, the above information for RGF is wrapped in a python class called *MyRegularizedGreedyForest* that exposes the hyperparameters."
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
},
{
"cell_type": "code",
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"execution_count": 19,
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
},
"outputs": [],
"source": [
"''' SKLearnEstimator is the super class for a sklearn learner '''\n",
"from flaml.model import SKLearnEstimator\n",
"from flaml import tune\n",
"from flaml.data import CLASSIFICATION\n",
"\n",
"\n",
"class MyRegularizedGreedyForest(SKLearnEstimator):\n",
" def __init__(self, task='binary', **config):\n",
" '''Constructor\n",
" \n",
" Args:\n",
" task: A string of the task type, one of\n",
" 'binary', 'multi', 'regression'\n",
" config: A dictionary containing the hyperparameter names\n",
" and 'n_jobs' as keys. n_jobs is the number of parallel threads.\n",
" '''\n",
"\n",
" super().__init__(task, **config)\n",
"\n",
" '''task=binary or multi for classification task'''\n",
" if task in CLASSIFICATION:\n",
" from rgf.sklearn import RGFClassifier\n",
"\n",
" self.estimator_class = RGFClassifier\n",
" else:\n",
" from rgf.sklearn import RGFRegressor\n",
" \n",
" self.estimator_class = RGFRegressor\n",
"\n",
" @classmethod\n",
" def search_space(cls, data_size, task):\n",
" '''[required method] search space\n",
"\n",
" Returns:\n",
" A dictionary of the search space. \n",
" Each key is the name of a hyperparameter, and value is a dict with\n",
" its domain (required) and low_cost_init_value, init_value,\n",
" cat_hp_cost (if applicable).\n",
" e.g.,\n",
" {'domain': tune.randint(lower=1, upper=10), 'init_value': 1}.\n",
" '''\n",
" space = { \n",
" 'max_leaf': {'domain': tune.lograndint(lower=4, upper=data_size), 'init_value': 4, 'low_cost_init_value': 4},\n",
" 'n_iter': {'domain': tune.lograndint(lower=1, upper=data_size), 'init_value': 1, 'low_cost_init_value': 1},\n",
" 'n_tree_search': {'domain': tune.lograndint(lower=1, upper=32768), 'init_value': 1, 'low_cost_init_value': 1},\n",
" 'opt_interval': {'domain': tune.lograndint(lower=1, upper=10000), 'init_value': 100},\n",
" 'learning_rate': {'domain': tune.loguniform(lower=0.01, upper=20.0)},\n",
" 'min_samples_leaf': {'domain': tune.lograndint(lower=1, upper=20), 'init_value': 20},\n",
" }\n",
" return space\n",
"\n",
" @classmethod\n",
" def size(cls, config):\n",
" '''[optional method] memory size of the estimator in bytes\n",
" \n",
" Args:\n",
" config - the dict of the hyperparameter config\n",
"\n",
" Returns:\n",
" A float of the memory size required by the estimator to train the\n",
" given config\n",
" '''\n",
" max_leaves = int(round(config['max_leaf']))\n",
" n_estimators = int(round(config['n_iter']))\n",
" return (max_leaves * 3 + (max_leaves - 1) * 4 + 1.0) * n_estimators * 8\n",
"\n",
" @classmethod\n",
" def cost_relative2lgbm(cls):\n",
" '''[optional method] relative cost compared to lightgbm\n",
" '''\n",
" return 1.0\n"
]
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
},
"source": [
"### Add Customized Learner and Run FLAML AutoML\n",
"\n",
"After adding RGF into the list of learners, we run automl by tuning hyperpameters of RGF as well as the default learners. "
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "code",
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"execution_count": 20,
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
},
"outputs": [],
"source": [
"automl = AutoML()\n",
"automl.add_learner(learner_name='RGF', learner_class=MyRegularizedGreedyForest)"
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "code",
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"execution_count": 21,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {
"slideshow": {
"slide_type": "slide"
},
"tags": []
},
warning -> info for low cost partial config (#231) * warning -> info for low cost partial config #195, #110 * when n_estimators < 0, use trained_estimator's * log debug info * test random seed * remove "objective"; avoid ZeroDivisionError * hp config to estimator params * check type of searcher * default n_jobs * try import * Update searchalgo_auto.py * CLASSIFICATION * auto_augment flag * min_sample_size * make catboost optional
2021-10-08 16:09:43 -07:00
"outputs": [
{
"name": "stderr",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"output_type": "stream",
warning -> info for low cost partial config (#231) * warning -> info for low cost partial config #195, #110 * when n_estimators < 0, use trained_estimator's * log debug info * test random seed * remove "objective"; avoid ZeroDivisionError * hp config to estimator params * check type of searcher * default n_jobs * try import * Update searchalgo_auto.py * CLASSIFICATION * auto_augment flag * min_sample_size * make catboost optional
2021-10-08 16:09:43 -07:00
"text": [
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"[flaml.automl: 11-23 01:37:21] {1861} INFO - task = classification\n",
"[flaml.automl: 11-23 01:37:21] {1863} INFO - Data split method: stratified\n",
"[flaml.automl: 11-23 01:37:21] {1867} INFO - Evaluation method: holdout\n",
"[flaml.automl: 11-23 01:37:22] {1933} INFO - Minimizing error metric: 1-accuracy\n",
"[flaml.automl: 11-23 01:37:22] {1985} INFO - List of ML learners in AutoML Run: ['RGF', 'lgbm', 'rf', 'xgboost']\n",
"[flaml.automl: 11-23 01:37:22] {2223} INFO - iteration 0, current learner RGF\n",
"/usr/local/lib/python3.7/site-packages/rgf/utils.py:224: UserWarning: Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage.\n",
warning -> info for low cost partial config (#231) * warning -> info for low cost partial config #195, #110 * when n_estimators < 0, use trained_estimator's * log debug info * test random seed * remove "objective"; avoid ZeroDivisionError * hp config to estimator params * check type of searcher * default n_jobs * try import * Update searchalgo_auto.py * CLASSIFICATION * auto_augment flag * min_sample_size * make catboost optional
2021-10-08 16:09:43 -07:00
" warnings.warn(\"Cannot find FastRGF executable files. \"\n",
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"[flaml.automl: 11-23 01:37:23] {2337} INFO - Estimated sufficient time budget=396003s. Estimated necessary time budget=396s.\n",
"[flaml.automl: 11-23 01:37:23] {2417} INFO - at 1.6s,\testimator RGF's best error=0.3787,\tbest estimator RGF's best error=0.3787\n",
"[flaml.automl: 11-23 01:37:23] {2223} INFO - iteration 1, current learner RGF\n",
"[flaml.automl: 11-23 01:37:23] {2417} INFO - at 2.2s,\testimator RGF's best error=0.3787,\tbest estimator RGF's best error=0.3787\n",
"[flaml.automl: 11-23 01:37:23] {2223} INFO - iteration 2, current learner RGF\n",
"[flaml.automl: 11-23 01:37:24] {2417} INFO - at 2.8s,\testimator RGF's best error=0.3787,\tbest estimator RGF's best error=0.3787\n",
"[flaml.automl: 11-23 01:37:24] {2223} INFO - iteration 3, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:24] {2417} INFO - at 2.9s,\testimator lgbm's best error=0.4459,\tbest estimator RGF's best error=0.3787\n",
"[flaml.automl: 11-23 01:37:24] {2223} INFO - iteration 4, current learner RGF\n",
"[flaml.automl: 11-23 01:37:25] {2417} INFO - at 3.5s,\testimator RGF's best error=0.3787,\tbest estimator RGF's best error=0.3787\n",
"[flaml.automl: 11-23 01:37:25] {2223} INFO - iteration 5, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:25] {2417} INFO - at 3.6s,\testimator lgbm's best error=0.4459,\tbest estimator RGF's best error=0.3787\n",
"[flaml.automl: 11-23 01:37:25] {2223} INFO - iteration 6, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:25] {2417} INFO - at 3.6s,\testimator lgbm's best error=0.3787,\tbest estimator RGF's best error=0.3787\n",
"[flaml.automl: 11-23 01:37:25] {2223} INFO - iteration 7, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:25] {2417} INFO - at 3.7s,\testimator lgbm's best error=0.3653,\tbest estimator lgbm's best error=0.3653\n",
"[flaml.automl: 11-23 01:37:25] {2223} INFO - iteration 8, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:25] {2417} INFO - at 3.7s,\testimator lgbm's best error=0.3646,\tbest estimator lgbm's best error=0.3646\n",
"[flaml.automl: 11-23 01:37:25] {2223} INFO - iteration 9, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:25] {2417} INFO - at 3.9s,\testimator lgbm's best error=0.3646,\tbest estimator lgbm's best error=0.3646\n",
"[flaml.automl: 11-23 01:37:25] {2223} INFO - iteration 10, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:25] {2417} INFO - at 3.9s,\testimator lgbm's best error=0.3646,\tbest estimator lgbm's best error=0.3646\n",
"[flaml.automl: 11-23 01:37:25] {2223} INFO - iteration 11, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:25] {2417} INFO - at 4.0s,\testimator lgbm's best error=0.3611,\tbest estimator lgbm's best error=0.3611\n",
"[flaml.automl: 11-23 01:37:25] {2223} INFO - iteration 12, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:25] {2417} INFO - at 4.1s,\testimator lgbm's best error=0.3611,\tbest estimator lgbm's best error=0.3611\n",
"[flaml.automl: 11-23 01:37:25] {2223} INFO - iteration 13, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:25] {2417} INFO - at 4.2s,\testimator lgbm's best error=0.3611,\tbest estimator lgbm's best error=0.3611\n",
"[flaml.automl: 11-23 01:37:25] {2223} INFO - iteration 14, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:25] {2417} INFO - at 4.3s,\testimator lgbm's best error=0.3611,\tbest estimator lgbm's best error=0.3611\n",
"[flaml.automl: 11-23 01:37:25] {2223} INFO - iteration 15, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:26] {2417} INFO - at 4.4s,\testimator lgbm's best error=0.3611,\tbest estimator lgbm's best error=0.3611\n",
"[flaml.automl: 11-23 01:37:26] {2223} INFO - iteration 16, current learner RGF\n",
"[flaml.automl: 11-23 01:37:26] {2417} INFO - at 5.0s,\testimator RGF's best error=0.3787,\tbest estimator lgbm's best error=0.3611\n",
"[flaml.automl: 11-23 01:37:26] {2223} INFO - iteration 17, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:26] {2417} INFO - at 5.1s,\testimator lgbm's best error=0.3567,\tbest estimator lgbm's best error=0.3567\n",
"[flaml.automl: 11-23 01:37:26] {2223} INFO - iteration 18, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:26] {2417} INFO - at 5.2s,\testimator lgbm's best error=0.3537,\tbest estimator lgbm's best error=0.3537\n",
"[flaml.automl: 11-23 01:37:26] {2223} INFO - iteration 19, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:27] {2417} INFO - at 5.4s,\testimator lgbm's best error=0.3537,\tbest estimator lgbm's best error=0.3537\n",
"[flaml.automl: 11-23 01:37:27] {2223} INFO - iteration 20, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:27] {2417} INFO - at 5.6s,\testimator lgbm's best error=0.3537,\tbest estimator lgbm's best error=0.3537\n",
"[flaml.automl: 11-23 01:37:27] {2223} INFO - iteration 21, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:27] {2417} INFO - at 5.7s,\testimator lgbm's best error=0.3537,\tbest estimator lgbm's best error=0.3537\n",
"[flaml.automl: 11-23 01:37:27] {2223} INFO - iteration 22, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:27] {2417} INFO - at 5.9s,\testimator lgbm's best error=0.3537,\tbest estimator lgbm's best error=0.3537\n",
"[flaml.automl: 11-23 01:37:27] {2223} INFO - iteration 23, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:27] {2417} INFO - at 6.3s,\testimator lgbm's best error=0.3481,\tbest estimator lgbm's best error=0.3481\n",
"[flaml.automl: 11-23 01:37:27] {2223} INFO - iteration 24, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:28] {2417} INFO - at 6.7s,\testimator lgbm's best error=0.3481,\tbest estimator lgbm's best error=0.3481\n",
"[flaml.automl: 11-23 01:37:28] {2223} INFO - iteration 25, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:29] {2417} INFO - at 7.5s,\testimator lgbm's best error=0.3419,\tbest estimator lgbm's best error=0.3419\n",
"[flaml.automl: 11-23 01:37:29] {2223} INFO - iteration 26, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:30] {2417} INFO - at 8.7s,\testimator lgbm's best error=0.3419,\tbest estimator lgbm's best error=0.3419\n",
"[flaml.automl: 11-23 01:37:30] {2223} INFO - iteration 27, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:31] {2417} INFO - at 9.4s,\testimator lgbm's best error=0.3419,\tbest estimator lgbm's best error=0.3419\n",
"[flaml.automl: 11-23 01:37:31] {2223} INFO - iteration 28, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:31] {2417} INFO - at 9.4s,\testimator xgboost's best error=0.3787,\tbest estimator lgbm's best error=0.3419\n",
"[flaml.automl: 11-23 01:37:31] {2223} INFO - iteration 29, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:31] {2417} INFO - at 9.5s,\testimator xgboost's best error=0.3787,\tbest estimator lgbm's best error=0.3419\n",
"[flaml.automl: 11-23 01:37:31] {2223} INFO - iteration 30, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:31] {2417} INFO - at 9.5s,\testimator xgboost's best error=0.3765,\tbest estimator lgbm's best error=0.3419\n",
"[flaml.automl: 11-23 01:37:31] {2223} INFO - iteration 31, current learner rf\n",
"[flaml.automl: 11-23 01:37:31] {2417} INFO - at 9.7s,\testimator rf's best error=0.3777,\tbest estimator lgbm's best error=0.3419\n",
"[flaml.automl: 11-23 01:37:31] {2223} INFO - iteration 32, current learner rf\n",
"[flaml.automl: 11-23 01:37:31] {2417} INFO - at 9.8s,\testimator rf's best error=0.3710,\tbest estimator lgbm's best error=0.3419\n",
"[flaml.automl: 11-23 01:37:31] {2223} INFO - iteration 33, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:31] {2417} INFO - at 9.8s,\testimator xgboost's best error=0.3765,\tbest estimator lgbm's best error=0.3419\n",
"[flaml.automl: 11-23 01:37:31] {2223} INFO - iteration 34, current learner rf\n",
"[flaml.automl: 11-23 01:37:31] {2417} INFO - at 10.0s,\testimator rf's best error=0.3710,\tbest estimator lgbm's best error=0.3419\n",
"[flaml.automl: 11-23 01:37:32] {2629} INFO - retrain lgbm for 0.6s\n",
"[flaml.automl: 11-23 01:37:32] {2634} INFO - retrained model: LGBMClassifier(colsample_bytree=0.6821331092849664,\n",
" learning_rate=0.2532267767742095, max_bin=255,\n",
" min_child_samples=3, n_estimators=188, num_leaves=9,\n",
" reg_alpha=0.0009765625, reg_lambda=0.05089268633400602,\n",
warning -> info for low cost partial config (#231) * warning -> info for low cost partial config #195, #110 * when n_estimators < 0, use trained_estimator's * log debug info * test random seed * remove "objective"; avoid ZeroDivisionError * hp config to estimator params * check type of searcher * default n_jobs * try import * Update searchalgo_auto.py * CLASSIFICATION * auto_augment flag * min_sample_size * make catboost optional
2021-10-08 16:09:43 -07:00
" verbose=-1)\n",
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"[flaml.automl: 11-23 01:37:32] {2014} INFO - fit succeeded\n",
"[flaml.automl: 11-23 01:37:32] {2016} INFO - Time taken to find the best model: 7.508742332458496\n",
"[flaml.automl: 11-23 01:37:32] {2030} WARNING - Time taken to find the best model is 75% of the provided time budget and not all estimators' hyperparameter search converged. Consider increasing the time budget.\n"
warning -> info for low cost partial config (#231) * warning -> info for low cost partial config #195, #110 * when n_estimators < 0, use trained_estimator's * log debug info * test random seed * remove "objective"; avoid ZeroDivisionError * hp config to estimator params * check type of searcher * default n_jobs * try import * Update searchalgo_auto.py * CLASSIFICATION * auto_augment flag * min_sample_size * make catboost optional
2021-10-08 16:09:43 -07:00
]
}
],
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"source": [
"settings = {\n",
" \"time_budget\": 10, # total running time in seconds\n",
" \"metric\": 'accuracy', \n",
" \"estimator_list\": ['RGF', 'lgbm', 'rf', 'xgboost'], # list of ML learners\n",
" \"task\": 'classification', # task type \n",
" \"log_file_name\": 'airlines_experiment_custom_learner.log', # flaml log file \n",
" \"log_training_metric\": True, # whether to log training metric\n",
"}\n",
"\n",
"automl.fit(X_train = X_train, y_train = y_train, **settings)"
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
coverage (#135) * coverage * readme * timeout
2021-07-20 17:00:44 -07:00
"cell_type": "markdown",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"source": [
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"## 5. Customized Metric\n",
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"\n",
test -> val; docstr (#300) * rename test -> val in custom metric function * add an example in docstr resolve #299
2021-11-22 22:17:29 -08:00
"It's also easy to customize the optimization metric. As an example, we demonstrate with a custom metric function which combines training loss and validation loss as the final loss to minimize."
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "code",
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"execution_count": 22,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
"outputs": [],
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"source": [
test -> val; docstr (#300) * rename test -> val in custom metric function * add an example in docstr resolve #299
2021-11-22 22:17:29 -08:00
"def custom_metric(X_val, y_val, estimator, labels, X_train, y_train,\n",
" weight_val=None, weight_train=None, config=None,\n",
" groups_val=None, groups_train=None):\n",
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
" from sklearn.metrics import log_loss\n",
" import time\n",
" start = time.time()\n",
test -> val; docstr (#300) * rename test -> val in custom metric function * add an example in docstr resolve #299
2021-11-22 22:17:29 -08:00
" y_pred = estimator.predict_proba(X_val)\n",
" pred_time = (time.time() - start) / len(X_val)\n",
" val_loss = log_loss(y_val, y_pred, labels=labels,\n",
" sample_weight=weight_val)\n",
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
" y_pred = estimator.predict_proba(X_train)\n",
" train_loss = log_loss(y_train, y_pred, labels=labels,\n",
" sample_weight=weight_train)\n",
" alpha = 0.5\n",
test -> val; docstr (#300) * rename test -> val in custom metric function * add an example in docstr resolve #299
2021-11-22 22:17:29 -08:00
" return val_loss * (1 + alpha) - alpha * train_loss, {\n",
" \"val_loss\": val_loss, \"train_loss\": train_loss, \"pred_time\": pred_time\n",
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
" }\n",
" # two elements are returned:\n",
" # the first element is the metric to minimize as a float number,\n",
" # the second element is a dictionary of the metrics to log"
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
coverage (#135) * coverage * readme * timeout
2021-07-20 17:00:44 -07:00
"cell_type": "markdown",
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"source": [
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"We can then pass this custom metric function to automl's `fit` method."
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
{
"cell_type": "code",
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"execution_count": 23,
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"[flaml.automl: 11-23 01:37:32] {1861} INFO - task = classification\n",
"[flaml.automl: 11-23 01:37:32] {1863} INFO - Data split method: stratified\n",
"[flaml.automl: 11-23 01:37:32] {1867} INFO - Evaluation method: holdout\n",
"[flaml.automl: 11-23 01:37:32] {1933} INFO - Minimizing error metric: customized metric\n",
"[flaml.automl: 11-23 01:37:32] {1985} INFO - List of ML learners in AutoML Run: ['lgbm', 'rf', 'catboost', 'xgboost', 'extra_tree', 'xgb_limitdepth', 'lrl1']\n",
"[flaml.automl: 11-23 01:37:32] {2223} INFO - iteration 0, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:32] {2337} INFO - Estimated sufficient time budget=13940s. Estimated necessary time budget=342s.\n",
"[flaml.automl: 11-23 01:37:32] {2417} INFO - at 0.6s,\testimator lgbm's best error=0.6647,\tbest estimator lgbm's best error=0.6647\n",
"[flaml.automl: 11-23 01:37:32] {2223} INFO - iteration 1, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:32] {2417} INFO - at 0.6s,\testimator lgbm's best error=0.6647,\tbest estimator lgbm's best error=0.6647\n",
"[flaml.automl: 11-23 01:37:32] {2223} INFO - iteration 2, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:32] {2417} INFO - at 0.6s,\testimator lgbm's best error=0.6491,\tbest estimator lgbm's best error=0.6491\n",
"[flaml.automl: 11-23 01:37:32] {2223} INFO - iteration 3, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:33] {2417} INFO - at 0.7s,\testimator xgboost's best error=0.6672,\tbest estimator lgbm's best error=0.6491\n",
"[flaml.automl: 11-23 01:37:33] {2223} INFO - iteration 4, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:33] {2417} INFO - at 0.8s,\testimator lgbm's best error=0.6423,\tbest estimator lgbm's best error=0.6423\n",
"[flaml.automl: 11-23 01:37:33] {2223} INFO - iteration 5, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:33] {2417} INFO - at 0.9s,\testimator lgbm's best error=0.6423,\tbest estimator lgbm's best error=0.6423\n",
"[flaml.automl: 11-23 01:37:33] {2223} INFO - iteration 6, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:33] {2417} INFO - at 1.0s,\testimator lgbm's best error=0.6400,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:33] {2223} INFO - iteration 7, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:33] {2417} INFO - at 1.0s,\testimator lgbm's best error=0.6400,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:33] {2223} INFO - iteration 8, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:33] {2417} INFO - at 1.1s,\testimator lgbm's best error=0.6400,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:33] {2223} INFO - iteration 9, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:33] {2417} INFO - at 1.2s,\testimator xgboost's best error=0.6672,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:33] {2223} INFO - iteration 10, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:33] {2417} INFO - at 1.2s,\testimator xgboost's best error=0.6500,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:33] {2223} INFO - iteration 11, current learner extra_tree\n",
"[flaml.automl: 11-23 01:37:33] {2417} INFO - at 1.7s,\testimator extra_tree's best error=0.6548,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:33] {2223} INFO - iteration 12, current learner rf\n",
"[flaml.automl: 11-23 01:37:34] {2417} INFO - at 2.1s,\testimator rf's best error=0.6474,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:34] {2223} INFO - iteration 13, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:34] {2417} INFO - at 2.2s,\testimator xgboost's best error=0.6413,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:34] {2223} INFO - iteration 14, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:34] {2417} INFO - at 2.2s,\testimator xgboost's best error=0.6413,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:34] {2223} INFO - iteration 15, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:34] {2417} INFO - at 2.3s,\testimator xgboost's best error=0.6413,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:34] {2223} INFO - iteration 16, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:34] {2417} INFO - at 2.4s,\testimator lgbm's best error=0.6400,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:34] {2223} INFO - iteration 17, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:34] {2417} INFO - at 2.4s,\testimator xgboost's best error=0.6413,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:34] {2223} INFO - iteration 18, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:34] {2417} INFO - at 2.5s,\testimator lgbm's best error=0.6400,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:34] {2223} INFO - iteration 19, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:34] {2417} INFO - at 2.6s,\testimator xgboost's best error=0.6413,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:34] {2223} INFO - iteration 20, current learner rf\n",
"[flaml.automl: 11-23 01:37:35] {2417} INFO - at 2.8s,\testimator rf's best error=0.6474,\tbest estimator lgbm's best error=0.6400\n",
"[flaml.automl: 11-23 01:37:35] {2223} INFO - iteration 21, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:35] {2417} INFO - at 2.8s,\testimator xgboost's best error=0.6393,\tbest estimator xgboost's best error=0.6393\n",
"[flaml.automl: 11-23 01:37:35] {2223} INFO - iteration 22, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:35] {2417} INFO - at 2.9s,\testimator lgbm's best error=0.6335,\tbest estimator lgbm's best error=0.6335\n",
"[flaml.automl: 11-23 01:37:35] {2223} INFO - iteration 23, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:35] {2417} INFO - at 3.0s,\testimator lgbm's best error=0.6335,\tbest estimator lgbm's best error=0.6335\n",
"[flaml.automl: 11-23 01:37:35] {2223} INFO - iteration 24, current learner extra_tree\n",
"[flaml.automl: 11-23 01:37:35] {2417} INFO - at 3.1s,\testimator extra_tree's best error=0.6548,\tbest estimator lgbm's best error=0.6335\n",
"[flaml.automl: 11-23 01:37:35] {2223} INFO - iteration 25, current learner extra_tree\n",
"[flaml.automl: 11-23 01:37:35] {2417} INFO - at 3.3s,\testimator extra_tree's best error=0.6548,\tbest estimator lgbm's best error=0.6335\n",
"[flaml.automl: 11-23 01:37:35] {2223} INFO - iteration 26, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:35] {2417} INFO - at 3.3s,\testimator lgbm's best error=0.6335,\tbest estimator lgbm's best error=0.6335\n",
"[flaml.automl: 11-23 01:37:35] {2223} INFO - iteration 27, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:35] {2417} INFO - at 3.4s,\testimator xgboost's best error=0.6342,\tbest estimator lgbm's best error=0.6335\n",
"[flaml.automl: 11-23 01:37:35] {2223} INFO - iteration 28, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:35] {2417} INFO - at 3.5s,\testimator xgboost's best error=0.6342,\tbest estimator lgbm's best error=0.6335\n",
"[flaml.automl: 11-23 01:37:35] {2223} INFO - iteration 29, current learner extra_tree\n",
"[flaml.automl: 11-23 01:37:35] {2417} INFO - at 3.6s,\testimator extra_tree's best error=0.6548,\tbest estimator lgbm's best error=0.6335\n",
"[flaml.automl: 11-23 01:37:35] {2223} INFO - iteration 30, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:36] {2417} INFO - at 3.7s,\testimator lgbm's best error=0.6335,\tbest estimator lgbm's best error=0.6335\n",
"[flaml.automl: 11-23 01:37:36] {2223} INFO - iteration 31, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:36] {2417} INFO - at 3.8s,\testimator lgbm's best error=0.6335,\tbest estimator lgbm's best error=0.6335\n",
"[flaml.automl: 11-23 01:37:36] {2223} INFO - iteration 32, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:36] {2417} INFO - at 3.9s,\testimator lgbm's best error=0.6335,\tbest estimator lgbm's best error=0.6335\n",
"[flaml.automl: 11-23 01:37:36] {2223} INFO - iteration 33, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:36] {2417} INFO - at 4.0s,\testimator xgboost's best error=0.6342,\tbest estimator lgbm's best error=0.6335\n",
"[flaml.automl: 11-23 01:37:36] {2223} INFO - iteration 34, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:36] {2417} INFO - at 4.3s,\testimator lgbm's best error=0.6328,\tbest estimator lgbm's best error=0.6328\n",
"[flaml.automl: 11-23 01:37:36] {2223} INFO - iteration 35, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:36] {2417} INFO - at 4.4s,\testimator xgboost's best error=0.6308,\tbest estimator xgboost's best error=0.6308\n",
"[flaml.automl: 11-23 01:37:36] {2223} INFO - iteration 36, current learner extra_tree\n",
"[flaml.automl: 11-23 01:37:36] {2417} INFO - at 4.6s,\testimator extra_tree's best error=0.6548,\tbest estimator xgboost's best error=0.6308\n",
"[flaml.automl: 11-23 01:37:36] {2223} INFO - iteration 37, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:37] {2417} INFO - at 4.7s,\testimator xgboost's best error=0.6308,\tbest estimator xgboost's best error=0.6308\n",
"[flaml.automl: 11-23 01:37:37] {2223} INFO - iteration 38, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:37] {2417} INFO - at 4.8s,\testimator xgboost's best error=0.6308,\tbest estimator xgboost's best error=0.6308\n",
"[flaml.automl: 11-23 01:37:37] {2223} INFO - iteration 39, current learner rf\n",
"[flaml.automl: 11-23 01:37:37] {2417} INFO - at 4.9s,\testimator rf's best error=0.6474,\tbest estimator xgboost's best error=0.6308\n",
"[flaml.automl: 11-23 01:37:37] {2223} INFO - iteration 40, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:37] {2417} INFO - at 5.0s,\testimator xgboost's best error=0.6308,\tbest estimator xgboost's best error=0.6308\n",
"[flaml.automl: 11-23 01:37:37] {2223} INFO - iteration 41, current learner extra_tree\n",
"[flaml.automl: 11-23 01:37:37] {2417} INFO - at 5.2s,\testimator extra_tree's best error=0.6548,\tbest estimator xgboost's best error=0.6308\n",
"[flaml.automl: 11-23 01:37:37] {2223} INFO - iteration 42, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:37] {2417} INFO - at 5.4s,\testimator lgbm's best error=0.6276,\tbest estimator lgbm's best error=0.6276\n",
"[flaml.automl: 11-23 01:37:37] {2223} INFO - iteration 43, current learner xgboost\n",
"[flaml.automl: 11-23 01:37:37] {2417} INFO - at 5.6s,\testimator xgboost's best error=0.6308,\tbest estimator lgbm's best error=0.6276\n",
"[flaml.automl: 11-23 01:37:37] {2223} INFO - iteration 44, current learner extra_tree\n",
"[flaml.automl: 11-23 01:37:38] {2417} INFO - at 5.7s,\testimator extra_tree's best error=0.6500,\tbest estimator lgbm's best error=0.6276\n",
"[flaml.automl: 11-23 01:37:38] {2223} INFO - iteration 45, current learner rf\n",
"[flaml.automl: 11-23 01:37:38] {2417} INFO - at 5.8s,\testimator rf's best error=0.6469,\tbest estimator lgbm's best error=0.6276\n",
"[flaml.automl: 11-23 01:37:38] {2223} INFO - iteration 46, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:38] {2417} INFO - at 6.1s,\testimator lgbm's best error=0.6276,\tbest estimator lgbm's best error=0.6276\n",
"[flaml.automl: 11-23 01:37:38] {2223} INFO - iteration 47, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:38] {2417} INFO - at 6.3s,\testimator lgbm's best error=0.6276,\tbest estimator lgbm's best error=0.6276\n",
"[flaml.automl: 11-23 01:37:38] {2223} INFO - iteration 48, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:38] {2417} INFO - at 6.6s,\testimator lgbm's best error=0.6259,\tbest estimator lgbm's best error=0.6259\n",
"[flaml.automl: 11-23 01:37:38] {2223} INFO - iteration 49, current learner catboost\n",
"[flaml.automl: 11-23 01:37:39] {2417} INFO - at 6.8s,\testimator catboost's best error=0.6828,\tbest estimator lgbm's best error=0.6259\n",
"[flaml.automl: 11-23 01:37:39] {2223} INFO - iteration 50, current learner catboost\n",
"[flaml.automl: 11-23 01:37:39] {2417} INFO - at 6.9s,\testimator catboost's best error=0.6828,\tbest estimator lgbm's best error=0.6259\n",
"[flaml.automl: 11-23 01:37:39] {2223} INFO - iteration 51, current learner catboost\n",
"[flaml.automl: 11-23 01:37:39] {2417} INFO - at 7.1s,\testimator catboost's best error=0.6738,\tbest estimator lgbm's best error=0.6259\n",
"[flaml.automl: 11-23 01:37:39] {2223} INFO - iteration 52, current learner lgbm\n",
"[flaml.automl: 11-23 01:37:41] {2417} INFO - at 9.1s,\testimator lgbm's best error=0.6222,\tbest estimator lgbm's best error=0.6222\n",
"[flaml.automl: 11-23 01:37:41] {2223} INFO - iteration 53, current learner xgb_limitdepth\n",
"[flaml.automl: 11-23 01:37:42] {2417} INFO - at 10.1s,\testimator xgb_limitdepth's best error=0.6682,\tbest estimator lgbm's best error=0.6222\n",
"[flaml.automl: 11-23 01:37:43] {2629} INFO - retrain lgbm for 1.1s\n",
"[flaml.automl: 11-23 01:37:43] {2634} INFO - retrained model: LGBMClassifier(colsample_bytree=0.9031374907114736,\n",
" learning_rate=0.24484122722842153, max_bin=1023,\n",
" min_child_samples=4, n_estimators=26, num_leaves=18,\n",
" reg_alpha=0.011566789591709752, reg_lambda=73.308812838521,\n",
" verbose=-1)\n",
"[flaml.automl: 11-23 01:37:43] {2014} INFO - fit succeeded\n",
"[flaml.automl: 11-23 01:37:43] {2016} INFO - Time taken to find the best model: 9.087406158447266\n",
"[flaml.automl: 11-23 01:37:43] {2030} WARNING - Time taken to find the best model is 91% of the provided time budget and not all estimators' hyperparameter search converged. Consider increasing the time budget.\n"
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
}
],
Forecast (#162) * added 'forecast' task with estimators ['fbprophet', 'arima', 'sarimax'] * update setup.py * add TimeSeriesSplit to 'regression' and 'classification' task * add 'time' split_type for 'classification' and 'regression' task Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com> * feature importance * variable name * Update test/test_split.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update test/test_forecast.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * prophet installation fail in windows * upload flaml_forecast.ipynb Signed-off-by: Kevin Chen <chenkevin.8787@gmail.com>
2021-08-23 16:26:46 -04:00
"source": [
remove catboost training dir; ensemble api; blendsearch for hierarchical space; ranking task; forecast improvement (#178) * remove catboost training dir * close #48 * bs for hierarchical space. close #85 * retrain for hierarchical space * clean ml (#180) Co-authored-by: Qingyun Wu <qxw5138@psu.edu> * support ranking task * examples * cv shuffle * forecast api and implementation cleaner * period constraints * delete groups after fit
2021-09-01 16:25:04 -07:00
"automl = AutoML()\n",
"settings = {\n",
" \"time_budget\": 10, # total running time in seconds\n",
" \"metric\": custom_metric, # pass the custom metric funtion here\n",
" \"task\": 'classification', # task type\n",
" \"log_file_name\": 'airlines_experiment_custom_metric.log', # flaml log file\n",
"}\n",
"\n",
"automl.fit(X_train = X_train, y_train = y_train, **settings)"
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
]
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
}
],
"metadata": {
coverage (#135) * coverage * readme * timeout
2021-07-20 17:00:44 -07:00
"interpreter": {
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"hash": "e9d36fc5b7c3dd4177ff1b60184dd696c0acc18150a44682abca4d769811bd46"
coverage (#135) * coverage * readme * timeout
2021-07-20 17:00:44 -07:00
},
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
"kernelspec": {
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
"display_name": "Python 3.8.0 64-bit ('blend': conda)",
"name": "python3"
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
add max_depth to xgboost search space (#282) * add max_depth to xgboost search space * notebook update * two learners for xgboost (max_depth or max_leaves)
2021-11-22 21:17:48 -08:00
"version": "3.7.12"
v0.2.5 (#30) * test distillbert * import check * complete partial config * None check * init config is not suggested by bo * badge * notebook for lightgbm
2021-02-22 22:10:41 -08:00
}
},
"nbformat": 4,
"nbformat_minor": 2
limit time and memory consumption (#264) * limit time and memory * separate tests * lrl1 can't be limited by limit_resource * free memory when possible * passthrough=False when ensemble fails; retrain when trained_estimator is None * use callback to for resource limit * handle lower version of xgb with no callback * free mem ratio * reduce verbosity * retrain_final when max_iter==1 * remove trained_estimator from result * model_history * wheel * retrain time as best_config_train_time * ci: libomp version for xgboost on macos * limit_resource not working in windows * test pickle load * mute forecaster * notebook update * check hard * preventive callback * add use_ray
2021-11-03 19:08:23 -07:00
}
Reference in New Issue Copy Permalink