alg/symbolic_metamodeling/3-_Building_Symbolic_Metamodels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building symbolic metamodels\n",
    "\n",
    "A symbolic metamodel takes as an input a machine learning model, and outputs a symbolic equation describing its response surface as illustrated in the Figure below. This notebook provides the steps needed for building a symbolic metamodel for an XGBoost model fitted to the \"UCI absenteeism at work dataset: https://archive.ics.uci.edu/ml/datasets/Absenteeism+at+work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/FigB1.png\", width=640>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first import the necessary modules for symbolic metamodeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pysymbolic.algorithms.symbolic_metamodeling import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we import the necessary libraries fordata processing, splitting the data into training and testing samples, and XGBoost model fitting "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from xgboost import XGBClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "data          = pd.read_csv(\"data/absenteeism.csv\", delimiter=';')\n",
    "\n",
    "feature_names = ['Transportation expense', 'Distance from Residence to Work',\n",
    "                 'Service time', 'Age', 'Work load Average/day ', 'Hit target',\n",
    "                 'Disciplinary failure', 'Education', 'Son', 'Social drinker',\n",
    "                 'Social smoker', 'Pet', 'Weight', 'Height', 'Body mass index']\n",
    "\n",
    "scaler        = MinMaxScaler(feature_range=(0, 1))\n",
    "X             = scaler.fit_transform(data[feature_names])\n",
    "Y             = ((data['Absenteeism time in hours'] > 4) * 1) \n",
    "\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.33, random_state=42)\n",
    "\n",
    "model         = XGBClassifier()\n",
    "\n",
    "model.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us examine the AUC-ROC performance of the fitted XGBoost model on the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "roc_auc_score(Y_test, model.predict_proba(X_test)[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How much better does XGBoost perform compared to a standard logistic regression?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_L = LogisticRegression()\n",
    "\n",
    "model_L.fit(X_train, Y_train)\n",
    "\n",
    "roc_auc_score(Y_test, model_L.predict_proba(X_test)[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us create a symbolic metamodel instance. This takes the fitted model **model** and the training features **X_train** as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metamodel = symbolic_metamodel(model, X_train)\n",
    "\n",
    "metamodel.fit(num_iter=10, batch_size=X_train.shape[0], learning_rate=.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us see how this metamodel performs using the **evaluate** method..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_metamodel = metamodel.evaluate(X_test)\n",
    "\n",
    "roc_auc_score(Y_test, Y_metamodel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It performs close to the original XGBoost model! Now let us see the exact symbolic equation of the model in terms of Meijer-G functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By invoking the **.fit()** method in the **symbolic_metamodel** class, we essentially transform the XGBoost model to a space of interpretable symbolic equations as shown in the Figure below, without much loss in predictive accuracy. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/FigB2.png\", width=320>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we show how to extract the exact and approximate equation $g(x)$ from the metamodel class..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metamodel.exact_expression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Because this equation involves Hypergeometric functions, we might prefer to work with the polynomial approximation..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metamodel.approx_expression"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}