notebook/10-03-cntext.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# cntext\n",
    "\n",
    "cntext 是一个文本分析包，提供基于词嵌入模型的语义距离和语义投影。 此外，cntext还提供了传统的方法，如字数统计、可读性、文档相似度、情感分析等。\n",
    "\n",
    "- github地址 https://github.com/hiDaDeng/cntext/\n",
    "- pypi地址 https://pypi.org/project/cntext/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-04T04:40:52.208419Z",
     "start_time": "2024-08-04T04:40:46.180045Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Requirement already satisfied: cntext in /opt/anaconda3/lib/python3.8/site-packages (1.7.9)\n",
      "Requirement already satisfied: nltk in /opt/anaconda3/lib/python3.8/site-packages (from cntext) (3.6.1)\n",
      "Requirement already satisfied: numpy==1.20.0 in /opt/anaconda3/lib/python3.8/site-packages (from cntext) (1.20.0)\n",
      "Requirement already satisfied: mittens in /opt/anaconda3/lib/python3.8/site-packages (from cntext) (0.2)\n",
      "Requirement already satisfied: jieba in /opt/anaconda3/lib/python3.8/site-packages (from cntext) (0.42.1)\n",
      "Requirement already satisfied: pyecharts in /opt/anaconda3/lib/python3.8/site-packages (from cntext) (1.9.1)\n",
      "Collecting gensim==4.0.0\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/81/09/6929fd1e882943d1764f2aaf1e66ed32fc1cef987dab6ddbec0291e3ae4a/gensim-4.0.0-cp38-cp38-macosx_10_9_x86_64.whl (23.9 MB)\n",
      "Requirement already satisfied: matplotlib in /opt/anaconda3/lib/python3.8/site-packages (from cntext) (3.3.4)\n",
      "Requirement already satisfied: scikit-learn==1.0 in /opt/anaconda3/lib/python3.8/site-packages (from cntext) (1.0)\n",
      "Requirement already satisfied: smart-open>=1.8.1 in /opt/anaconda3/lib/python3.8/site-packages (from gensim==4.0.0->cntext) (5.2.1)\n",
      "Requirement already satisfied: scipy>=0.18.1 in /opt/anaconda3/lib/python3.8/site-packages (from gensim==4.0.0->cntext) (1.6.2)\n",
      "Requirement already satisfied: joblib>=0.11 in /opt/anaconda3/lib/python3.8/site-packages (from scikit-learn==1.0->cntext) (1.0.1)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/anaconda3/lib/python3.8/site-packages (from scikit-learn==1.0->cntext) (2.1.0)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /opt/anaconda3/lib/python3.8/site-packages (from matplotlib->cntext) (2.4.7)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /opt/anaconda3/lib/python3.8/site-packages (from matplotlib->cntext) (1.3.1)\n",
      "Requirement already satisfied: cycler>=0.10 in /opt/anaconda3/lib/python3.8/site-packages (from matplotlib->cntext) (0.10.0)\n",
      "Requirement already satisfied: pillow>=6.2.0 in /opt/anaconda3/lib/python3.8/site-packages (from matplotlib->cntext) (8.2.0)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /opt/anaconda3/lib/python3.8/site-packages (from matplotlib->cntext) (2.8.1)\n",
      "Requirement already satisfied: tqdm in /opt/anaconda3/lib/python3.8/site-packages (from nltk->cntext) (4.59.0)\n",
      "Requirement already satisfied: regex in /opt/anaconda3/lib/python3.8/site-packages (from nltk->cntext) (2021.4.4)\n",
      "Requirement already satisfied: click in /opt/anaconda3/lib/python3.8/site-packages (from nltk->cntext) (7.1.2)\n",
      "Requirement already satisfied: prettytable in /opt/anaconda3/lib/python3.8/site-packages (from pyecharts->cntext) (3.3.0)\n",
      "Requirement already satisfied: simplejson in /opt/anaconda3/lib/python3.8/site-packages (from pyecharts->cntext) (3.17.6)\n",
      "Requirement already satisfied: jinja2 in /opt/anaconda3/lib/python3.8/site-packages (from pyecharts->cntext) (2.11.3)\n",
      "Requirement already satisfied: six in /opt/anaconda3/lib/python3.8/site-packages (from cycler>=0.10->matplotlib->cntext) (1.15.0)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /opt/anaconda3/lib/python3.8/site-packages (from jinja2->pyecharts->cntext) (1.1.1)\n",
      "Requirement already satisfied: wcwidth in /opt/anaconda3/lib/python3.8/site-packages (from prettytable->pyecharts->cntext) (0.2.5)\n",
      "Installing collected packages: gensim\n",
      "  Attempting uninstall: gensim\n",
      "    Found existing installation: gensim 3.8.3\n",
      "    Uninstalling gensim-3.8.3:\n",
      "      Successfully uninstalled gensim-3.8.3\n",
      "Successfully installed gensim-4.0.0\n",
      "\u001b[33mWARNING: You are using pip version 22.0.4; however, version 24.2 is available.\n",
      "You should consider upgrading via the '/opt/anaconda3/bin/python -m pip install --upgrade pip' command.\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    " pip install -i https://pypi.tuna.tsinghua.edu.cn/simple cntext"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-04T04:42:04.753866Z",
     "start_time": "2024-08-04T04:41:56.317408Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on package cntext:\n",
      "\n",
      "NAME\n",
      "    cntext\n",
      "\n",
      "PACKAGE CONTENTS\n",
      "    dictionary\n",
      "    mind\n",
      "    similarity\n",
      "    stats\n",
      "\n",
      "VERSION\n",
      "    1.7.9\n",
      "\n",
      "FILE\n",
      "    /opt/anaconda3/lib/python3.8/site-packages/cntext/__init__.py\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import cntext as ct\n",
    "\n",
    "help(ct)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-04T04:42:32.087353Z",
     "start_time": "2024-08-04T04:42:31.270546Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building prefix dict from the default dictionary ...\n",
      "Dumping model to file cache /var/folders/l6/ntr5b4610hx38gy0_2xp7ngh0000gn/T/jieba.cache\n",
      "Loading model cost 0.802 seconds.\n",
      "Prefix dict has been built successfully.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Counter({'看待': 1,\n",
       "         '网文': 1,\n",
       "         '作者': 1,\n",
       "         '黑客': 1,\n",
       "         '大佬': 1,\n",
       "         '盗号': 1,\n",
       "         '改文因': 1,\n",
       "         '万分': 1,\n",
       "         '惭愧': 1,\n",
       "         '停': 1})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = '如何看待一网文作者被黑客大佬盗号改文，因万分惭愧而停更。'\n",
    "\n",
    "ct.term_freq(text, lang='chinese')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## readability\n",
    "文本可读性，指标越大，文章复杂度越高，可读性越差。\n",
    "\n",
    "> readability(text, lang='chinese')\n",
    "\n",
    "徐巍,姚振晔,陈冬华.中文年报可读性：衡量与检验[J].会计研究,2021(03):28-44.\n",
    "\n",
    "- readability1 ---每个分句中的平均字数\n",
    "- readability2 ---每个句子中副词和连词所占的比例\n",
    "- readability3 ---参考Fog Index， readability3=(readability1+readability2)×0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-04T04:43:01.222065Z",
     "start_time": "2024-08-04T04:43:01.215733Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'readability1': 28.0,\n",
       " 'readability2': 0.15789473684210525,\n",
       " 'readability3': 14.078947368421053}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text1 = '如何看待一网文作者被黑客大佬盗号改文，因万分惭愧而停更。'\n",
    "\n",
    "ct.readability(text1, lang='chinese')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-04T04:46:15.124091Z",
     "start_time": "2024-08-04T04:46:15.119890Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['DUTIR.pkl',\n",
       " 'HOWNET.pkl',\n",
       " 'Chinese_Loughran_McDonald_Financial_Sentiment.pkl',\n",
       " 'SentiWS.pkl',\n",
       " 'ChineseFinancialFormalUnformalSentiment.pkl',\n",
       " 'ANEW.pkl',\n",
       " 'LSD2015.pkl',\n",
       " 'NRC.pkl',\n",
       " 'geninqposneg.pkl',\n",
       " 'HuLiu.pkl',\n",
       " 'Loughran_McDonald_Financial_Sentiment.pkl',\n",
       " 'AFINN.pkl',\n",
       " 'ADV_CONJ.pkl',\n",
       " 'STOPWORDS.pkl',\n",
       " 'Concreteness.pkl',\n",
       " 'ChineseEmoBank.pkl']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import cntext as ct\n",
    "\n",
    "# 获取cntext内置词典列表(pkl格式)\n",
    "ct.dict_pkl_list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-04T04:48:41.259954Z",
     "start_time": "2024-08-04T04:48:41.249663Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.7.9\n",
      "dict_keys(['DUTIR', 'Referer', 'Desc'])\n"
     ]
    }
   ],
   "source": [
    "import cntext as ct\n",
    "\n",
    "print(ct.__version__)\n",
    "# 导入pkl词典文件,\n",
    "dutir = ct.load_pkl_dict('DUTIR.pkl')\n",
    "print(dutir.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## sentiment\n",
    "\n",
    "> sentiment(text, diction, lang='chinese') \n",
    "\n",
    "使用diy词典进行情感分析，计算各个情绪词出现次数; 未考虑强度副词、否定词对情感的复杂影响，\n",
    "\n",
    "- text: 待分析中文文本\n",
    "- diction: 情感词字典；\n",
    "- lang: 语言类型，\"chinese\"或\"english\"，默认\"chinese\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-04T04:49:32.782452Z",
     "start_time": "2024-08-04T04:49:32.287078Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'乐_num': 2,\n",
       " '好_num': 0,\n",
       " '怒_num': 0,\n",
       " '哀_num': 0,\n",
       " '惧_num': 0,\n",
       " '恶_num': 0,\n",
       " '惊_num': 0,\n",
       " 'stopword_num': 8,\n",
       " 'word_num': 14,\n",
       " 'sentence_num': 1}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import cntext as ct\n",
    "\n",
    "text = '我今天得奖了，很高兴，我要将快乐分享大家。'\n",
    "\n",
    "ct.sentiment(text=text,\n",
    "             diction=ct.load_pkl_dict('DUTIR.pkl')['DUTIR'],\n",
    "             lang='chinese')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-04T04:50:25.934568Z",
     "start_time": "2024-08-04T04:50:25.909495Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Referer': 'Brysbaert, M., Warriner, A. B., & Kuperman, V. (2014). Concreteness ratings for 40 thousand generally known English word lemmas. Behavior Research Methods, 46, 904–911',\n",
       " 'Desc': '语言具体性词典， 具体性计算应用案例可参考Packard, Grant, and Jonah Berger. \"How concrete language shapes customer satisfaction.\" *Journal of Consumer Research* 47, no. 5 (2021): 787-806.',\n",
       " 'Concreteness':                   word  valence\n",
       " 0          roadsweeper     4.85\n",
       " 1          traindriver     4.54\n",
       " 2                 tush     4.45\n",
       " 3            hairdress     3.93\n",
       " 4        pharmaceutics     3.77\n",
       " ...                ...      ...\n",
       " 39949         unenvied     1.21\n",
       " 39950     agnostically     1.20\n",
       " 39951  conceptualistic     1.18\n",
       " 39952  conventionalism     1.18\n",
       " 39953    essentialness     1.04\n",
       " \n",
       " [39954 rows x 2 columns]}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import cntext as ct\n",
    "\n",
    "# load the concreteness.pkl dictionary file\n",
    "concreteness_df = ct.load_pkl_dict('concreteness.pkl')\n",
    "concreteness_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-04T04:53:34.420645Z",
     "start_time": "2024-08-04T04:53:34.404483Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'valence': 9.28, 'word_num': 5}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reply = \"I'll go look for that\"\n",
    "\n",
    "score=ct.sentiment_by_valence(text=reply, \n",
    "                              diction=concreteness_df, \n",
    "                              lang='english')\n",
    "score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-04T04:52:33.083998Z",
     "start_time": "2024-08-04T04:52:33.076815Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'anger_num': 0,\n",
       " 'anticipation_num': 1,\n",
       " 'disgust_num': 0,\n",
       " 'fear_num': 0,\n",
       " 'joy_num': 1,\n",
       " 'negative_num': 0,\n",
       " 'positive_num': 1,\n",
       " 'sadness_num': 0,\n",
       " 'surprise_num': 0,\n",
       " 'trust_num': 1,\n",
       " 'stopword_num': 1,\n",
       " 'word_num': 5,\n",
       " 'sentence_num': 1}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = 'What a happy day!'\n",
    "\n",
    "ct.sentiment(text=text,\n",
    "             diction=ct.load_pkl_dict('NRC.pkl')['NRC'],\n",
    "             lang='english')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-04T04:52:50.963134Z",
     "start_time": "2024-08-04T04:52:50.948664Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>valence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>roadsweeper</td>\n",
       "      <td>4.85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>traindriver</td>\n",
       "      <td>4.54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tush</td>\n",
       "      <td>4.45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>hairdress</td>\n",
       "      <td>3.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>pharmaceutics</td>\n",
       "      <td>3.77</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            word  valence\n",
       "0    roadsweeper     4.85\n",
       "1    traindriver     4.54\n",
       "2           tush     4.45\n",
       "3      hairdress     3.93\n",
       "4  pharmaceutics     3.77"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load the concreteness.pkl dictionary file;  cntext version >=1.7.1\n",
    "concreteness_df = ct.load_pkl_dict('concreteness.pkl')['Concreteness']\n",
    "concreteness_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-04T04:53:50.123421Z",
     "start_time": "2024-08-04T04:53:50.021025Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Concreteness Score: 1.86 | Example-0: I'll go look for that\n",
      "Concreteness Score: 1.86 | Example-1: I'll go search for that\n",
      "Concreteness Score: 2.21 | Example-2: I'll go search for that top\n",
      "Concreteness Score: 2.04 | Example-3: I'll go search for that t-shirt\n",
      "Concreteness Score: 2.37 | Example-4: I'll go look for that t-shirt in grey\n",
      "Concreteness Score: 2.37 | Example-5: I'll go search for that t-shirt in grey\n"
     ]
    }
   ],
   "source": [
    "employee_replys = [\"I'll go look for that\",\n",
    "                   \"I'll go search for that\",\n",
    "                   \"I'll go search for that top\",\n",
    "                   \"I'll go search for that t-shirt\",\n",
    "                   \"I'll go look for that t-shirt in grey\",\n",
    "                   \"I'll go search for that t-shirt in grey\"]\n",
    "\n",
    "for idx, reply in enumerate(employee_replys):\n",
    "    score=ct.sentiment_by_valence(text=reply, \n",
    "                                  diction=concreteness_df, \n",
    "                                  lang='english')\n",
    "    \n",
    "    \n",
    "    template = \"Concreteness Score: {score:.2f} | Example-{idx}: {example}\"\n",
    "    print(template.format(score=score['valence']/score['word_num'], \n",
    "                          idx=idx, \n",
    "                          example=reply))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-04T04:56:34.798503Z",
     "start_time": "2024-08-04T04:55:33.788633Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 1/4:...Preprocess   Corpus ...\n",
      "Step 2/4:...Collect co-occurrency information ...\n",
      "Step 3/4:...Calculate   mutual information ...\n",
      "Step 4/4:...Save    candidate words ...\n",
      "Finish! used 60.98 s\n"
     ]
    }
   ],
   "source": [
    "# This module is used to build or expand the vocabulary (dictionary), including\n",
    "\n",
    "# SoPmi Co-occurrence algorithm to extend vocabulary (dictionary), Only support chinese\n",
    "# W2VModels using word2vec to extend vocabulary (dictionary), support english & chinese\n",
    "\n",
    "import os\n",
    "\n",
    "sopmier = ct.SoPmi(cwd=os.getcwd(),\n",
    "                   #raw corpus data，txt file.only support chinese data now.\n",
    "                   input_txt_file='/Users/chengjun/GitHub/cntext/examples/data/sopmi_corpus.txt', \n",
    "                   #muanually selected seed words\n",
    "                   seedword_txt_file='/Users/chengjun/GitHub/cntext/examples/data/sopmi_seed_words.txt', #人工标注的初始种子词\n",
    "                   )   \n",
    "\n",
    "sopmier.sopmi()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T12:12:14.842526Z",
     "start_time": "2022-07-21T12:12:14.837856Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/chengjun/GitHub/css/notebook'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T12:14:13.096781Z",
     "start_time": "2022-07-21T12:12:50.789549Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 1/4:...Preprocess   corpus ...\n",
      "Step 2/4:...Train  word2vec model\n",
      "            used   72 s\n",
      "Step 3/4:...Prepare similar candidates for each seed word in the word2vec model...\n",
      "Step 4/4 Finish! Used 82 s\n",
      "Step 3/4:...Prepare similar candidates for each seed word in the word2vec model...\n",
      "Step 4/4 Finish! Used 82 s\n",
      "Step 3/4:...Prepare similar candidates for each seed word in the word2vec model...\n",
      "Step 4/4 Finish! Used 82 s\n",
      "Step 3/4:...Prepare similar candidates for each seed word in the word2vec model...\n",
      "Step 4/4 Finish! Used 82 s\n",
      "Step 3/4:...Prepare similar candidates for each seed word in the word2vec model...\n",
      "Step 4/4 Finish! Used 82 s\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "#init W2VModels, corpus data w2v_corpus.txt\n",
    "model = ct.W2VModels(cwd=os.getcwd(), lang='english')  \n",
    "model.train(input_txt_file='/Users/chengjun/GitHub/cntext/examples/data/w2v_corpus.txt')\n",
    "\n",
    "\n",
    "#According to the seed word, filter out the top 100 words that are most similar to each category words\n",
    "model.find(seedword_txt_file='/Users/chengjun/GitHub/cntext/examples/data/w2v_seeds/integrity.txt', \n",
    "           topn=100)\n",
    "model.find(seedword_txt_file='/Users/chengjun/GitHub/cntext/examples/data/w2v_seeds/innovation.txt', \n",
    "           topn=100)\n",
    "model.find(seedword_txt_file='/Users/chengjun/GitHub/cntext/examples/data/w2v_seeds/quality.txt', \n",
    "           topn=100)\n",
    "model.find(seedword_txt_file='/Users/chengjun/GitHub/cntext/examples/data/w2v_seeds/respect.txt', \n",
    "           topn=100)\n",
    "model.find(seedword_txt_file='/Users/chengjun/GitHub/cntext/examples/data/w2v_seeds/teamwork.txt', \n",
    "           topn=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T12:15:53.463147Z",
     "start_time": "2022-07-21T12:15:53.454324Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-1.315214  ,  0.8820462 ,  1.1123291 ,  0.57502085, -0.4536534 ,\n",
       "        0.04593553,  1.7608879 ,  1.8781638 ,  0.62472534, -0.40170938,\n",
       "       -0.74852383,  0.20071138, -0.1615613 , -1.1544902 , -1.8266941 ,\n",
       "        0.07904029,  0.04081295, -0.30807   , -0.25948036,  0.7637631 ,\n",
       "        0.30233267,  0.02158124, -0.4548134 ,  0.22692135, -0.26944858,\n",
       "       -0.24163176,  1.2302433 ,  0.70777947,  1.1255033 , -0.17567617,\n",
       "       -0.7234768 , -1.0653226 , -0.09055816, -1.46483   ,  0.4885035 ,\n",
       "       -0.9827379 ,  0.95400816, -0.13374516, -0.52517796, -0.23480052,\n",
       "        0.62640554, -0.5553755 , -0.83863366,  1.3104644 , -0.07400908,\n",
       "        1.1336509 ,  0.32375774,  0.354964  ,  0.10843342, -0.54200864,\n",
       "       -0.8003504 , -0.7266025 ,  1.4322623 , -0.8662505 , -0.7207146 ,\n",
       "        0.16009226,  1.746251  ,  1.7912021 ,  1.9516977 ,  0.7451547 ,\n",
       "       -0.43844843, -0.30046785,  1.7179738 ,  0.8191196 , -0.41387284,\n",
       "       -0.26602927, -1.3589116 , -0.0668382 , -0.02384867, -0.5849235 ,\n",
       "        0.55971706,  0.96517783,  1.5546738 ,  0.7545263 ,  1.4992219 ,\n",
       "        1.0422503 ,  0.51255804, -0.67980427, -0.8455454 ,  1.3597207 ,\n",
       "       -0.31741807, -1.42675   , -0.04144925, -0.6972174 , -0.11606634,\n",
       "        1.8798316 ,  0.11557695, -1.1349543 ,  1.3116952 ,  0.3919832 ,\n",
       "        0.59277475, -0.45137736,  0.93078315,  0.45508283,  0.8245617 ,\n",
       "        0.02893427,  1.0617537 , -0.78552085,  0.43378317,  2.971067  ],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim.models import KeyedVectors\n",
    "\n",
    "w2v_model = KeyedVectors.load(\"/Users/chengjun/GitHub/cntext/examples/output/w2v_candi_words/w2v.model\")\n",
    "w2v_model.get_vector('company')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T12:16:16.087154Z",
     "start_time": "2022-07-21T12:16:16.079654Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('execution', 0.8257297873497009),\n",
       " ('capabilities', 0.8198432326316833),\n",
       " ('continue_focus', 0.8191850781440735),\n",
       " ('dsd_system', 0.8175386786460876),\n",
       " ('continue_expand', 0.8135412931442261),\n",
       " ('emphasis', 0.8049392700195312),\n",
       " ('informatics', 0.8048107624053955),\n",
       " ('continue_drive', 0.8032825589179993),\n",
       " ('technologies', 0.8026941418647766),\n",
       " ('dsd', 0.8008151650428772)]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v_model.most_similar('innovation')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T12:16:25.722448Z",
     "start_time": "2022-07-21T12:16:25.717226Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.10703183, -0.5493035 ,  1.0623612 , -0.43180484, -0.9257197 ,\n",
       "       -1.0833972 , -0.07150532,  1.8700444 ,  0.26021707,  0.04272432,\n",
       "       -0.93499535,  0.05657623,  1.1683912 ,  0.3376583 ,  0.49347457,\n",
       "       -0.7696766 , -0.91470116, -0.66640836,  1.2935071 , -0.6879933 ,\n",
       "        0.01370158, -0.07028562, -1.3170564 , -0.4464321 , -0.08877528,\n",
       "        0.634105  , -0.9797471 ,  1.221016  , -0.5741402 , -0.18807778,\n",
       "       -1.5563715 ,  0.14269993,  0.53213865, -0.16015887, -0.16328476,\n",
       "       -0.03469868,  0.8123352 , -0.89830345, -0.4654071 ,  0.59473026,\n",
       "       -0.01235237, -0.16233891,  0.0882808 ,  1.2530324 ,  0.63678694,\n",
       "       -0.4859363 ,  0.63639694,  0.5577657 ,  1.1487194 ,  1.2548769 ,\n",
       "        0.08172881,  1.2036918 ,  0.31136826, -0.33539033, -0.73395115,\n",
       "       -0.35624295,  0.00873698, -0.7198372 ,  0.09266879,  0.0259969 ,\n",
       "        0.5248478 ,  1.0185517 , -0.3534738 , -0.11561136, -0.64694124,\n",
       "       -0.13806236, -0.7061978 ,  0.4576997 , -0.9011815 ,  0.47045934,\n",
       "        0.7067756 ,  0.08959749,  0.31015027,  0.284999  ,  1.887349  ,\n",
       "        0.38284793,  0.35984853, -0.413301  , -0.2978032 , -0.13867766,\n",
       "        0.6174871 ,  0.55524594, -0.1517662 ,  0.05696797,  0.06463829,\n",
       "       -0.0501307 , -0.7806092 ,  0.2816824 ,  0.19331798,  0.2766436 ,\n",
       "        1.5176852 ,  0.01909402, -0.2754046 ,  0.41802418,  0.44312304,\n",
       "        1.2473223 ,  1.5042527 , -1.0180748 ,  0.35682076, -0.550147  ],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v_model.get_vector('innovation')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T12:04:27.409744Z",
     "start_time": "2022-07-21T12:04:27.397148Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>.</th>\n",
       "      <th>bus</th>\n",
       "      <th>by</th>\n",
       "      <th>day</th>\n",
       "      <th>every</th>\n",
       "      <th>go</th>\n",
       "      <th>i</th>\n",
       "      <th>night</th>\n",
       "      <th>school</th>\n",
       "      <th>theatre</th>\n",
       "      <th>to</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>.</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bus</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>by</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>every</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>go</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>i</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>night</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>school</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>theatre</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>to</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         .  bus  by  day  every  go  i  night  school  theatre  to\n",
       ".        0    1   1    0      0   0  0      0       0        0   0\n",
       "bus      1    0   2    1      0   0  0      1       0        0   0\n",
       "by       1    2   0    1      2   0  0      1       0        0   0\n",
       "day      0    1   1    0      1   0  0      0       1        0   0\n",
       "every    0    0   2    1      0   0  0      1       1        1   2\n",
       "go       0    0   0    0      0   0  2      0       1        1   2\n",
       "i        0    0   0    0      0   2  0      0       0        0   2\n",
       "night    0    1   1    0      1   0  0      0       0        1   0\n",
       "school   0    0   0    1      1   1  0      0       0        0   1\n",
       "theatre  0    0   0    0      1   1  0      1       0        0   1\n",
       "to       0    0   0    0      2   2  2      0       1        1   0"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documents = [\"I go to school every day by bus .\",\n",
    "         \"i go to theatre every night by bus\"]\n",
    "\n",
    "ct.co_occurrence_matrix(documents, \n",
    "                        window_size=2, \n",
    "                        lang='english')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T12:17:06.028525Z",
     "start_time": "2022-07-21T12:17:06.018573Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.67\n",
      "0.50\n",
      "1.00\n",
      "0.90\n"
     ]
    }
   ],
   "source": [
    "text1 = 'Programming is fun!'\n",
    "text2 = 'Programming is interesting!'\n",
    "\n",
    "print(ct.cosine_sim(text1, text2))\n",
    "print(ct.jaccard_sim(text1, text2))\n",
    "print(ct.minedit_sim(text1, text2))\n",
    "print(ct.simple_sim(text1, text2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T12:29:35.994334Z",
     "start_time": "2022-07-21T12:29:13.289711Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading the model of /Users/chengjun/bigdata/glove_w2v.6B.100d.txt\n",
      "Load successfully, used 22.7 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "-0.43"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# download glove_w2v.6B.100d.txt from google Driver\n",
    "# https://drive.google.com/file/d/1tuQB9PDx42z67ScEQrg650aDTYPz-elJ/view\n",
    "#Note: this is a word2vec format model\n",
    "tm = ct.Text2Mind(w2v_model_path='/Users/chengjun/bigdata/glove_w2v.6B.100d.txt')\n",
    "#tm = ct.Text2Mind(w2v_model_path='/Users/chengjun/GitHub/cntext/examples/output/Glove/brown_corpus_w2v.txt')\n",
    "\n",
    "\n",
    "engineer = ['program', 'software', 'computer']\n",
    "mans =  [\"man\", \"he\", \"him\"]\n",
    "womans = [\"woman\", \"she\", \"her\"]\n",
    "\n",
    "\n",
    "tm.sematic_distance(words=animals, \n",
    "                    c_words1=mans, \n",
    "                    c_words2=womans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T12:29:55.558880Z",
     "start_time": "2022-07-21T12:29:55.553608Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('mouse', -1.68),\n",
       " ('cat', -0.92),\n",
       " ('pig', -0.46),\n",
       " ('whale', -0.24),\n",
       " ('horse', 0.4)]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "animals = ['mouse', 'cat', 'horse',  'pig', 'whale']\n",
    "small_words = [\"small\", \"little\", \"tiny\"]\n",
    "large_words = [\"large\", \"big\", \"huge\"]\n",
    "\n",
    "tm.sematic_projection(words=animals, \n",
    "                      c_words1=small_words, \n",
    "                      c_words2=large_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T11:52:55.792131Z",
     "start_time": "2022-07-21T11:52:55.762508Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Referer-1': 'Lee, Lung-Hao, Jian-Hong Li, and Liang-Chih Yu. \"Chinese EmoBank: Building Valence-Arousal Resources for Dimensional Sentiment Analysis.\" Transactions on Asian and Low-Resource Language Information Processing 21, no. 4 (2022): 1-18.',\n",
       " 'Referer-2': 'Liang-Chih Yu, Lung-Hao Lee, Shuai Hao, Jin Wang, Yunchao He, Jun Hu, K. Robert Lai, and Xuejie Zhang. 2016. \"Building Chinese affective resources in valence-arousal dimensions. In Proceedings of NAACL/HLT-16, pages 540-545.',\n",
       " 'Desc': 'Chinese Sentiment Dictionary, includes 「valence」「arousal」. In cntext, we only take Chinese valence-arousal words (CVAW, single word) into account, ignore CVAP, CVAS, CVAT.',\n",
       " 'ChineseEmoBank':       word  valence  arousal\n",
       " 0     不可思议      5.4      7.2\n",
       " 1       不平      3.6      5.8\n",
       " 2       不甘      3.2      6.4\n",
       " 3       不安      3.8      5.4\n",
       " 4       不利      3.6      5.6\n",
       " ...    ...      ...      ...\n",
       " 5505    黏闷      2.8      5.6\n",
       " 5506    黏腻      2.7      5.8\n",
       " 5507    艳丽      5.8      4.5\n",
       " 5508    苗条      6.7      3.8\n",
       " 5509    修长      7.0      4.5\n",
       " \n",
       " [5510 rows x 3 columns]}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct.load_pkl_dict('ChineseEmoBank.pkl')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T11:53:06.871093Z",
     "start_time": "2022-07-21T11:53:06.857225Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>valence</th>\n",
       "      <th>arousal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>不可思议</td>\n",
       "      <td>5.4</td>\n",
       "      <td>7.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>不平</td>\n",
       "      <td>3.6</td>\n",
       "      <td>5.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>不甘</td>\n",
       "      <td>3.2</td>\n",
       "      <td>6.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>不安</td>\n",
       "      <td>3.8</td>\n",
       "      <td>5.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>不利</td>\n",
       "      <td>3.6</td>\n",
       "      <td>5.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5505</th>\n",
       "      <td>黏闷</td>\n",
       "      <td>2.8</td>\n",
       "      <td>5.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5506</th>\n",
       "      <td>黏腻</td>\n",
       "      <td>2.7</td>\n",
       "      <td>5.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5507</th>\n",
       "      <td>艳丽</td>\n",
       "      <td>5.8</td>\n",
       "      <td>4.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5508</th>\n",
       "      <td>苗条</td>\n",
       "      <td>6.7</td>\n",
       "      <td>3.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5509</th>\n",
       "      <td>修长</td>\n",
       "      <td>7.0</td>\n",
       "      <td>4.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5510 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      word  valence  arousal\n",
       "0     不可思议      5.4      7.2\n",
       "1       不平      3.6      5.8\n",
       "2       不甘      3.2      6.4\n",
       "3       不安      3.8      5.4\n",
       "4       不利      3.6      5.6\n",
       "...    ...      ...      ...\n",
       "5505    黏闷      2.8      5.6\n",
       "5506    黏腻      2.7      5.8\n",
       "5507    艳丽      5.8      4.5\n",
       "5508    苗条      6.7      3.8\n",
       "5509    修长      7.0      4.5\n",
       "\n",
       "[5510 rows x 3 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diction_df = ct.load_pkl_dict('ChineseEmoBank.pkl')['ChineseEmoBank']\n",
    "diction_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T11:53:10.588940Z",
     "start_time": "2022-07-21T11:53:09.881112Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building prefix dict from the default dictionary ...\n",
      "Loading model from cache /var/folders/l6/ntr5b4610hx38gy0_2xp7ngh0000gn/T/jieba.cache\n",
      "Loading model cost 0.677 seconds.\n",
      "Prefix dict has been built successfully.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'valence': 14.8, 'arousal': 24.8, 'word_num': 13}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = '很多车主抱怨新车怠速抖动严重---冷车时更严重。'\n",
    "\n",
    "ct.sentiment_by_weight(text = text, \n",
    "                       diction = diction_df,\n",
    "                       params = ['valence', 'arousal'],\n",
    "                       lang = 'chinese')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T11:54:27.162717Z",
     "start_time": "2022-07-21T11:54:27.158700Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- valence是句子中各个chinese_emobank词valence得分的加总。\n",
    "- arousal是句子中各个chinese_emobank词arousal得分的加总。\n",
    "- word_num是句子中的词语数(含标点符号)，短文本的情况下，word_num会不太准确，长文本情况下无限接近真实词语数。\n",
    "\n",
    "需要注意，文本越长，valence和arousal指标应该会越大。使用这两个指标时，需要结合word_num进行均值处理，即\n",
    "\n",
    "- Valence = valence/word_num\n",
    "- Arousal = arousal/word_num\n",
    "这里未做均值处理，尽量保留文本的原始信息。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![image.png](img/chengjun2.png)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "幻灯片",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}