Text Summarisation using Natural Language Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Text Summarisation using Natural Language Processing\n",
    "\n",
    "In practice, there are two main approaches to automatic text summarisation: extraction and abstraction.\n",
    "\n",
    "Abstraction is the notion of generating a summary based on the text, with words in the summary that are not necessarily present in the text itself. This method however, can be too complex at times. Extraction therefore is more commonly used, and works by selecting certain words or sentences from the text, and creating summaries using them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk import FreqDist\n",
    "from nltk.tokenize import word_tokenize, sent_tokenize\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "lemma = WordNetLemmatizer()\n",
    "from nltk.corpus import stopwords\n",
    "stop = stopwords.words('english')\n",
    "from bs4 import BeautifulSoup\n",
    "from urllib.request import urlopen\n",
    "from gensim.models import Phrases\n",
    "from gensim.models.phrases import Phraser\n",
    "import os\n",
    "from collections import Counter\n",
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "punctuations = list(string.punctuation)\n",
    "#Add some more punctuation, as the list doesn't cover all cases.\n",
    "punctuations.extend(['”', '–', '``', \"''\"])\n",
    "stop = stop + punctuations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The core idea behind unsupervised text summarisation is the following:\n",
    "- split text into sentences;\n",
    "- tokenise these sentences into separate words;\n",
    "- assign scores to sentences based on their importance;\n",
    "- select several top sentences and display them in original order;\n",
    "\n",
    "The key idea is assigning scores to sentences. Here are some of the ways to do this:\n",
    "- calculate the similarity between each pair of sentences and select sentences which are most similar to most sentences;\n",
    "- calculate word frequences, select most frequent words and select sentences which have most of these words;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook I'll use the following old news article from 2017:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Snap election to be held in March after Northern Ireland government collapses \n",
      " ____________________________________________________________ \n",
      " An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced.\n",
      "\n",
      "Northern Ireland Secretary James Brokenshire said the devolved Northern Ireland Assembly will sit for the last time on 25 January, before it is dissolved the following day.\n",
      "\n",
      "The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.\n",
      "\n",
      "The \"cash for ash\" scandal prompted the resignation of deputy first minister Martin McGuinness, who called for DUP first minister Arlene Foster to quit.\n",
      "\n",
      "She refused, calling Mr McGuinness' actions \"not principled\" and \"purely political\".\n",
      "\n",
      "On Monday afternoon, Sinn Fein announced it would not replace Mr McGuinness - triggering the snap election.\n",
      "\n",
      "Despite a last-ditch attempt by Theresa May to urge a resolution, Sinn Fein MLA Conor Murphy said his party had decided to \"call time on the arrogance of the DUP\".\n",
      "\n",
      "\n",
      "                  RAF shot down 'a number of drones' in Iran's attack on Israel\n",
      "                \n",
      "\n",
      "\n",
      "                  Politics latest: RAF 'flew into the face of danger to protect civilians', says PM as pilots 'shot down' Iranian drones\n",
      "                \n",
      "\n",
      "\n",
      "                  Is Great Britain still that 'Great'? | Adam Boulton \n",
      "                \n",
      "\n",
      "He said: \"We have had scandal after scandal, allegations of corruption need to be investigated properly and the people responsible need to be held to account.\"\n",
      "\n",
      "Mrs Foster, who presided over the controversial renewable energy scheme as enterprise minister, claimed Sinn Fein \"did not like the election result last May and are therefore looking to have another go\".\n",
      "\n",
      ":: What does the Northern Ireland crisis mean for Brexit?\n",
      "\n",
      "Announcing the dissolution of the Northern Ireland Assembly, Mr Brokenshire urged both parties \"to conduct this election with a view to...re-establishing a partnership government at the earliest opportunity after that poll.\"\n",
      "\n",
      "He said: \"This is essential for the operation of devolved government. And this means that all must remain open to dialogue.\"\n",
      "\n",
      "Please use Chrome browser for a more accessible video player\n",
      "\n",
      "\n",
      "\n",
      "Sinn Fein and the DUP are expected to remain the two largest parties following the election, meaning they will still have to hammer out a power-sharing arrangement.\n",
      "\n",
      "If they fail to agree terms after three weeks, Mrs May could be forced to suspend devolution and reinstate direct rule from Westminster.\n",
      "\n",
      "Please use Chrome browser for a more accessible video player\n",
      "\n",
      "\n",
      "\n",
      "Sky News Ireland Correspondent David Blevins said the relationship between Sinn Fein and the DUP had been \"slowly breaking down for a period of months\".\n",
      "\n",
      "He said: \"Some would suggest that the British and Irish governments took their eye off the ball.\n",
      "\n",
      "\"The botched renewable energy scheme is being blamed for the collapse of the devolved government but it was just the tip of the iceberg.\"\n",
      "\n",
      "He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\".\n",
      "\n",
      "\n",
      " © 2024 Sky UK\n",
      "\n"
     ]
    }
   ],
   "source": [
    "url = urlopen('http://news.sky.com/story/snap-election-to-be-held-in-march-after-northern-ireland-government-collapses-10731488')\n",
    "soup = BeautifulSoup(url.read().decode('utf8'), \"lxml\")\n",
    "text = '\\n\\n'.join(map(lambda p: p.text, soup.find_all('p')))\n",
    "\n",
    "text = text[text.find('An early election'):]\n",
    "title = soup.find('h1').text.strip()\n",
    "print(title, '\\n', '_' * 60, '\\n', text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculating the similarity between sentences\n",
    "\n",
    "In general, this is how it works:\n",
    "\n",
    "- the text will be split into sentences;\n",
    "- the split sentences into words/tokens (there are several ways to do it, which give various results);\n",
    "- calculate similarity between sentences - while there are many ways to do it, I'll use a simple one: comparing tokens in each sentence. Similarity between sentences is calculated as number of words which are present in both sentences divided by average length of sentences (for normalization);\n",
    "- assign scores to sentences based on their similarity with other sentences - for each sentence get a sum of similarity scores with each other sentence;\n",
    "- select several best sentences and show them in order, in which they appeare in the article;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At first, we simply split sentences into words, using space as a separator. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def intersection(sent1, sent2):\n",
    "    s1 = sent1.split(' ')\n",
    "    s2 = sent2.split(' ')\n",
    "\n",
    "    intersection = [i for i in s1 if i in s2]\n",
    "    #Normalization\n",
    "    return len(intersection) / ((len(s1) + len(s2)) / 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we form a matrix of similarities between each pair of sentences. Spatially, this is a 2D-matrix with a length equal to the number of sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1.0,\n",
       "  0.40816326530612246,\n",
       "  0.1568627450980392,\n",
       "  0.08695652173913043,\n",
       "  0.0,\n",
       "  0.10256410256410256,\n",
       "  0.15384615384615385,\n",
       "  0.07936507936507936,\n",
       "  0.1791044776119403,\n",
       "  0.1111111111111111,\n",
       "  0.1875,\n",
       "  0.3018867924528302,\n",
       "  0.12121212121212122,\n",
       "  0.0,\n",
       "  0.13793103448275862,\n",
       "  0.08888888888888889,\n",
       "  0.17857142857142858,\n",
       "  0.10256410256410256,\n",
       "  0.34782608695652173,\n",
       "  0.4,\n",
       "  0.0],\n",
       " [0.24489795918367346,\n",
       "  1.0,\n",
       "  0.10714285714285714,\n",
       "  0.11764705882352941,\n",
       "  0.0,\n",
       "  0.09090909090909091,\n",
       "  0.17543859649122806,\n",
       "  0.030534351145038167,\n",
       "  0.027777777777777776,\n",
       "  0.1016949152542373,\n",
       "  0.21621621621621623,\n",
       "  0.20689655172413793,\n",
       "  0.21052631578947367,\n",
       "  0.0,\n",
       "  0.19047619047619047,\n",
       "  0.0,\n",
       "  0.19672131147540983,\n",
       "  0.09090909090909091,\n",
       "  0.3137254901960784,\n",
       "  0.24,\n",
       "  0.0]]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = sent_tokenize(text)\n",
    "matrix = [[intersection(sentences[i], sentences[j]) for i in range(0,len(sentences))] for j in range(0,len(sentences))]\n",
    "matrix[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We calculate the score for each sentence, which is a sum of similarity scores with other sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced.': 4.144353911770331,\n",
       " 'Northern Ireland Secretary James Brokenshire said the devolved Northern Ireland Assembly will sit for the last time on 25 January, before it is dissolved the following day.': 3.5615137735140294,\n",
       " 'The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.': 4.356835581188402,\n",
       " 'The \"cash for ash\" scandal prompted the resignation of deputy first minister Martin McGuinness, who called for DUP first minister Arlene Foster to quit.': 3.7467069848694576,\n",
       " 'She refused, calling Mr McGuinness\\' actions \"not principled\" and \"purely political\".': 1.4685232694430241,\n",
       " 'On Monday afternoon, Sinn Fein announced it would not replace Mr McGuinness - triggering the snap election.': 3.285412517463482,\n",
       " 'Despite a last-ditch attempt by Theresa May to urge a resolution, Sinn Fein MLA Conor Murphy said his party had decided to \"call time on the arrogance of the DUP\".': 3.8666869354760585,\n",
       " \"RAF shot down 'a number of drones' in Iran's attack on Israel\\n                \\n\\n\\n                  Politics latest: RAF 'flew into the face of danger to protect civilians', says PM as pilots 'shot down' Iranian drones\\n                \\n\\n\\n                  Is Great Britain still that 'Great'?\": 2.3017234906237785,\n",
       " '| Adam Boulton \\n                \\n\\nHe said: \"We have had scandal after scandal, allegations of corruption need to be investigated properly and the people responsible need to be held to account.\"': 4.0186309889770655,\n",
       " 'Mrs Foster, who presided over the controversial renewable energy scheme as enterprise minister, claimed Sinn Fein \"did not like the election result last May and are therefore looking to have another go\".': 3.734051305031632,\n",
       " ':: What does the Northern Ireland crisis mean for Brexit?': 3.914935398933713,\n",
       " 'Announcing the dissolution of the Northern Ireland Assembly, Mr Brokenshire urged both parties \"to conduct this election with a view to...re-establishing a partnership government at the earliest opportunity after that poll.\"': 3.859815891242049,\n",
       " 'He said: \"This is essential for the operation of devolved government.': 4.262011629913893,\n",
       " 'And this means that all must remain open to dialogue.\"': 1.9463973314187863,\n",
       " 'Please use Chrome browser for a more accessible video player\\n\\n\\n\\nSinn Fein and the DUP are expected to remain the two largest parties following the election, meaning they will still have to hammer out a power-sharing arrangement.': 4.008856758727256,\n",
       " 'If they fail to agree terms after three weeks, Mrs May could be forced to suspend devolution and reinstate direct rule from Westminster.': 2.10928346192458,\n",
       " 'Please use Chrome browser for a more accessible video player\\n\\n\\n\\nSky News Ireland Correspondent David Blevins said the relationship between Sinn Fein and the DUP had been \"slowly breaking down for a period of months\".': 4.2300315397277055,\n",
       " 'He said: \"Some would suggest that the British and Irish governments took their eye off the ball.': 3.323093452779095,\n",
       " '\"The botched renewable energy scheme is being blamed for the collapse of the devolved government but it was just the tip of the iceberg.\"': 3.775714845678408,\n",
       " 'He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\".': 4.702536247024379,\n",
       " '© 2024 Sky UK': 1.0}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = {sentences[i]: sum(matrix[i]) for i in range(len(matrix))}\n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We select five best sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\".',\n",
       " 'The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.',\n",
       " 'He said: \"This is essential for the operation of devolved government.',\n",
       " 'Please use Chrome browser for a more accessible video player\\n\\n\\n\\nSky News Ireland Correspondent David Blevins said the relationship between Sinn Fein and the DUP had been \"slowly breaking down for a period of months\".',\n",
       " 'An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced.']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sents = sorted(scores, key=scores.__getitem__, reverse=True)[:5]\n",
    "sents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced.',\n",
       " 'He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\".',\n",
       " 'He said: \"This is essential for the operation of devolved government.',\n",
       " 'Please use Chrome browser for a more accessible video player\\n\\n\\n\\nSky News Ireland Correspondent David Blevins said the relationship between Sinn Fein and the DUP had been \"slowly breaking down for a period of months\".',\n",
       " 'The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tuples = [(i, text.find(i)) for i in sents]\n",
    "sorted_tuples = sorted(tuples, key=lambda x: x[0])\n",
    "#Leave only sentences.\n",
    "best_sents = [i[0] for i in sorted_tuples]\n",
    "best_sents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can put everything together as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def intersection(sent1, sent2):\n",
    "    s1 = sent1.split(' ')\n",
    "    s2 = sent2.split(' ')\n",
    "    intersection = [i for i in s1 if i in s2]\n",
    "    return len(intersection) / ((len(s1) + len(s2)) / 2)\n",
    "\n",
    "def get_summary(text, limit=3):\n",
    "    sentences = sent_tokenize(text)\n",
    "    matrix = [[intersection(sentences[i], sentences[j]) for i in range(0,len(sentences))] for j in range(0,len(sentences))]\n",
    "    scores = {sentences[i]: sum(matrix[i]) for i in range(len(matrix))}\n",
    "    sents = sorted(scores, key=scores.__getitem__, reverse=True)[:limit]\n",
    "    best_sents = [i[0] for i in sorted([(i, text.find(i)) for i in sents], key=lambda x: x[0])]\n",
    "    return best_sents\n",
    "\n",
    "def summarize(text, limit=3):\n",
    "    summary = get_summary(text, limit)\n",
    "    print(title)\n",
    "    print()\n",
    "    print(' '.join(summary))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Snap election to be held in March after Northern Ireland government collapses\n",
      "\n",
      "An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced. He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\". He said: \"This is essential for the operation of devolved government. Please use Chrome browser for a more accessible video player\n",
      "\n",
      "\n",
      "\n",
      "Sky News Ireland Correspondent David Blevins said the relationship between Sinn Fein and the DUP had been \"slowly breaking down for a period of months\". The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.\n"
     ]
    }
   ],
   "source": [
    "summarize(text,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is our summary. The number of sentences in the summary is actually arbitrary, and can be changed where appropriate, to get the desired results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to improve this algorithm, I think that splitting sentences while calculating intersections might be changed. If we split by space, then punctuation stays attached to the words, and this yields problems when calculating sentence - wise similarities. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def intersection(sent1, sent2):\n",
    "    s1 = [i for i in word_tokenize(sent1) if i not in punctuations and i not in stop]\n",
    "    s2 = [i for i in word_tokenize(sent2) if i not in punctuations and i not in stop]\n",
    "    intersection = [i for i in s1 if i in s2]\n",
    "    return len(intersection) / ((len(s1) + len(s2)) / 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Snap election to be held in March after Northern Ireland government collapses\n",
      "\n",
      "An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced. He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\". Please use Chrome browser for a more accessible video player\n",
      "\n",
      "\n",
      "\n",
      "Sinn Fein and the DUP are expected to remain the two largest parties following the election, meaning they will still have to hammer out a power-sharing arrangement. Please use Chrome browser for a more accessible video player\n",
      "\n",
      "\n",
      "\n",
      "Sky News Ireland Correspondent David Blevins said the relationship between Sinn Fein and the DUP had been \"slowly breaking down for a period of months\". The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.\n"
     ]
    }
   ],
   "source": [
    "summarize(text,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there are bigrams and trigrams among the most common words. Now I'll use this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def intersection(sent1, sent2):\n",
    "    #As sentences are lists of tokens, there is no need to split them.\n",
    "    intersection = [i for i in sent1 if i in sent2]\n",
    "    return len(intersection) / ((len(sent1) + len(sent2)) / 2)\n",
    "\n",
    "def split_sentences(sents):\n",
    "    sentence_stream = [[i for i in word_tokenize(sent) if i not in stop] for sent in sents]\n",
    "    bigram = Phrases(sentence_stream, min_count=2, threshold=2, delimiter=b'_')\n",
    "    bigram_phraser = Phraser(bigram)\n",
    "    bigram_tokens = bigram_phraser[sentence_stream]\n",
    "    trigram = Phrases(bigram_tokens,min_count=2, threshold=2, delimiter=b'_')\n",
    "    trigram_phraser = Phraser(trigram)\n",
    "    trigram_tokens = trigram_phraser[bigram_tokens]\n",
    "    return [i for i in trigram_tokens]\n",
    "\n",
    "def get_summary(text, limit=3):\n",
    "    sents = sent_tokenize(text)\n",
    "    sentences = split_sentences(sents)\n",
    "    matrix = [[intersection(sentences[i], sentences[j]) for i in range(0,len(sentences))] for j in range(0,len(sentences))]\n",
    "    scores = {sents[i]: sum(matrix[i]) for i in range(len(matrix))}\n",
    "    sents = sorted(scores, key=scores.__getitem__, reverse=True)[:limit]\n",
    "    best_sents = [i[0] for i in sorted([(i, text.find(i)) for i in sents], key=lambda x: x[0])]\n",
    "    return best_sents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Snap election to be held in March after Northern Ireland government collapses\n",
      "\n",
      "\"The botched renewable energy scheme is being blamed for the collapse of the devolved government but it was just the tip of the iceberg.\" An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced. Announcing the dissolution of the Northern Ireland Assembly, Mr Brokenshire urged both parties \"to conduct this election with a view to...re-establishing a partnership government at the earliest opportunity after that poll.\" He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\". The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.\n"
     ]
    }
   ],
   "source": [
    "summarize(text,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The summary changed again. Numerous ways to split sentences may work better on some types of texts and worse on others."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}