FDS-CW1-2324.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<span style=\"display: none\">\n",
    "\\setcounter{secnumdepth}{0}\n",
    "    \n",
    "\\newcommand{\\answerbox}[1][5]{\n",
    "  \\noindent\\hskip -0.025\\linewidth\\framebox[1.05\\linewidth]{\\vbox to #1\\baselineskip{}}\n",
    "\n",
    "  \\vspace{-#1\\baselineskip}\n",
    "\n",
    "\\vspace{-\\baselineskip}}\n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Inf2-FDS 2023-24: Coursework 1 - Data wrangling and visualisation \n",
    "\n",
    "**Released:** See Assessment Section on FDS Learn Ultra pages\n",
    "\n",
    "**Submission deadline:** See Assessment Section on FDS Learn Ultra pages\n",
    "\n",
    "**Late submission rules:** See Assessment Section on FDS Learn Ultra pages\n",
    "\n",
    "**It is very important that you read and follow the instructions below to the letter: you will be deducted marks for not adhering to the advice below.**\n",
    "\n",
    "## Good Scholarly Practice \n",
    "\n",
    "Please remember the University requirement as regards all assessed work for credit.  Details about this can be found at: [http://web.inf.ed.ac.uk/infweb/admin/policies/academic-misconduct](http://web.inf.ed.ac.uk/infweb/admin/policies/academic-misconduct)\n",
    "\n",
    "Specifically, this coursework **must be your own work**. We want you to be able to discuss the class material with each other, but the coursework you submit must be your own work. You are free to form study groups and discuss the concepts related to, and the high-level approach to the coursework. **You may never share code or share write-ups**. It is also not permitted to discuss this coursework on Piazza. The only exception is that if you believe there is an error in the coursework, you may ask a private question to the instructors, and if we feel that the issue is justified, we will send out an announcement.\n",
    "\n",
    "## Assessment information and criteria\n",
    "\n",
    "- This coursework counts for 20% of the marks of Foundations of Data Science.\n",
    "\n",
    "- The coursework is designed to assess the following course Learning Outcomes: \n",
    "\n",
    "    1. Describe and apply good practices for storing, manipulating, summarising, and visualising data.\n",
    "    2. Use standard packages and tools for data analysis and describing this analysis, such as Python and LaTeX.\n",
    "\n",
    "- It is marked based on the PDF export from a Jupyter notebook (see __General instructions__ below), which you are to submit via Gradescope (see __Assessment/Assignment Submission/Coursework 1 - Data wrangling and visualisation__ folder in Learn).\n",
    "\n",
    "- The assignment is marked out of 100 and the number of points is indicated by each question. \n",
    "\n",
    "- We will assess your work on the following criteria:\n",
    "  \n",
    "  - functional code that\n",
    "    (a) performs the computations asked for, as measured by verifying some of the numeric outputs from your processing and reading your code if there is doubt about what you have done and\n",
    "    (b) is clear and makes good use of Pandas functions\n",
    "    \n",
    "  - the quality of the visualisations, measured against the _Visualisation principles and guidance_ handout used in the [S1 Week 5 workshop](https://opencourse.inf.ed.ac.uk/inf2-fds/course-materials/semester-1/week-5/workshop) \n",
    "  \n",
    "  - the quality of your textual comments - as measured by how accurate, clear, complete and insightful they are.\n",
    "\n",
    "\n",
    "## General instructions\n",
    "\n",
    "- Read the instructions carefully, answering what is required. \n",
    "\n",
    "- Read through all of a question before starting, as some parts build on each other.\n",
    "\n",
    "- In order to understand the meaning the datasets, you may need to follow the links citing the data.\n",
    "\n",
    "- Fill in your answers in the cells indicated. You may delete text like \"Your answer to Q1.2 goes here\". __Do not edit or delete any other cells.__\n",
    "\n",
    "- Once you have finished a question, use a Jupyter notebook server to export your PDF, by selecting __File→Download as→PDF via LaTeX (.pdf)__. Check this PDF document looks as you expect it to. If changes are needed, update the notebook and export again.\n",
    "\n",
    "- We ask for multiple types of responses:\n",
    "\n",
    "  1. Numeric responses, which we will use to verify that the processing has been done correctly.\n",
    "  2. Visualisations, which will be assessed using the parts of the _Visualisation principles and guidance_ PDF that apply to the visualisation in question.\n",
    "  3. Comments on your findings, which may ask you to describe, explain or interpret your results, and which may ask you to reflect on your design choices. These comments will be assessed on how accurate, complete and insightful they are. \n",
    "  \n",
    "- Keep your answers brief and concise - in the PDF output, textual answers should remain inside the box.\n",
    "\n",
    "- For answers involving numerical values, use correct units where appropriate and format floating point values to a reasonable number of decimal places.\n",
    "\n",
    "- For answers involving visualisations:\n",
    "  - Make sure to label the visualisations clearly and provide titles, and legends where necessary, as per the Visualisation guidelines.\n",
    "  - All visualisations have to fit on one page. For example, 3 graphs on 3 separate pages counts as 3 visualisations; We would only mark one of these 3 visualisations.\n",
    "  - \"One visualisation\" could  be a figure with multiple subplots, for example created using the using the matplotlib subplot and grid system.\n",
    "  - Creating plots in separate code cells that appear on the same page is not recommended. We may consider both plots as one visualisation, but mark you down for it. At worst we may only mark one visualisation.\n",
    "  - Please follow the guidance on font size in the _Visualisation Principles and Guidance_, i.e. a real font size of 8 points at a minimum. We instruct the markers to view all submissions at the same magnification and not to zoom in, to give a fair comparison between students.\n",
    "\n",
    "- We strongly recommend restarting the kernel and running all (__Kernel→Restart & Run all__) from time to time, to test your code works properly. \n",
    "\n",
    "- Once you have finished all the questions, submit the final PDF using the submission instructions in the __Assessment→Assignment Submission→Coursework 1 - Data wrangling and visualisation__ folder in Learn Ultra. __Please allow enough time to upload your PDF before the deadline.__\n",
    "\n",
    "## How long should I spend on this coursework?\n",
    "\n",
    "We have designed the coursework to take around 8 hours for a student who has completed the lab sessions, attended the lectures and workshops and done the comprehension questions. There will be some spread around this value, but if you're up-to-date with the course, and you're taking much longer than 8 hours (say 12 hours), then you might want to consider how many marks you are gaining for each extra hour of work.\n",
    "\n",
    "Good luck - we hope you enjoy it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "# Imports - run this cell first, and add your own imports here.\n",
    "# You can use any package you want, but we suggest you stick to ones used in the labs\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "\n",
    "matplotlib.rcParams['figure.dpi'] = 150 # Make figures have reasonable resolution when exporting\n",
    "\n",
    "# My imports\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "# Question 1 (20 points) - Identifying and correcting bad visualisation practices\n",
    "\n",
    "Designing a good visualisation can be time-consuming, but is important for clear communication. In this question we will ask you to identify and correct bad visualisation practices.  (Note that we'd like you to continue using good visualisation practices in other parts of this coursework too!)  \n",
    "\n",
    "Our visualisation journey starts in Scotland in the year 2016. We'll visualise the number of deaths associated with different cancer types, discovering which ones pose a greater threat.\n",
    "\n",
    "In the cell below, the code reads in a dataset containing information about the number of deaths for each cancer type in various countries and across different years. The dataset includes several columns, such as 'Entity' displaying country names, 'Year' indicating the occurrence year, and multiple columns representing the number of deaths for each cancer type.\n",
    "\n",
    "We specifically filtered the rows corresponding to Scotland and the year 2016. Using this subset of data, we created a new dataframe that presents the number of deaths for each cancer type recorded in Scotland during the year 2016. \n",
    "\n",
    "The data was adapted from a [Kaggle dataset](https://www.kaggle.com/datasets/programmerrdai/cancer?resource=download) that is under a CC0: Public Domain license.\n",
    "\n",
    "1. Run the code in the cell below. After a few seconds, a plot should appear, which is an example of bad visualisation practice. \n",
    "2. Export this document to PDF, by using by selecting __File→Download as→PDF via LaTeX (.pdf)__.\n",
    "3. Look at the plot in Jupyter and in the PDF.\n",
    "\n",
    "The plot displays a pie chart presenting the proportion of deaths for each cancer type in Scotland during the year 2016."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Data of cancer types and their respective deaths number in Scotland in 2016\n",
    "# Read dataframe\n",
    "df = pd.read_csv('cancer-deaths-by-type-grouped.csv')\n",
    "# Select only rows related to Scotland and year 2016\n",
    "df = df[(df.Entity=='Scotland') & (df.Year==2016)]\n",
    "# Create a new dataframe with the selected data\n",
    "df_new = pd.DataFrame({'Deaths_number': df.values[0][3:],\n",
    "                       'Cancer_type': df.columns[3:]})\n",
    "\n",
    "# Function to generate a pie chart \n",
    "def bad_visualization(df):\n",
    "    # Create a pie chart\n",
    "    plt.pie(df['Deaths_number'], autopct='%1.6f%%', startangle=90, shadow=True,\n",
    "            textprops={'fontsize': 3, 'color': 'red'})\n",
    "    # This changes from 150 to 72 when \"running all\"...\n",
    "    if matplotlib.rcParams['figure.dpi'] != 150:\n",
    "        print(\"matplotlib.rcParams['figure.dpi'] is not equal to 150 dpi. Reset using\\nmatplotlib.rcParams['figure.dpi'] = 150\")\n",
    "    # Show the plot\n",
    "    plt.show()\n",
    "\n",
    "# Call the function to generate the plot\n",
    "bad_visualization(df_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 1.1 (10 points)\n",
    "\n",
    "The plot above is meant to allow for a visual comparison between the number of deaths due to different cancer types in Scotland in 2016, but it's an example of bad visualisation practice.\n",
    "\n",
    "List as many problems as you can with this visualisation and what you would do to fix them. For each problem you should include:\n",
    "- which one of the 5 principles in the _Visualisation principles and guidance_ handout is broken\n",
    "- the reason why the plot breaks that principle\n",
    "- how you would change the plot to fix this problem (this may be obvious, in which case you don't need to say much)\n",
    "\n",
    "You should be able to list at least 8 changes, and you will score 1 point for each problem correctly identified with a principle, and an extra point for suggested fixes, if they are not obvious ones. If you do not identify the connection between the principle and the reason the visualisation breaks the principle, you will score 0 marks.\n",
    "\n",
    "Please make sure that your answer is no longer than 20 lines, i.e. stays in the box in the PDF file you have produced from this notebook. You can delete the text saying \"Your answer goes here\".\n",
    "\n",
    "<span style=\"display: none\">\\answerbox[20]{}</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Your answer for Q1.1 goes here. Please use the format below and delete this line.\n",
    "\n",
    "1. < Name of principle >: < Reason the principle is broken. >. < What you would fix if not obvious. >\n",
    "2. < Name of principle >: < Reason the principle is broken. >. < What you would fix if not obvious. >\n",
    "3. < Name of principle >: < Reason the principle is broken. >. < What you would fix if not obvious. >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 1.2 (10 points)\n",
    "\n",
    "Write the code in the cell below to implement the changes you listed in Q1.1 so someone presented with it could easily interpret the improved plot. Afterwards, run the code to produce the plot itself. Remember that when you export this notebook to a PDF, it will have a text width of 6 inches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for Q1.2 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "# Question 2 (15 points) - Exploring the Unintentional Injury dataset\n",
    "\n",
    "In Question 2, we will start exploring the [Unintentional Injury dataset](https://www.opendata.nhs.scot/dataset/unintentional-injuries), which contains hospital admissions data from the National Health Service across Scotland up to and including the financial year 2021/22. In addition, data from death registrations ([Deaths dataset](https://www.opendata.nhs.scot/dataset/unintentional-injuries/resource/89807e07-fc5f-4b5e-a077-e4cf59491139)), obtained from the National Records of Scotland up to and including the calendar year 2022, adds valuable insights into the severity and consequences of these incidents. The datasets provide insights into the occurrences of unintentional injuries and their impact. The datasets cover attributes such as financial year, health board residence, council area, age groups, gender, injury location, injury type, and the total number of admissions.\n",
    "\n",
    "To discover the meaning of the attributes of each dataset, look at the \"Data Dictionary\" section in the links to the data:\n",
    "\n",
    "- [Admissions dataset](https://www.opendata.nhs.scot/dataset/unintentional-injuries/resource/aee43295-2a13-48f6-bf05-92769ca7c6cf)\n",
    "\n",
    "- [Deaths dataset](https://www.opendata.nhs.scot/dataset/unintentional-injuries/resource/89807e07-fc5f-4b5e-a077-e4cf59491139)\n",
    "\n",
    "In Question 3, we will perform data wrangling to clean and prepare the data for further analysis. We'll combine this data with another relevant dataset in Question 4 to gain a more comprehensive understanding. Through a set of questions and data manipulation tasks, we will create meaningful visualisations and perform trend analysis to identify patterns that might help injury prevention. The questions are organised so that the first few questions give more detailed instructions and the later questions are more open-ended. We also want you to consider whether there are any limitations of the data that should be considered. Finally, the datasets used in this notebook are not large enough to cause issues, but do consider computational complexity and efficiency of your code.\n",
    "\n",
    "Let's dive into this journey of exploring the Unintentional Injury dataset, unraveling the insights that might contribute to a safer and healthier community."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 2.1 (2 points)\n",
    "\n",
    "Read in the Unintentional Injury datasets, both Admissions and Deaths datasets. They are located in the folder `ui_data` and are called `ui_admissions_2022_df.csv` and `ui_deaths_2022_df.csv` respectively. Then load the file called `ca11_ca19.csv`, which contains labels for Council Area in one column and its associated Council Code in another. Name the read dataframes `admissions`, `deaths` and `council_areas`. Then, print the number of rows and columns in all three data sets in the format:\n",
    "```\n",
    "<Dataset>: <n> rows and <m> columns\n",
    "```\n",
    "Here `<Dataset>` is a string that makes it clear which dataset you're referring to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for Q2.1 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 2.2 (2 points)\n",
    "\n",
    "Calculate and print the following information:\n",
    "\n",
    "1. Number of injury types in the Admissions dataset\n",
    "2. Number of council areas in the Admissions dataset \n",
    "3. Number of council areas in the Council dataset\n",
    "4. Total number of deaths in the Deaths dataset\n",
    "\n",
    "Print the results in following format:\n",
    "```\n",
    "<Number of Injury Types in Admissions dataset>: value\n",
    "<Number of Council areas in Admissions dataset>: value\n",
    "<Number of Council areas in Council dataset>: value\n",
    "<Total number of deaths in the Deaths dataset>: value\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Your code for Q2.2 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 2.3 (3 points)\n",
    "\n",
    "Report the __codes__ of those council areas that are in the Council dataset but not in the Admissions dataset and vice versa; print them out in the following format:\n",
    "```\n",
    "Council codes in Admissions dataset but not in Council dataset:\n",
    "<List of council codes>\n",
    "Council codes in Council dataset but not in Admissions dataset:\n",
    "<List of council codes>\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Your code for Q2.3 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 2.4 (2 points)\n",
    "\n",
    "1. Check the number of missing values in the Admissions dataset, specially for AgeGroup, CA, and Sex columns. \n",
    "2. What do you observe? Is there a significant number of missing values in the data?\n",
    "\n",
    "Please make sure that your answer is no longer than 5 lines, i.e. stays in the box in the PDF file you have produced from this notebook. You can delete the text saying \"Your answer goes here\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for Q2.4 part 1 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\answerbox[5]{}</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Your answer to Q2.4 part 2 goes here:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 2.5 (2 points)\n",
    "\n",
    "Filter the Admissions dataset by selecting only the columns for Sex, NumberOfAdmissions, InjuryType, FinancialYear, HBR, CA, and AgeGroup.\n",
    "\n",
    "Filter the Deaths dataset by selecting only the columns for Sex, NumberofDeaths, InjuryType, Year, HBR, CA, and AgeGroup.\n",
    "\n",
    "Print the head of both dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for Q2.5 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 2.6 (4 points)\n",
    "Using the dataframes prepared in Q2.5:\n",
    "\n",
    "1. Print the list of injury types in both the Admissions and Deaths datasets.\n",
    "2. What do you observe? If we wanted to merge the deaths and admissions datasets, what changes would you propose making to the naming of the injury types?\n",
    "3. Apply the necessary cleaning and changes to make the datasets ready for the next steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for Q2.6 part 1 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\answerbox[5]{}</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Your observation for Q2.6 part 2 goes here:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for Q2.6 part 3 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "# Question 3 (10 points) - Merging in council names and tidying\n",
    "    \n",
    "The Admissions and Deaths datasets currently only contain Council area codes. To enhance the datasets, we aim to add the names of those council areas into the datasets.\n",
    "\n",
    "**Note: if you get stuck you can skip this question, and go to question 5.** \n",
    "\n",
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 3.1 (5 points)\n",
    "\n",
    "Using the Admissions and Deaths dataframes created in Q2.6:\n",
    "\n",
    "- Merge each dataframe with the Council dataframe to include the CAName column in the each of the two new merged dataframes. The dataframes with the CAName column merged in should have the same number of rows as the dataframes created in Q2.6. \n",
    "\n",
    "- Calculate the percentage of rows in each dataframe that lack corresponding council names.\n",
    "\n",
    "- Remove rows with null CAName values from both the Admissions and Deaths datafames.\n",
    "\n",
    "Afterwards:\n",
    "\n",
    "1. Print the percentage (to two decimal places) of data with null CAName for both Admissions and Deaths datasets separately (i.e. the datafames created step 1 above)\n",
    "\n",
    "2. Print the dimensionality of the new Admissions and Deaths datasets, after removing rows with null CAName (i.e. the datafames created in step 3 above)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for Q3.1 goes here\n",
    "\n",
    "# Merge dataframes...\n",
    "\n",
    "# Calculate percentage of rows without council names\n",
    "\n",
    "# Remove rows with null CANames\n",
    "\n",
    "# 1. Print the percentage (to two decimal places) of data with null CAName for both Admissions and Deaths datasets separately\n",
    "\n",
    "# 2. Print the dimensionality of the new Admissions and Deaths datasets, after removing rows with null CAName"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 3.2 (5 points)\n",
    "\n",
    "1. Calculate the sum of NumberofDeaths for each injury type during year 2020. \n",
    "\n",
    "2. Make the meaning of the numbers clear by putting them in context. You answer should not exceed 5 lines.  \n",
    "\n",
    "**Use the dataset before removing null values.**\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for Q3.2 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\answerbox[5]{}</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Your answer to Q3.2 goes here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "# Question 4 (30) - Trends across Council areas\n",
    "\n",
    "In this question we will see if there are any trends across council areas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 4.1 (5 points)\n",
    "\n",
    "Using the dataframes calculated in Q3.1, which were obtained after removing rows with null CouncilName, now create a new dataframe that contains the **total** NumberofDeaths and NumberofAdmissions over the period 2012 unitil 2021 for for each CouncilName and InjuryType. Assume that the period of financial years from 2012/13 until 2021/22 is equivalent to the period 2012-2021. The new dataframe should contain only the records for which you have __both__ the number of admissions and the number of deaths. After creating the new DataFrame, report its dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for Q4.1 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 4.2 (7 points)\n",
    "\n",
    "Utilize the dataframe created in Q4.1 and write code that provides the following information:\n",
    "\n",
    "1. The top 2 council areas with the highest total number of deaths due to all injury types.\n",
    "2. The top 2 council areas with the highest total number of admissions due to all injury types.\n",
    "3. Identify the two types of injuries that had the lowest total number of deaths due to all injury types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for Q4.2 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 4.3 (10 points)\n",
    "\n",
    "Compare the number of deaths and admissions between council areas using a visualisation of your choice. You can use either separate DataFrames or a Tidy DataFrame (which you may have created in Q4.1). The points here are for the visualisation - you won't lose points for not having done Q4.1. If you want to create a map, use `gb.csv`, which contains latitude and longitude of each council area. However, a map plot will not necessarily gain extra points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for Q4.3 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 4.4 (8 points)\n",
    "\n",
    "1. Explain your reasons behind your visualisation design choice and summerise its pros and cons.\n",
    "2. Interpret the `Unintentional Injury` visualisation that you created; i.e. summarise the patterns in the visualisation and any particularly noteworthy features.\n",
    "\n",
    "Please make sure that your answer is no longer than 20 lines, i.e. stays in the box in the PDF file you have produced from this notebook. You can delete the text saying \"Your answer goes here\".\n",
    "\n",
    "\n",
    "<span style=\"display: none\">\\answerbox[20]{}</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Your answer for Q4.4 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "## Question 5: Create your own visualisation (25 points)\n",
    "\n",
    "It's now your opportunity to create your own visualisation of the Unintentional Injury data. You can use either the Deaths or Admissions data, or you can use the combined DataFrame created in Question 4.  Please don't bring in external datasets - the idea is that this is a small exercise, not a whole project.\n",
    "\n",
    "Here are some ideas:\n",
    "\n",
    "- How does the number of unintentional injury admissions/deaths vary across different age groups, and which age group is most affected?\n",
    "- Is there a gender disparity in the number of unintentional injury admissions/deaths, and if so, how does it vary across different injury types?\n",
    "- What is the trend in the number of unintentional injury admissions over the years? Are there any notable changes or patterns in the data?\n",
    "- Is there a correlation between the number of injury admissions and the number of injury deaths? Does a higher number of admissions lead to a higher number of fatalities?\n",
    "\n",
    "We will give credit for particularly interesting questions, good visualisations (as measured by the criteria in the visualisation principles handout) and insightful interpretation.\n",
    "\n",
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "### Question 5.1 (15 points)\n",
    "\n",
    "Create your visualisation here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code for Q5.1 goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<span style=\"display: none\">\\newpage</span>\n",
    "\n",
    "### Question 5.2 (10 points)\n",
    "\n",
    "1. Note the question your visualisation addresses.\n",
    "2. Explain your reasons behind your visualisation design choice and summarise its pros and cons. Please make sure that your answer is no longer than 5 lines.\n",
    "3. Write your interpretation of the visualisation. Please make sure that your answer is no longer than 20 lines (plus to 5 lines for your references in the case that you referenced to external resources), i.e. stays in the box in the PDF file you have produced from this notebook. You can delete the text saying \"Your answer goes here\".\n",
    "\n",
    "<span style=\"display: none\">\\answerbox[35]{}</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "**Question:** Your question for Q5.3 goes here\n",
    "\n",
    "**Reasoning:** Your reasons behind your visualisation design choice for Q5.3 go here\n",
    "\n",
    "**Interpretation:** Your interpretation of the visualisation for Q5.3 goes here"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "deepnote": {
   "is_reactive": false
  },
  "deepnote_execution_queue": [],
  "deepnote_notebook_id": "3669a0bc-e82c-4d6a-be64-b70117ad1d2f",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  },
  "title": "Inf2 - Foundations of Data Science 2021-22",
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "311.383px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}