03-Continuous_Integration.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<link href='http://fonts.googleapis.com/css?family=Open+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>\n",
       "<link href='http://fonts.googleapis.com/css?family=Merriweather:400,700,400italic' rel='stylesheet' type='text/css'>\n",
       "<link href='http://fonts.googleapis.com/css?family=Inconsolata' rel='stylesheet' type='text/css'>\n",
       "<style>\n",
       "\n",
       "@font-face {\n",
       "    font-family: \"Computer Modern\";\n",
       "    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
       "}\n",
       "\n",
       "#notebook_panel { /* main background */\n",
       "    background: rgb(245,245,245);\n",
       "}\n",
       "\n",
       "div.cell { /* set cell width */\n",
       "    width: 75em;\n",
       "}\n",
       "\n",
       "div #notebook { /* centre the content */\n",
       "    background: #fff; /* white background for content */\n",
       "    width: 80em;\n",
       "    margin: auto;\n",
       "    padding-left: 0em;\n",
       "}\n",
       "\n",
       "#notebook li { /* More space between bullet points */\n",
       "margin-top:0.8em;\n",
       "}\n",
       "\n",
       "/* draw border around running cells */\n",
       "div.cell.border-box-sizing.code_cell.running {\n",
       "    border: 1px solid #111;\n",
       "}\n",
       "\n",
       "/* Put a solid color box around each cell and its output, visually linking them*/\n",
       "div.cell.code_cell {\n",
       "    /*font-family: 'Inconsolata', monospace;*/\n",
       "    background-color: rgb(256,256,256);\n",
       "    border-radius: 0px;\n",
       "    padding: 0.5em;\n",
       "    margin-left:1em;\n",
       "    margin-right:1em;\n",
       "    margin-top: 1em;\n",
       "}\n",
       "\n",
       "div.text_cell_render{\n",
       "    font-family: 'Open Sans', sans-serif;\n",
       "    line-height: 140%;\n",
       "    font-size: 110%;\n",
       "    font-weight: 400;\n",
       "    padding: 0.5em;\n",
       "    margin-left:1em;\n",
       "    margin-right:1em;\n",
       "}\n",
       "\n",
       ".text_cell_render pre {\n",
       "    white-space: pre;\n",
       "    white-space: pre-wrap;\n",
       "    white-space: -moz-pre-wrap;\n",
       "    white-space: -o-pre-wrap;\n",
       "    background-color: #f5f5f5;\n",
       "    padding: 0.5em;\n",
       "}\n",
       "\n",
       ".text_cell_render code {\n",
       "    font-size: 95%;\n",
       "    white-space: pre;\n",
       "    white-space: pre-wrap;\n",
       "    white-space: -moz-pre-wrap;\n",
       "    white-space: -o-pre-wrap;\n",
       "    background-color: #f5f5f5;\n",
       "}\n",
       "\n",
       "/* Formatting for header cells */\n",
       ".text_cell_render h1 {\n",
       "    font-family: 'Merriweather', serif;\n",
       "    font-style: regular;\n",
       "    font-weight: 400;\n",
       "    font-size: 3em;\n",
       "    line-height: 100%;\n",
       "    color: rgb(36,67,87);\n",
       "    margin-bottom: 0.5em;\n",
       "    margin-top: 0.5em;\n",
       "    display: block;\n",
       "}\n",
       ".text_cell_render h2 {\n",
       "    font-family: 'Merriweather', serif;\n",
       "    font-weight: 400;\n",
       "    font-size: 2.2em;\n",
       "    line-height: 100%;\n",
       "    color: rgb(36,67,87);\n",
       "    margin-bottom: 0.3em;\n",
       "    margin-top: 0.3em;\n",
       "    display: block;\n",
       "}\n",
       "\n",
       ".text_cell_render h3 {\n",
       "    font-family: 'Merriweather', serif;\n",
       "    margin-top:1em;\n",
       "\tfont-size: 1.6em;\n",
       "    font-weight: 400;\n",
       "    margin-bottom: 0.3em;\n",
       "    font-style: regular;\n",
       "    color: rgb(85, 111, 127);\n",
       "}\n",
       "\n",
       ".text_cell_render h4 {\n",
       "    font-family: 'Inconsolata', monospace;\n",
       "    /*font-family: 'Merriweather', serif;*/\n",
       "    font-size: 1.3em;\n",
       "    margin-top: 0.3em;\n",
       "    margin-bottom: 0.5em;\n",
       "    font-style: regular;\n",
       "    color: rgb(85, 111, 127);\n",
       "}\n",
       "\n",
       ".text_cell_render h5 {  /*Captions*/\n",
       "    font-family: 'Merriweather', serif;\n",
       "    font-weight: 400;\n",
       "    font-size: 1em;\n",
       "    color: rgb(85, 111, 127);\n",
       "    font-style: regular;\n",
       "    margin-bottom: .1em;\n",
       "    margin-top: 0.8em;\n",
       "    display: block;\n",
       "    text-align: center;\n",
       "}\n",
       "\n",
       ".text_cell_render h6 { /*use this for copyright note*/\n",
       "    font-family: 'Inconsolata', sans-serif;\n",
       "    font-weight: 300;\n",
       "    font-size: 9pt;\n",
       "    line-height: 100%;\n",
       "    color: grey;\n",
       "    margin-bottom: 1px;\n",
       "    margin-top: 1px;\n",
       "}\n",
       "\n",
       "img {\n",
       "    margin-left: auto;\n",
       "    margin-right: auto;\n",
       "}\n",
       "\n",
       "/*.CodeMirror{\n",
       "        font-size: 120%;\n",
       "}*/\n",
       "\n",
       "</style>\n",
       "<script>\n",
       "    MathJax.Hub.Config({\n",
       "                        TeX: {\n",
       "                           extensions: [\"AMSmath.js\"],\n",
       "                           equationNumbers: { autoNumber: \"AMS\", useLabelIds: true}\n",
       "                           },\n",
       "                tex2jax: {\n",
       "                    inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
       "                    displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
       "                },\n",
       "                displayAlign: 'center', // Change this to 'center' to center equations.\n",
       "                \"HTML-CSS\": {\n",
       "                    styles: {'.MathJax_Display': {\"margin\": 4}}\n",
       "                }\n",
       "        });\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "css_file = \"./notebook_style.css\"\n",
    "HTML(open(css_file, 'r').read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Continuous Integration & Code Coverage\n",
    "\n",
    "- [Continuous integration](#Continuous-intergration)\n",
    "    * [Why?](#Why?)\n",
    "    * [Travis](#Travis-CI)\n",
    "    * [Local CI](#Local-CI)\n",
    "- [Code coverage](#Code-coverage)\n",
    "- [Read more](#Read-more)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Continuous integration\n",
    "\n",
    "![](images/continuous_integration.jpg)\n",
    "\n",
    "### Why?\n",
    "\n",
    "So, you've written a set of tests for your code, you run them and everything passes - great! However, you then go back to work on developing your code and quickly forget about testing it. A few months later, after implementing several new features and refactoring several important functions, you remember to try testing your code again. You run the set of tests, only to find that they fail. At this point, you face trawling through months of changes to try and locate where the bugs were first introduced.\n",
    "\n",
    "This could have been prevented if you had run your tests regularly, updating them often so that they reflect changes to your code. However, remembering to do this manually is less than ideal, plus it would be nice to have some kind of log of test results for each version of your code so you can avoid repeating past mistakes (as you can see things that broke the code in the past and avoid doing that again). \n",
    "\n",
    "The solution to this is continuous integration. This is a way of automating your tests so that they are run regularly (e.g. every night, every time you push changes to a repository), generating reports of test results for you to peruse at leisure and notifying you (almost) instantly when tests fail. Assuming you have a comprehensive, effective set of tests, you can now go ahead and develop your code safe in the knowledge that if you break something, you should find out almost as soon as you have committed (and pushed) the change and therefore be able to fix it before the buggy code becomes too entrenched. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Travis CI\n",
    "\n",
    "[Travis CI](https://travis-ci.org) is a remote continuous integration service that can be linked to your GitHub repository so that every time you push a change your tests get executed on a remote server. It will then generate a report detailing which tests passed and which failed and (if you wish) send you an email letting you know what happened. \n",
    "\n",
    "For simple python projects, Travis CI is pretty straightforward to set up. First, set up your account by [signing in](https://travis-ci.org/auth) to Travis CI with your GitHub account. Then go to your [profile page](https://travis-ci.org/profile) and enable Travis CI for your chosen repository.\n",
    "\n",
    "Next, you need to tell Travis what to do to test your project - how it should set up the remote server and which commands it should run to execute your tests. This is done by creating a `.travis.yml` file in your repository. For a python project, this file will look something like:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### .travis.yml\n",
    "```yml\n",
    "language: python # which language is our project written in\n",
    "python: # which versions of python should the code be tested in\n",
    "    - \"2.7\"\n",
    "    - \"3.5\"\n",
    "# command to install dependencies\n",
    "install: \n",
    "    - pip install -r requirements.txt\n",
    "# command to run tests\n",
    "script: pytest # or py.test for Python versions 3.5 and below\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because travis will be testing our code on a remote server, we need to tell it if we have used any python libraries which are not included in the core python installation. For this example, we will assume that all the packages needed for our project are listed on [`PyPI`](https://pypi.python.org/) so can be installed using `pip`. These packages are listed in the file `requirements.txt`. Once these packages have been installed, travis will then execute `pytest`. This particular `.travis.yml` file will test the code for python versions 2.7 and 3.5. \n",
    "\n",
    "Finally, add this `.travis.yml` file to git, commit and push it to the remote repository. This will trigger a Travis CI build. You can check to see if the build passes or fails by checking the [build status](https://travis-ci.org/repositories) page.\n",
    "\n",
    "Unfortunately, packages such as `numpy` are not included in the default python environment where travis does its testing. Therefore, if your project includes packages from the anaconda distribution, you'll need a [slightly more complicated script](http://conda-test.pydata.org/docs/travis.html) such as the one below. This downloads and installs a barebones version of anaconda, `miniconda`, onto the remote server before it runs the tests."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### .travis.yml\n",
    "```yml\n",
    "language: python\n",
    "python:\n",
    "  - \"2.7\"\n",
    "  - \"3.5\"\n",
    "before_install:\n",
    "  - sudo apt-get update\n",
    "# Setup anaconda\n",
    "install:\n",
    "  # We do this conditionally because it saves us some downloading if the\n",
    "  # version is the same.\n",
    "  - if [[ \"$TRAVIS_PYTHON_VERSION\" == \"2.7\" ]]; then\n",
    "      wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh;\n",
    "    else\n",
    "      wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;\n",
    "    fi\n",
    "  - bash miniconda.sh -b -p $HOME/miniconda\n",
    "  - export PATH=\"$HOME/miniconda/bin:$PATH\"\n",
    "  - hash -r\n",
    "  - conda config --set always_yes yes --set changeps1 no\n",
    "  - conda update -q conda\n",
    "  # Useful for debugging any issues with conda\n",
    "  - conda info -a\n",
    "  - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION numpy scipy matplotlib ipython nose\n",
    "  - source activate test-environment\n",
    "  - pip install -r requirements.txt\n",
    "# Run test\n",
    "script:\n",
    "  - pytest\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More advanced features for customising your travis build and a description of the travis build lifecycle can be found [here](https://docs.travis-ci.com/user/customizing-the-build/). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Local CI\n",
    "\n",
    "Remote continuous integration services are great for open source codes, however if your code is closed or uses non-standard hardware (e.g. GPUs), you may want instead to run your tests locally. Rather than manually running these tests every time you make changes to your code, you can use tools such as [`jenkins`](https://jenkins.io) to manage this for you. `Jenkins` is an automation server which can be used to automate a variety of tasks including building and testing. Once set up, much like `travis.ci`, it will automatically run your tests every time you commit changes to your code and generate reports on whether your tests passed or failed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code coverage\n",
    "\n",
    "So, you have written a test suite and are using continuous integration to run it regularly - awesome! However, how do you know that you are testing all parts of your code? If your tests only cover 20% of the code that executes, that is no guarantee that the other 80% is doing what it's supposed to, and therefore you still cannot trust your code's results. If you can't trust your results, how are you going to convince other people that the cool new result your code has produced is correct? \n",
    "\n",
    "Fortunately, rather than trawling through your code, inspecting each line to see if it's being tested, there are *code coverage* tools that can automate this. These tools can be linked with your continuous integration service so that they are run whenever your tests execute, generating a `.coveragerc` report and allowing you to spot parts of your code that could do with some more testing. \n",
    "\n",
    "There are code coverage libraries for most languages that will generate these coverage reports: `coverage.py` for python, `gcov` for C/C++, `tcov` for C/C++/fortran. [Codecov](https://codecov.io/) is a nice tool that can then be used to analyse these coverage reports. It keeps track of code coverage for each version of the code, providing a graphical interface which highlights which exact lines are covered / not covered by testing, and generally provides a lot of graphs and features which make monitoring the coverage of your tests a lot easier. It is also particularly useful if your code is written in multiple languages, as it will combine the reports produced by each of the languages.\n",
    "\n",
    "To run codecov on a python project, modify your `.travis.yml` file as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### With pytest\n",
    "\n",
    "```yml\n",
    "language: python\n",
    "python: \n",
    "    - \"2.7\"\n",
    "    - \"3.5\"\n",
    "# command to install dependencies\n",
    "install: \n",
    "    - pip install codecov\n",
    "    - pip install pytest-cov\n",
    "# command to run tests\n",
    "script: \n",
    "    - pytest --cov=./ # or py.test for Python versions 3.5 and below\n",
    "# if tests were all successful, run codecov\n",
    "after_sucess:\n",
    "    - codecov\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With nose\n",
    "\n",
    "```yml\n",
    "language: python\n",
    "python: \n",
    "    - \"2.7\"\n",
    "    - \"3.5\"\n",
    "# command to install dependencies\n",
    "install: \n",
    "    - pip install codecov\n",
    "    - pip install coverage\n",
    "# command to run tests\n",
    "script: \n",
    "    - nosetests -- with-coverage\n",
    "# if tests were all successful, run codecov\n",
    "after_sucess:\n",
    "    - codecov\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There may be a few sections of our code that do not make sense to test using automated testing and so we wish to exclude from coverage reports. For example, code that produces plots or graphical output would be difficult to write tests for and are perhaps better tested visually. We can exclude these sections from the coverage report using a `.coveragerc` file. For example, if all our plotting functions had names that began with `plot_`, our file would look like"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### .coveragerc\n",
    "\n",
    "```\n",
    "[report]\n",
    "exclude_lines =\n",
    "    def plot_*\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Read more\n",
    "- Check out this [Guide](https://docs.travis-ci.com/user/languages/python/) for building python projects in travis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}