Adds videos to testing chapter

Author: nickbradley

reader/content/_index.md

@@ -12,6 +12,12 @@ This is a high-level overview of what the course will be about.
 The materials roughly break down into 6 high-level modules that are spread across the 13 week of standard academic semester at UBC.
 Readings and videos are available for most course concepts.
 
+Videos throughout the reader provide additional explanations of the course material. The video below introduces the codebase that is used throughout the course reader. 
+
+{{< expand title="Introduction to running example" >}}
+{{% youtube S4GEa6JMo4I %}}
+{{< /expand >}}
+
 ## License
 
 The readings for this course are licensed using [CC-by-SA](https://creativecommons.org/licenses/by-sa/3.0/). However, it is important to note that the deliverable descriptions, code implementing the deliverables, exams, and exam solutions are considered private materials. We go to considerable lengths to make the project an interesting and useful learning experience for this course. This is a great deal of work, and while future students may be tempted by your solutions, posting them does not do them any real favours. Please be considerate with these private materials and not pass them along to others, make your repos public, or post them to other sites online.

reader/content/testing/_index.md

@@ -5,32 +5,23 @@ title: "Software Testing"
 ---
 
 
-{{% notice default "Learning Outcomes" "graduation-cap" %}}
-By the end of this chapter you should:
+[//]: # ({{% notice default "Learning Outcomes" "graduation-cap" %}})
 
-* [x] Identify
-* [x] [Interpreted vs Complied Languages]({{< ref "languages#interpreted-vs-compiled" >}})
-* [x] Static vs Dynamic Representations of the System
-* [x] [Static vs Dynamic Types]({{< ref "languages#static-vs-dynamic-types" >}})
-  {{% /notice %}}
+[//]: # (By the end of this chapter you should:)
 
+[//]: # ()
+[//]: # (* [x] Identify)
 
-{{< youtube Uamo4Ej0tWk >}}
-
-
-{{< youtube ll1k3Pks3ZA >}}
+[//]: # (* [x] [Interpreted vs Complied Languages]&#40;{{< ref "languages#interpreted-vs-compiled" >}}&#41;)
 
-     * [Properties of Tests]&#40;http://www.youtube.com/watch?v=ll1k3Pks3ZA)
+[//]: # (* [x] Static vs Dynamic Representations of the System)
 
-[//]: # (      * [Kinds of Tests]&#40;http://www.youtube.com/watch?v=_Th3f9vks_w&#41;)
+[//]: # (* [x] [Static vs Dynamic Types]&#40;{{< ref "languages#static-vs-dynamic-types" >}}&#41;)
 
-[//]: # (      * [Unit and System Properties]&#40;http://www.youtube.com/watch?v=x2DWjxDiOQo&#41;)
+[//]: # (  {{% /notice %}})
 
-[//]: # (      * [Red Green Refactor]&#40;http://www.youtube.com/watch?v=v0q1MKhSQVM&#41;)
 
-{{< youtube x2DWjxDiOQo >}}
-
-{{< youtube v0q1MKhSQVM >}}
+{{< youtube Uamo4Ej0tWk >}}
 
 
 Software systems are only useful if they do what they are supposed to do. This is increasingly important given the vital safety-critical roles modern software systems play. Unfortunately, proving that a system is correct is exceedingly difficult and expensive for large software systems. This leads to the fundamental tradeoff at the heart of most commonly applied testing approaches:
@@ -43,6 +34,10 @@ Given the constraints above, the most prevalent quality validation approach in u
 
 This is because what we are evaluating is whether the probability that there are no bugs in a system given that the tests pass is not actually 0. Since tests themselves are programs (and can have bugs) and specifications are often incomplete or imprecise we therefore must admit that the chance of a defect slipping through a set of tests is > 0 [(online discussion)](http://tonyxzt.blogspot.ca/2010/01/tests-can-show-presence-of-bugs-not.html).
 
+## Testing Process
+
+{{< youtube v0q1MKhSQVM >}}
+
 The modern test cycle is summarized below. The cycle between steps 1-7 can occur many times before the code is ready to commit. 
 
 1. Develop some new code.
@@ -60,20 +55,28 @@ Steps 2/7 may not happen on small teams or when testing happens solely on a sing
 
 {{< figure src="test-cycle.png" >}}
 
-### Terminology
+## Terminology
 
 {{< youtube WKrvx7qCUDI >}}
 
 A number of different terms are commonly used in the testing space:
 
 * ```SUT/CUT```: System / code under test. This is the thing that you are actually trying to validate.
-* ```Glass box testing```: When testing in a glass box manner one typically carefully examines the program source code in order to identify potentially problematic sets of inputs or control flow paths. More details can be found in the [Glass box testing](GlassBoxTesting.md) reading.
-* ```Black box testing```: Black box testing validates programs without any knowledge of how the system is implemented. This form of testing relies heavily on predicting problematic inputs by examining public API signatures and any available documentation for the CUT. More details can be found in the [Black box testing](BlackBoxTesting.md) reading.
+* ```Glass box testing```: When testing in a glass box manner one typically carefully examines the program source code in order to identify potentially problematic sets of inputs or control flow paths. More details can be found in the [Glass box testing]({{% ref "glassbox" %}}) reading.
+* ```Black box testing```: Black box testing validates programs without any knowledge of how the system is implemented. This form of testing relies heavily on predicting problematic inputs by examining public API signatures and any available documentation for the CUT. More details can be found in the [Black box testing]({{% ref "blackbox" %}}) reading.
 * ```Effectiveness```: The simplest way to reason about the effectiveness of a test or test suite is to measure the probability the test will find a real fault (per unit of effort, which can be something like developer creation / maintenance time or number of test executions).
-* ```Higher/lower testability```: Some systems are significantly easier to test than others due to the way they are constructed. A highly testable system will enable more effective tests for the same cost than a system whose tests are largely ineffective (or require an outsized amount of creation and maintenance effort). More details can be found in the [Testability](TestabilityAssertions.md) reading.
+* ```Testability```: Some systems are significantly easier to test than others due to the way they are constructed. A highly testable system will enable more effective tests for the same cost than a system whose tests are largely ineffective (or require an outsized amount of creation and maintenance effort). More details can be found in the [Testability]({{% ref "testability" %}}) reading.
 * ```Repeatability```: The likelihood that running the same test twice under the same conditions will yield the same result.
 
 
+## Properties of tests
+
+{{< youtube ll1k3Pks3ZA >}}
+
+## Kinds of tests
+
+{{< youtube _Th3f9vks_w >}}
+
 There are a number of different *levels* of test; these range in size, complexity, execution duration, repeatability, along with how easy they are to write, maintain, and debug.
 
 * ```Unit```: Unit tests exercise individual components, usually methods or functions, in isolation. This kind of testing is usually quick to write and the tests incur low maintenance effort since they touch such small parts of the system. They typically ensure that the unit fulfills its contract making test failures more straightforward to understand.
@@ -84,17 +87,21 @@ There are a number of different *levels* of test; these range in size, complexit
 
 {{< figure src="test-levels.png" >}}
 
+{{< expand title="Comparing unit and system tests" >}}
+{{% youtube x2DWjxDiOQo %}}
+{{< /expand >}}
+
 For additional reference, take a look at this in-depth talk about how [Google tests](https://www.infoq.com/presentations/Continuous-Testing-Build-Cloud) their systems.
 
-### Why not test?
+## Why not test?
 
 There are a number of reasons why software systems are not tested with automated suites. These range from "bad design" to "slow", "boring", "doesn't catch bugs" and "that's QA's job". Ultimately, testing does have a cost: tests are programs too and take time to write, debug, and execute and must also be evolved along with the system. 
 
 One core paradox of automated test suites is how they are used. If you think that a test only provides value when it fails by catching a bug that would have otherwise made it to production, then that all passing tests are just a waste of computational cycles. At the same time though, passing tests _do_ give us the warm feeling that we have not introduced unexpected regression bugs into our systems. But at the same time, these feelings only work if we believe our test suite is capable of finding faults and _could_ fail. Yes: this contradicts the 'we should only run failing tests' implication above. 
 
 An area of interesting future research would be to figure out how often a suite needs to fail to impart trust, while also figuring out how few passing tests you could run to have a sense of confidence in your automated suite. 
 
-### Common testing assumptions
+## Common testing assumptions
 
 It has been long held that the cost of fixing a fault rises exponentially with how late in the development process (e.g., requirements, design, implementation, deployment) the fault is detected. This statement arises from several influential studies Barry Bohem performed in the 60s and 70s. A more in-depth description of these costs can be found [here](http://www.agilemodeling.com/essays/costOfChange.htm) or [here](http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20100036670.pdf). 
 
@@ -106,7 +113,7 @@ It is also important to accurately consider the costs of automated testing. Writ
 Testing as an engineering discipline notes
 -->
 
-### References
+## References
 
 * Another great talk about how [Google stores](https://www.youtube.com/watch?v=W71BTkUbdqE) its source code.
 

reader/content/testing/blackbox/_index.md

@@ -32,8 +32,7 @@ There are a variety of black box testing approaches that are used in practice:
 
 	Use case testing is similar to a customer validating a user story by evaluating the definitions of done for a story in a sprint review. In this way use case testing and user story testing form a kind of user acceptance test.
 
-<a name="input"></a>
-## Input partitioning
+## Input partitioning {id="input"}
 
 {{< youtube 5PtxXnwyU3Y >}}
 
@@ -109,8 +108,7 @@ number). The final step is combining the sets to arrive at four test cases, as e
 * Sometimes the inputs to a function are less interesting than the outputs; in these cases, output partitioning may be a better choice.
 * Defects often arise at the boundaries of partitions. Input partitioning is often combined with boundary value analysis to cover these cases more comprehensively.
 
-<a name="output"></a>
-## Output partitioning 
+## Output partitioning {id="output"}
 
 {{< youtube 0yvHDKI-DSA >}}
 
@@ -166,9 +164,3 @@ The inputs to these functions are just both non-negative numbers. But the output
 
 * Functions usually have fewer outputs than inputs (simplistically a function has a single output value and can have many input parameters). This usually drives testers to explore 'easier' input space more thoroughly than the output space (although it should be noted that the complexity of the output space is often underestimated, and the 'single' return value could in fact be a complex object or side effecting operation).
 * The effort required to derive inputs to achieve a given output is often challenging. This can be especially observed in erroneous situations (e.g., an output behaviour dictates how a system should behave when disk space is exhausted, but actually testing this can be hard).
-
-  
----
-[![](figures/CCSA.png "Creative Commons: Attribution-ShareAlike")](https://creativecommons.org/licenses/by-sa/3.0/) [Reid Holmes](https://www.cs.ubc.ca/~rtholmes/)
-
-

reader/content/testing/testability/_index.md

@@ -27,12 +27,12 @@ Sometimes the CUT can be invoked by a test but its outcome cannot be observed. F
 
 {{< youtube Z93-c4ngxGw >}}
 
-{{< youtube NMuhE-XnFe8 >}}
-
 Being able to isolate a fault within the code under test is crucial to be able to quickly determine what has caused a failure so it can be resolved. This is challenging in large modern systems due to the number of (often third party) dependencies software systems have. For example, if a data access routine fails is it the logic in the routine or is it a failure in the underlying database? At its simplest level, isolateability can be increased by decomposing larger functions into smaller more self-contained functions that can be tested independently.
 
 Sometimes code will have complex dependencies, requiring isolation through simulation. In simulation-based environments code dependencies are _mocked_ or _stubbed_ whereby they are replaced with developer-created fake components that take known inputs and return known values (e.g., a ```MockLoginRejectController```) would always return false for a ```login(user, pass)``` without needing to check a user store, database, or external system. In this way the developer can test their code that uses ```login(..)``` and ensure it handles the false case correctly without fear that a bug in the real login controller may return an incorrect or inconsistent value. In addition to isolation, mocking also greatly increases performance and makes components less prone to non-determinism as the result being returned is usually fixed and not dependent on some external complex computation. Mocking can also make it possible to test program states that would otherwise be hard to trigger in practice (for instance if you want to test a situation where a remote service is down you can have a ```MockTimeoutService``` that just does not respond to requests).
 
+{{< expand title="Code Example" >}} {{% youtube NMuhE-XnFe8 %}} {{< /expand >}}
+
 ### Automatability
 
 {{< youtube Q83W5zH8LUY >}}