Fix CI flakiness

[JohnSundell]

Our CI can sometimes exhibit some flaky behavior in that it'll fail for no good reason, and a retrigger solves the problem. Let's investigate what's going wrong (maybe it's a caching thing, or some state that isn't properly cleaned up, or some race condition?) and fix it!

Interestingly, the flakiness is different on BuddyBuild vs Travis.

Travis

Here's what a failed build log usually looks like on Travis:

Test Suite 'All tests' started at 08:31:26.018
Test Suite 'debug.xctest' started at 08:31:26.042
Test Suite 'MarathonTests' started at 08:31:26.042
Test Case 'MarathonTests.testInvalidCommandThrows' started at 08:31:26.042
Test Case 'MarathonTests.testInvalidCommandThrows' passed (0.002 seconds)
Test Case 'MarathonTests.testAddingAndRemovingRemotePackage' started at 08:31:26.044
Test Case 'MarathonTests.testAddingAndRemovingRemotePackage' passed (1.959 seconds)
Test Case 'MarathonTests.testAddingAndRemovingLocalPackage' started at 08:31:28.004
Test Case 'MarathonTests.testAddingAndRemovingLocalPackage' passed (0.729 seconds)
Test Case 'MarathonTests.testRemovingAllPackages' started at 08:31:28.733
Test Case 'MarathonTests.testRemovingAllPackages' passed (9.061 seconds)
Test Case 'MarathonTests.testAddingLocalPackage' started at 08:31:37.794
Test Case 'MarathonTests.testAddingLocalPackage' passed (0.54 seconds)
Test Case 'MarathonTests.testAddingLocalPackageWithDependency' started at 08:31:38.334
Test Case 'MarathonTests.testAddingLocalPackageWithDependency' passed (1.286 seconds)
Test Case 'MarathonTests.testAddingLocalPackageWithUnsortedVersionsContainingLetters' started at 08:31:39.620
Test Case 'MarathonTests.testAddingLocalPackageWithUnsortedVersionsContainingLetters' passed (0.914 seconds)
Test Case 'MarathonTests.testAddingAlreadyAddedPackageThrows' started at 08:31:40.534
Test Case 'MarathonTests.testAddingAlreadyAddedPackageThrows' passed (5.021 seconds)
Test Case 'MarathonTests.testTreatingNestedDependenciesAsAdded' started at 08:31:45.555
Test Case 'MarathonTests.testTreatingNestedDependenciesAsAdded' passed (2.605 seconds)
Test Case 'MarathonTests.testRunningScriptWithoutPathThrows' started at 08:31:48.160
Test Case 'MarathonTests.testRunningScriptWithoutPathThrows' passed (0.004 seconds)
Test Case 'MarathonTests.testRunningScript' started at 08:31:48.164
Test Case 'MarathonTests.testRunningScript' passed (3.542 seconds)
Test Case 'MarathonTests.testRunningScriptWithNewDependency' started at 08:31:51.705

travis_time:end:0d599c36:start=1496651472166114307,finish=1496651515992986208,duration=43826871901
The command "swift test" exited with 1.�

Done. Your build exited with 1.

As you can see above, it just exits with code 1, no additional debug information seems to be found 😢 It also seems like it stops on different test cases every time this happens, so don't think it's a problem with a single test case. Perhaps a timeout issue?

BuddyBuild

On BuddyBuild things are a lot more consistent, and should be easier to fix. It always fails on this test case:

Test Case '-[MarathonTests.MarathonTests testAddingAlreadyAddedPackageThrows]' started.

fatal error: unexpectedly found nil while unwrapping an Optional value

Current stack trace:
0    libswiftCore.dylib                 0x00000001104ba130 swift_reportError + 129
1    libswiftCore.dylib                 0x00000001104d6b50 _swift_stdlib_reportFatalError + 60
2    libswiftCore.dylib                 0x00000001102c7250 specialized specialized StaticString.withUTF8Buffer<A> ((UnsafeBufferPointer<UInt8>) -> A) -> A + 342
3    libswiftCore.dylib                 0x0000000110441e90 partial apply for (_fatalErrorMessage(StaticString, StaticString, file : StaticString, line : UInt, flags : UInt32) -> Never).(closure #2) + 109
4    libswiftCore.dylib                 0x00000001102c7250 specialized specialized StaticString.withUTF8Buffer<A> ((UnsafeBufferPointer<UInt8>) -> A) -> A + 342
5    libswiftCore.dylib                 0x00000001103f49a0 specialized _fatalErrorMessage(StaticString, StaticString, file : StaticString, line : UInt, flags : UInt32) -> Never + 96
6    MarathonPackageTests               0x00000001100bc200 MarathonTests.tearDown() -> () + 108
7    MarathonPackageTests               0x00000001100bc340 @objc MarathonTests.tearDown() -> () + 34
8    XCTest                             0x000000010dc45657 __24-[XCTestCase invokeTest]_block_invoke_2 + 1492
9    XCTest                             0x000000010dc83b4c -[XCTestContext performInScope:] + 190
10   XCTest                             0x000000010dc45546 -[XCTestCase invokeTest] + 254
11   XCTest                             0x000000010dc45e7e -[XCTestCase performTest:] + 565
12   XCTest                             0x000000010dc42efa __27-[XCTestSuite performTest:]_block_invoke + 300
13   XCTest                             0x000000010dc42bf9 -[XCTestSuite _performProtectedSectionForTest:testSection:] + 29
14   XCTest                             0x000000010dc42d26 -[XCTestSuite performTest:] + 214
15   XCTest                             0x000000010dc42efa __27-[XCTestSuite performTest:]_block_invoke + 300
16   XCTest                             0x000000010dc42bf9 -[XCTestSuite _performProtectedSectionForTest:testSection:] + 29
17   XCTest                             0x000000010dc42d26 -[XCTestSuite performTest:] + 214
18   XCTest                             0x000000010dc42efa __27-[XCTestSuite performTest:]_block_invoke + 300
19   XCTest                             0x000000010dc42bf9 -[XCTestSuite _performProtectedSectionForTest:testSection:] + 29
20   XCTest                             0x000000010dc42d26 -[XCTestSuite performTest:] + 214
21   XCTest                             0x000000010dc96b24 __44-[XCTTestRunSession runTestsAndReturnError:]_block_invoke + 40
22   XCTest                             0x000000010dc5950c -[XCTestObservationCenter _observeTestExecutionForBlock:] + 587
23   XCTest                             0x000000010dc968d1 -[XCTTestRunSession runTestsAndReturnError:] + 281
24   XCTest                             0x000000010dc2e9ac -[XCTestDriver runTestsAndReturnError:] + 254
25   XCTest                             0x000000010dc85fb8 _XCTestMain + 773
26   xctest                             0x000000010dc1c48d <unavailable> + 5261
27   libdyld.dylib                      0x00007fffea83c234 start + 1

Rebuilding without cache seems to fix the problem. Perhaps some cleanup not being made?

[cojoj]

Most of the time when I was getting strange errors in different CIs it was regarding timeouts. We're using free plans, so the bandwidth etc. is relatively slow.
Marathon test suite is heavy, so I wouldn't be surprised if it will turn out to be timeout errors.

Regarding buddybuild error. Maybe extracting try? folder.subfolder(named: "Packages").file(named: "Files").read() to a let can will help?

[cojoj]

Feels like Swift 3.2 introduced new problem. Take a look at this integration .
I've faced this issue locally as well when I was running tests. It's stuck on testAddingAndRemovingRemotePackage or testAddingLocalPackage.

[JohnSundell]

@cojoj Fixed that issue here: #104