Xcode 12, iOS 14, and Swift 5.3 bring with them a number of significant improvements in testing. If you live and die by tests — or even if you just wish you did — you’re going to be very happy to hear about these. Some of these changes actually appeared earlier, in Swift 5.2, Xcode 11.4, that sort of thing.

Fail or Throw

The motto of testing is: Fail early, fail often. Did you know that there are two ways to do that?

• Assertions: A test method may call one or more assertions. These are the global functions whose names begin with XCTAssert. If any assertion fails, the test method fails. (By default, a test method continues running after an assertion fails. That might not be what you want; arguably, failure puts the test method in a bad state. To prevent it, set the continueAfterFailure property of your XCTestCase to false beforehand.)

• Throws: A test method can be declared with throws. In that case, throwing an error in the body of the method counts as the test failing. Throwing aborts the test method immediately.

  The setUp and tearDown XCTTestCase instance methods also have throwing alternatives, setUpWithError and tearDownWithError. If you implement both setUp and setUpWithError, the error method is executed first. If you implement both tearDown and tearDownWithError, the error method is executed last.

The new setUpWithError and tearDownWithError methods are very nice, but in my opinion Apple does something rather rude with them: starting in Xcode 11.4, when you create a new unit test bundle, the template gives you these new methods, rather than the old setUp and tearDown. That can cause trouble in production, especially if (as happened to me) your remote CI hasn’t been updated yet. In my opinion, Apple should let you adopt new features when you’re ready, rather then injecting them into your code without warning.

Or Skip

Besides passing or failing, there is now a third option. You know, like win, lose, or draw. Rather than succeeding or failing, a test method declared with throws may be skipped.

The idea is that you skip a test based on conditions discovered at runtime (for instance, perhaps this test makes sense only on a certain kind of device, or only on your computer but not at your remote CI site). To test those conditions, you can call a global function, either XCTSkipIf or XCTSkipUnless; these are throwing functions, so the call is preceded by try, and what they throw is an XCTSkip instance. Or you can test the conditions yourself, and construct and throw your own XCTSkip instance. When an XCTSkip instance is thrown, the test method is aborted and neither succeeds nor fails.

Or Time Out

It’s possible that a test will hang — perhaps there’s a threading deadlock — or just take too long. You don’t want to come back from running your tests overnight and discover that they’re still incomplete because of a hang.

New in Xcode 12, an XCTestCase or a test plan can declare an execution time allowance. If a test takes longer than that, it fails, and a spindump file is attached to the test report.

More Useful Utilities

It’s always a good idea to factor out common functionality into utility methods that are called by many test methods. In the past, however, this could make tracking down failures rather tricky. If the utility method calls an assertion that fails, the point of the failure is the utility method — not the test method that was calling it. So which test actually failed?

New in Xcode 12, the call stack is preserved and displayed in the source file, in the Issue navigator, and in the report. So if you were shying away from test utility methods because they were more trouble than they were worth, shy no more.

Authorization Alerts

In UI tests, a complication can arise when you want to test your app’s behavior with regard to a resource that’s protected by user authorization, such as calendar access, contacts access, and so forth. What you’d probably like to do is to clear out the destination’s access settings so that your test can bring up the authorization dialog and behave in the different ways the user can behave, either allowing or denying access.

By the same token, you’d probably like to launch your app into a known state. If you’re going to be testing something that depends on having access, you want to pass through the authorization dialog so that you know you do have access.

New in Xcode 12 (actually this seems to have started in Xcode 11.4), you can call the XCUIApplication method resetAuthorizationStatus(for:). The idea is that you do that before launch. Now when the app runs and requests access, you know the dialog will appear, your UI test can tap the appropriate button, and your test can proceed with a knowledge of what the authorization status is.

Metrics

You probably know about performance testing: it lets you check that the speed of an operation has not fallen off by running that operation repeatedly and timing the result. You call the XCTestCase instance method measure; it takes a function whose execution time will be recorded. The first time you run a performance test, you establish a baseline measurement, and on subsequent runs, it fails if the standard deviation of the times is too far from the baseline, or if the average time has grown too much.

It turns out that, starting in Xcode 11, in addition to measuring elapsed time, your performance tests can measure things like CPU and memory usage. You call measure(metrics:) with an array of XCTMetric objects, such as XCTCPUMetric. (I seem to have missed the memo on this one.) There is also XCTApplicationLaunchMetric, which lets a performance test exercise launching your app and report whether it launches as quickly as it should.

Also, there is now an os_signpost type .animationBegin, along with various other XCTOSSignpostMetric additions allowing you to test animations such as scrolling and custom view controller transitions; animation “hitches” are reported as a ratio between the overall time spent animating and the amount of delay between each screen refresh and the actual appearance of the next “frame” of the animation. More than 5ms of hitch per second of animation should be considered bad.

By the way, for realistic performance testing you should run a release build without the debugger attached and with all sanitizers and diagnostics turned off. You can manage that with a custom scheme (and possibly a test plan).

Massaging the Report

Nothing is more important after running your tests than understanding what happened. Your key source of information is the report. You can see this within Xcode; your remote CI may provide it as an xcreport file.

It turns out that XCTest includes powerful features that give you a lot of control over what goes into the report.
One simple but effective trick is to group the report output into meaningful activities. To do so, in your test method call XCTContext.runActivity(named:). In addition to the name, which should be some explanatory string, it takes a function; that function is where you’ll do your testing and call your assertions. The outcome is that, in the test report summary, the activity name appears as a line in the report, and the test results are grouped hierarchically under that line.

Another useful device is attachments. You probably know that you can deliberately attach a screen shot to your report during UI testing. But an attachment can be any kind of data, and it isn’t confined to UI testing. XCTAttachment has initializers for attaching a string, an image, the contents of a file or a directory, a Data object, and more. You can call self.add (where self is the XCTest) to get your attachment into the test report. A call to XCTContext.runActivity is also a great place to add an attachment: the function parameter hands you an XCTActivity object, and you call add with your attachment.

Finally, new in Xcode 12, the entire report mechanism relies upon an XCTIssue object that is passed into the XCTestCase record method to contribute to the report. You can intervene in this mechanism! Instead of throwing or calling a built-in assert, you can create your own XCTIssue, populate it, and record it. Here’s a simple example:

let loc = XCTSourceCodeLocation(filePath: #file, lineNumber: #line) // point of failure
var issue = XCTIssue(type: .assertionFailure, compactDescription: "oh darn")
issue.add(XCTAttachment(string:"yipes"))
issue.sourceCodeContext = XCTSourceCodeContext(location: loc)
self.record(issue)

At an even deeper level, you can override record and modify the incoming XCTIssue before calling super — or don’t call super and thus suppress the failure entirely.

Learn more about Xcode 12: What a Pane