I had 34 UI tests, 135 switchable bugs, a mock backend that answered in 120 ms, a price feed that ticked every second, and a minus sign that wasn’t a hyphen. Not that I needed all of that to test one banking app. But once you’ve built a serious collection of bugs, you want to switch every one of them on and see what happens.

On the clean build, all 34 tests were green. That’s nice, but it only says the app works. It says nothing about whether the tests would notice if it didn’t.

So I started breaking the app on purpose and watching which tests went red. The first results weren't great. Nine of the 34 tests never went red, whatever I broke. And when I switched on the 23 money bugs one at a time, 18 of them got through the whole suite without a single red test.

Below are the four that taught me the most, and what I changed in the tests. None of it is specific to my app or my framework. I'd bet most UI suites have at least one of these.

How the app gets broken

The app is ChaosBank, a banking app I built to practise mobile testing on (intro post). By default its code is correct. It also has 135 named bugs, and each one can be switched on at launch:

if Defects.isActive(.pnlSign) { // the small, deliberate bug } else { // the correct implementation }

The tests are the ChaosBank × KassiOS reference suite, written on KassiOS, my layer over XCUITest (the post about it). A bug is switched on through the test runner's environment, so the tests themselves stay the same:

TEST_RUNNER_CB_INJECT_DEFECTS=pnlSign xcodebuild test-without-building …

If this sounds like mutation testing, it is, just at a different level. Tools like PIT, Stryker and Muter change your code automatically thousands of times and rerun the unit tests after each change. UI tests are far too slow for that, so ChaosBank's bugs are written by hand, at the level a user would notice: a loss shown as a gain, a total that skips an account. The suite runs once per bug. If some test goes red, the bug is "killed". If everything stays green, it "survived".

A wrong number is still a number

This was the first test that stayed green with every bug on at once:

func test_portfolio_showsTotalAndPnl() { launchUnlocked(tab: "portfolio") onScreen(PortfolioScreen.self) { p in p.totalValue.assertVisible() p.pnl.assertPresent() p.list.assertExists() } }

One of those bugs, pnlSign, shows TSLA's loss as a gain. The test didn't care. It checks that the numbers are on the screen, and a wrong number is on the screen just as well as a right one. To be fair, the name says exactly that: "shows total and P&L". It never promised they'd be correct.

The fix should have been one line. TSLA is held at a loss, so its P&L should have a minus in it:

func test_portfolio_tslaLoss_isRenderedAsLoss() { launchUnlocked(tab: "portfolio") onScreen(PortfolioScreen.self) { p in p.holdingPnl("TSLA").assertLabelContains("−") } }

It took two detours.

The first: on the clean build the test failed with "does not exist", even though portfolio.holding.TSLA.pnl was right there in the view code. I dumped the accessibility tree and saw that every text in the holdings list had the identifier of the list itself. It turns out that in SwiftUI, .accessibilityIdentifier on a container gets passed down to its children and replaces their own identifiers, unless the container is marked .accessibilityElement(children: .contain). The row identifiers had never reached XCUITest at all.

That explained the old test. It could only check presence, because there was no way to find the value. I looked for the same pattern in the rest of the app and found it in 14 more places: the login gate, the card, the market rows, the transaction rows, the confirm sheets. The tests had quietly worked around nearly all of them. The CVV, for example, was found by looking for a text containing "CVV" among everything on the card. Fixing the app took one modifier per container, and after that I could delete 81 lines of workarounds from the tests.

The second detour: my first version checked for "-" and failed on the clean build with expected label containing '-' but found '−$229.84'. ChaosBank formats money with U+2212, the real minus sign. It looks like a hyphen but it's a different character. If you check a formatted value, check it for the characters the formatter actually puts there.

€100 was the wrong amount to test with

transferRoundsUp takes €101 from your account when you send €100.40. The original transfer test sent €100. With €100, a correct transfer and a rounded-up one are the same €100, so no check in the world could have caught this with that input.

There's a second bug on the same flow, transferDebitsWrongAccount, which takes the money from USD instead of EUR. That one is easy to catch, as long as the test looks at EUR at all. The old test only checked that the confirm sheet closed.

The new test sends an amount with cents, and also checks the account that shouldn't change:

/// A €100.40 transfer debits exactly €100.40 from EUR and leaves USD alone. /// Catches `transferRoundsUp` and `transferDebitsWrongAccount`. func test_transfer_debitsExactlyTheAmountFromEUR() { launchUnlocked() var eurBefore = Double.nan, usdBefore = Double.nan onScreen(HomeScreen.self) { home in eurBefore = home.account("EUR").lastNumber() usdBefore = home.account("USD").lastNumber() home.transferAction.tap() } onScreen(TransferScreen.self) { transfer in transfer.recipientField.typeText("Alex") transfer.amountField.typeText("100.40") transfer.continueButton.tap() transfer.confirmButton.tap() transfer.confirmButton.within(timeout: 20).waitUntilGone() } onScreen(HomeScreen.self) { home in XCTAssertEqual(home.account("EUR").lastNumber(), eurBefore - 100.40, accuracy: 0.005) XCTAssertEqual(home.account("USD").lastNumber(), usdBefore, accuracy: 0.005) } }

Every number was right except the total

totalValueOmitsHolding leaves ETH out of the portfolio total. Each holding shows the right value. The total is formatted correctly. It's just the wrong sum, and none of my tests ever compared the total with the holdings.

So I wrote that test: read the holdings, add them up, compare with the total. It failed on the clean build every now and then. Prices in ChaosBank tick, and reading four labels one after another can catch two different ticks. The values have to be read at the same moment:

func test_portfolio_total_equalsSumOfHoldings() throws { launchUnlocked(tab: "portfolio") let ids = ["AAPL", "TSLA", "ETH"].map { "portfolio.holding.\($0).value" } var labels: [String: String] = [:] onScreen(PortfolioScreen.self) { p in labels = p.readTexts(ids + ["portfolio.totalValue"]) } let sum = try ids.map { try XCTUnwrap(numbers(in: try XCTUnwrap(labels[$0])).first) }.reduce(0, +) let total = try XCTUnwrap(numbers(in: try XCTUnwrap(labels["portfolio.totalValue"])).first) XCTAssertEqual(total, sum, accuracy: 0.02) }

readTexts reads all the labels from one snapshot of the screen. Once I had it, I wrote a few more checks of the same kind: the Home total against its three accounts at the app's exchange rates, a holding's value against the market price, and the loan's monthly payment against the advertised rate. On a money screen, a lot of what can go wrong is in how the numbers relate to each other, not in any single number.

The double tap that never raced

exchangeDoubleSubmit removes the guard that stops you from tapping Execute twice, so a fast double tap makes two exchanges. I had a test that taps twice. It stayed green.

There were two reasons. The mock backend answers in about 120 ms, and two separate synthesized taps arrive further apart than that. So the first exchange had already finished when the second tap came in, the guard had nothing to stop, and two exchanges in a row is also what two deliberate taps would give you. XCUIElement.doubleTap() wouldn't help either: it's one system gesture, and the button turns it into one action. It can't race anything.

What worked was slowing the network down. ChaosBank's dev menu has a "Slow" network setting (a network condition, not a bug), which keeps the request in flight for 3 seconds. The test turns it on, taps twice on the same spot without checking the button in between, and then counts rows in the history instead of checking that the sheet closed:

onScreen(ExchangeScreen.self) { ex in ex.amountField.typeText("100") ex.executeButton.rapidTap(times: 2) ex.executeButton.within(timeout: 20).waitUntilGone() } onScreen(HomeScreen.self) { $0.seeAllActivity.tap() } onScreen(TransactionsScreen.self) { tx in tx.count.assertValue(closeTo: countBefore + 1, tolerance: 0.001) }

With the guard in place there's one new row, without it there are two. The same trick caught the double charge on transfers and the double buy on orders.

Whole groups that turned nothing red

In the first run, the localization and flaky groups didn't turn a single test red, and security turned one. Nothing was wrong with the tests there. There just weren't any tests for those things. You don't see that from a green run on the clean build. You only see it when you switch the bugs on.

Most of the tests I added for them are short:

  • under the German locale, numbers group as 1.234.567,89 and the euro sign comes last;
  • a transfer amount typed as 1,000.50 parses correctly;
  • a wrong OTP is rejected, and three wrong ones lock the step;
  • the card number is masked except for the last four digits;
  • the session token is in the Keychain (the dev menu reports where it is);
  • a deep link opened while the app is locked doesn't skip the login screen.

Two bugs no test can catch on this data

Of the 23 money bugs, 21 are caught now. The other two, detailChangeWrongBase and pnlPercentAbsOnly, look exactly like the correct code. The first computes a percentage against the wrong base, but on the simulated price feed both bases give the same number. The second drops the sign from a P&L percentage, but the seeded portfolio is up overall, so the sign is positive anyway.

In mutation testing these are called equivalent mutants. Here they're only equivalent on this particular data. I didn't write tests for them, but they're a good reminder that the test data decides which bugs you can see in the first place.

Where the suite is now

There are 71 tests. Every test that's meant to catch a bug says which one in a doc comment:

/// A €100.40 transfer debits exactly €100.40 from EUR and leaves USD alone. /// Catches `transferRoundsUp` and `transferDebitsWrongAccount`.

A script collects these lines into a list, and kassios mutants switches on each listed bug and runs its tests, one bug at a time. Before that it runs all the tests on the clean build, and they all have to pass. All 59 bugs on the list are caught. That's 59 of ChaosBank's 135. The performance and network bugs have no tests yet, and most of the UI and state ones aren't covered by any test.

There's one problem I haven't fixed. On a slower CI machine running two simulators at once, two of the 59 got through. One was the deep link test. It waits a fixed 2 seconds and then checks that the login screen is still there. Under load, the app most likely hadn't handled the link within those 2 seconds even with the bug on, so the test passed. A test that checks "nothing happened" by waiting a while will pass whenever the machine is too slow for anything to happen yet. The fix is to wait for a sign that the link was handled, and that's still on my list.

Trying this on your own app

You don't need ChaosBank or KassiOS for any of this. Pick one screen with numbers on it. In a local build, break one thing on purpose: flip a sign, drop an item from a total, round an amount. Run your tests. If nothing goes red, you know which test to write next. And before you trust a new test, make it fail once on a build where the thing it checks is broken.

If you'd rather have something ready to break, every ChaosBank bug is listed in exercises.json, with the screen, the locators and what the broken build does. You can switch one on from the command line:

xcrun simctl launch booted VadimToptunov.ChaosBank \ -ChaosBankStartUnlocked 1 -ChaosBankTab portfolio -ChaosBankDefects pnlSign

Nobody has touched the performance and network bugs yet, if you're looking for somewhere to start. And I'd like to hear which of these mistakes you've seen in your own tests, and which ones I haven't run into yet.

Links