A Quest for better Testing of iOS Applications

When we started to work on our first big project together - Comicol - we were all about testing. We wrote a lot of unit test cases for every single method of every single class. About 30%-40% of the project’s overall code was code for unit tests.

Back then, we didn’t know much about mocking and stubbing or BDD and we’ve just used what Xcode offered - SenTestingKit UnitTest. Our testcases were overly specific and very hard to read.

Maintaining these tests to match the changing application became harder, and statements like “We have to get this feature done now, we’ll fix the tests later.” eventually became common before we stopped testing completely. Most of the testcases were failing because they were out of date and fixing them was less important than getting the project done (Spoiler: the project was never finished, as seems to happen ever so often with overly ambitious first projects, but it was a huge learning experience nevertheless).

Rails, RSpec and BDD

Fast forward: my first contact wit Rails and more specifically with R-Spec.

If you don’t know about RSpec yet, you should definitely take a look at it. Basically, you don’t write Unit Tests per se, you rather describe the behavior of objects or classes via specifications (that’s why the the design methodology is called Behavioral Driven Development). One of the very fundamental principles of RSpec is that the specifications as well as the output and especially error-messages are very readable, even for people who don’t know how to write code.

A very basic specification for a user would look something like that in Ruby:

describe "User" do
    let(:user) { User.new }
    subject { user }
    its(:name) { should == "Franz" }
end

In this specification we describe a user. A new user is created and set as the current test-subject. Then we expect its name to be equal to “Franz”.

Obviously this test makes little sense (why should every new user be named Franz?), but it shows how similar the syntax is to the description in natural language that I just gave.

Adhering to BDD when working on some Rails projects didn’t feel like the burden it was at the end of Comicol. It was easy to work with and gave us a lot of confidence in our work when changing something.

Naturally, when coming back to iOS I was wondering if we could do something similar there.

Search

My search started at Mattt Thompson’s NSHipster with his post about UnitTesting. His list of open source libraries gave me some important pointers on where to look further.

Furthermore, this post on Stackoverflow about iOS Tests/Specs TDDD/BDD contains a pretty extensive list of available methods and frameworks, including Acceptance Testing which I will investigate at a later point in time.

Kiwi

Since I was looking for something RSpec style, Cedar and Kiwi seemd to be both viable and seemingly similar options.

I’ve decided to use the latter for now since it’s built on top of OCUnit and therefore integrates into Xcode while Cedar has its own runner that needs to be run outside of Xcode. Additionally, Kiwi comes with its own set of matchers and a framework for mocking/stubbing, so I don’t have to find additional libraries for these tasks for now.

Getting Started

You can get Kiwi via CocoaPods (which you should consider using anyway if you aren’t already).

Just follow the Getting Started Guide.

Custom Matchers

One thing I was missing a little bit was a better documentation on how to implement custom matchers, the one example in the wiki was a little bit too basic. So I’ll explain a little bit more about this with another example:

Let’s say we have a simple user model. A user has a name and an access-token. Let’s say, the name must always be non-nil to be valid and the access-token must be nil, if the user is logged out and non-nil if the user is logged in. We want to build a matcher that can validate both, a logged-in and a logged-out user.

Our custom matcher will be a subclass of KWMatcher.

First, it must be declared which strings are provided by the custom matcher.

+ (NSArray *)matcherStrings {
    return @[@"beLoggedOutUser", @"beLoggedInUser"];
}

These matcher strings correspond to the methods that will be called, so these two methods have to be implemented:

- (void)beLoggedOutUser {
    _loggedIn = NO;
}

-(void)beLoggedInUser {
    _loggedIn = YES;
}

This implementation is fairly simple, the information if the user is supposed to be logged in or not is stored for later.

The real magic happens in the evaluate method. This is just a shortened version where I’ve scrapped the construction of the shouldNot failure message, but you’ll get the idea.

- (BOOL)evaluate {
    BOOL valid = YES;
    IVUser *user = (IVUser *)self.subject;

    self.shouldErrorMessage = @"Expected subject to be valid, but:";

    // A User must always have a username
    if(!user.userName) {
        valid = NO;
        self.shouldErrorMessage = [self.shouldErrorMessage stringByAppendingFormat:@"\n\tUsername was nil"];
    }

    if(self.loggedIn) {
        // If the User is logged in, the AccessToken must not be empty
        if(!user.accessToken) {
            valid = NO;
            self.shouldErrorMessage = [self.shouldErrorMessage stringByAppendingFormat:@"\n\tAccessToken was nil"];
        }
    }
    else {
        // If the User is logged out, the AccessToken must be nil
        if(user.accessToken) {
            valid = NO;
            self.shouldErrorMessage = [self.shouldErrorMessage stringByAppendingFormat:@"\n\tAccessToken was not nil"];
        }
    }

    return valid;
}

Finally, the failureMessage has to be provided:

-(NSString *)failureMessageForShould {
    return self.shouldErrorMessage;
}

I’ve provided a demo-project that demonstrates a working implementation of a custom matcher here on GitHub. To be honest, I’m not completely satisfied with how the evaluate method looks, it gets bloated with boilerplate code very fast. If you know a better way or some best practice please let me know!!

Missing Features

I’m currently still getting to know Kiwi, but I really like it so far.

Theres two things I’m missing already though:

• Shared Examples: The ability do combine several expectations into a behavior and then specifying that an object should behave like a specific example. Shared example are a great way to DRY up testing code by moving a lot of repeated code into one common place.
• Factories: In Rails, FactoryGirl provides a great way to specify test-data in a organized way and, again, DRY up test code by removing initialization code of subjects.

If you know some frameworks for these kind of tasks please let me know!

Conclusio

We are currently using Kiwi in one of our iOS projects that recently started. I’m curious how its use will unfold over the course of the project. Later on, we will have to find something additional for user interaction and GUI tests and maybe (hopefully :D ) we’ll look into continuous integration again.