We recently had to finish a project under a lot of pressure, and decided that it would be nice if we could empower the ui/application exerciser monkey to automatically send us crash reports.

There are a few setbacks to using the adb monkey as is, though:

It’s inconvenient to kill a running monkey.
You can’t let it run unobserved for extended periods of time because it ends after a crash or a freeze.
You either watch the log in your (running) sdk, or you manually handle logcat.
Managing all of the above on multiple devices is a real pain.

To this cause, I’ve created monkey_master, a convenience tool for running monkeys with multi-device support and automatic log creation, that can be easily combined with other scripts.

If you already know Crashlytics and the adb monkey, and already guess where this is going, feel free to check out monkey_master on github right now: https://github.com/j4zz/monkey_master

monkey_master is also available as ruby gem:

gem install monkey_master

The Setup

When finished, you’ve got a setup on your hands that tests your Android app using monkey_master, automatically reports crashes using Crashlytics if they occur, and logs each run. You could run it on your regular development machine over night, while you’re on a break during your workday, or completely automated on your build server.

monkey_master uses the adb monkey, a program that simulates a monkey messing around on your device by generating a pseudo-random stream of events such as touches, gestures or system-level events.

Since monkey_master doesn’t report errors itself, it relies on an error reporting framework like Crashlytics. Crashlytics advertises itself as the most powerful, yet lightest weight crash reporting solution. This description is pretty spot on, and the framework can be integrated easily.

After you’ve integrated Crashlytics into your app, it will automatically report crashes in real-time, provide detailed analysis, send e-mails, and, rather important for our use case, do all of this even if the wireless connection of the test device is temporarily disabled (this is especially nice, since the adb monkey may disable all connections using the pull-down menu). The only thing monkey_master does in regards to error reporting, is trying to create crashes that Crashlytics can report.

To install the described setup, integrate Crashlytics into your app, and go to github to install monkey_master.

Start your Engines

As soon as you’ve integrated Crashlytics and installed monkey_master, you can start to test your app on all connected devices by calling:

monkey_master com.my.App

For prolonged testing, use more than one iteration. Per iteration, the monkey executes a certain number of interactions. If the app freezes, the monkey ignores the freeze and continues sending interactions until it’s done, the app automatically restarted and the error reported. It is therefore advisable to have a high number of iterations with relatively short-lived monkeys:

monkey_master com.my.App --iterations 100

That’s it! The monkeys now randomly fool around in your app, and you get notifications and error reports if you encounter freezes or crashes.

If you’d like to check out what happened during a particular monkey_master iteration in more detail, take a look at the monkey_logs folder.

Testing Considerations

Is using monkey_master enough testing for your app? In most cases, the answer will clearly be no. While the adb monkey is good at detecting most regular crashes, it has no semantic knowledge whatsoever. Since app users associate crashes with low quality, fixing most crashes is a nice start, but no substitution for real unit tests.

Our setup only notifies you if a crash occurs. It does not, however, tell you if the behavior of your app is correct. For example, your news app might have a bug where it only fetches entries that are 30 years old. As long as the app doesn’t crash, monkey_master won’t mind. A well made unit test case, or a manual test, would detect such a bug.

Tricks

Here are some tricks you might need when using monkey_master.

Check if the Monkey is Running

Sometimes you encounter errors that only a monkey can generate, but that you can’t reasonably fix in your code. In case of such an error, you will want to exit gracefully if the user is a monkey (in order to suppress useless crash reports), but leave the error handling as it is for production.

Furthermore, you might want to disable certain network calls, or redirect to them to a test server.

For such cases, there’s ActivityManager.isUserAMonkey():

if(ActivityManager.isUserAMonkey()) {
    // Work on the test server
} else {
    // Work on the production server
}

Thanks to @MarcRichards for that advice.

Listen to them, the children of the night. What music they make!

If your device is one of those that comes with a pre-installed music player app that you can’t completely remove, and which automatically re-enables itself, then it might start to play music in the middle of the night, because a certain monkey thought it’d be funny.

The nightly quick-fix in my case was to plug in the headphones and do this:

test device sound prevention system :P

If you have a more graceful solution, though, please contact me: [email protected] or @LukasJNagl

Update: @PoliInkorrPappa kindly pointed out his old trick of cutting the wires on the headphones in order to eliminate any sound. As soon as a 3.5mm jack is connected, the external speakers are bypassed.