Testing a mobile app is challenging, thanks to operating system fragmentation combined with diversity of hardware. In April 2013 Xamarin acquired a company called LessPainful, specialists in functional testing for mobile apps, which had created a mobile app testing tool called Calabash. Calabash is based on Cucumber, and lets you define test steps and then combine them into natural language tests. LessPainful also had a cloud testing service which let you run tests on remote physical devices and see visual test reports.

Eighteen months on, Xamarin has now gone live with Test Cloud, and has announced some big names which it says are using the service, including Dropbox, Flipboard and eBay.

There are currently 1036 devices (the number changes regularly) in the Test Cloud, including 273 iOS and 763 Android (Windows Phone is not supported, but Amazon’s Fire Phone and Kindle Fire does appear in the list).

You write your tests either in Calabash or in C#, upload your app and the tests to Test Cloud, wait a while, and then get notification that the tests are done and a report ready to view.

You can simulate events such as changes in location, device rotation, network dropouts, and of course user interactions like taps and gestures. You get screenshots and performance data (memory and CPU usage) for each test step.

You can also integrate with CI (Continuous Integration) systems like TFS, Jenkins and TeamCity to automate testing.

Writing and maintaining tests is hard work, of course, but for businesses that can afford the investment in both time and money, Test Cloud is likely to be a great improvement on manually gathering up as many devices as you can find and installing your app on all of them.

The cost is significant though, starting at $1000 per month for up to 2 apps and 200 device hours. You have to pay annually too, so it looks like a strategy of just buying one month towards the end of your development cycle will not work.

That said, I have been told that Xamarin will be coming out with an Indie version in the future that has a lower price.