Now that I have a lovely ITWriting.com App I thought I should check out whether it is ready to fly.
I therefore ran the App Certification Kit that installs with Visual Studio 2012.
The tool asks you to select an installed app and then exercises it. I saw my app open, though I did not see it get beyond the first screen.
Eventually – bad news:
However, there is only one thing wrong with it:
Yes, the version installed is the debug build. I can fix this simply by rebuilding in release configuration.
What does the Kit test? Here is the list:
- Crashes and hangs test
- App manifest compliance test
- Windows security features test
- Supported API test
- Performance test
- App manifest resources test
- Debug configuration test
- File encoding
- Direct3D feature level support
- App Capabilities test
- Windows Runtime metadata validation
That sounds most impressive and makes a great list for you to show to your customer.
I am sceptical though. If the app was not exercised beyond the opening screen, might it not be a bit buggy after all?
I inserted the following line of code into the the Click event handler for reading a blog:
int iCrash = 1 / string.Empty.Length;
I then rebuilt the app in release mode and ran the App Certification Test. Great news!
and specifically:
Thanks though to my umm, bug-unfix, the app crashes whenever I click to read a blog.
I mention this not to poke fun at the App Certification Kit, but to observe that it does not do a good job of automatically detecting crashes and hangs.
The implication is that the human testers are the ones who will do this before an app is admitted to the store. I think they would find my obvious bug; but how much time will they have to test every feature of an app?
I think a proof that an app does not crash or hang is essentially a version of the halting problem, and as such I did not expect this little tool to solve it.
I submitted an app last week with a rather obvious bug in it, and indeed the testers from Microsoft found it and added a short description to the certification report.
I found that very helpful.
@Martin agreed; I am not sure why it claims to test it.
Shatters your faith in the App Certification Kit when it proclaims to have done something it hasn’t. Reminds of an incident a few years ago when I used to run a Porsche. For sevice I always took it to the authorized Porsche dealer because they provided a lengthy itemized checklist of work completed. This gave me faith in their thoroughness. Until on collecting the car from one service I noticed the ashtray hadn’t been emptied, although it had been checked on the list as completed. Shattered my faith in the Porsche dealer and I sold the car. So app test or car service either way “don’t say it’s done if it isn’t”.