This week we asked the Mobile Team:
What best practices do you follow to make sure your apps have been thoroughly tested before release?
|It’s important to have a structured test plan in place that you repeat when releasing. This will ensure a consistent level of testing at the least. In addition it’s important to learn from past mistakes, update your test plan to cover scenarios that have caused problems in released apps. It is also key to have someone who is dedicated to QA testing the app, rather than the developers themselves.
Developers know too much about how the app was written and how they expect it to be used. Get some external people to use it the way they expect it to work and this will expose issues guaranteed, beta testing is a good way to exercise the app in ways the developers never imagined!
|During the development process, the best testing methodology I have seen is a mix of science and art – meaning automated error checking as well as a having real beta users. Automated testbeds are great and help to surface bugs and coding errors and that is essential for a solid release. Equally important is having internal and external beta testers beating on the app to find UX bugs and feature issues that can only be found by human interaction.
|Even though we’re a small team at Realmac, we have someone in-house dedicated to QA for all our products. We find this catches most major issues before any app ships. Having someone on the team testing and looking for bugs is invaluable and more small teams should do it.
Along with doing QA in-house we also find twitter to be a great place to recruit beta testers. Having users that are genuinely interested in using your app always results in much better feedback.
|The best practice that I’ve identified is to have the best people. Smart team members who care about an app and are invested in its success will always produce quality releases. To catch edge cases we use a continuous integration system to generate builds after every commit. It is also responsible for running an initial pass on our apps to test core functionality.
Before any app is submitted to the store we install and test every build configuration (including an Ad Hoc version). That Ad Hoc version is then re-signed and submitted directly to the store.
|Remember that the point of testing is not to prove that something does work, but to disprove that something does not work; not to break the app, but to break the illusion that the app is not broken; not to coddle to codebase, but to treat the codebase as if it were written by your worst enemy.
|Start early and make sure you have a clear and well thought-out testing process. User testing is also essential – make sure you pick a good set of testers outside of those actually working on the app who can thoroughly test. They will be able to provide you feedback and criticism on your app from angles that your team may not have even thought of in the first place. It’s always eye-opening to hear how others view things and you can never have enough.
|The problem most developers and testers face when running quality assurance test on their apps is that they tend to naturally follow the golden path. This is the path that they intend the user to take or that they are use to following themselves. Users, if anything, are unpredictable and will hit all those edge cases that never show up. One of the most useful tools to combat this problem is Mind Maps.
A Mind Map by Nick Arnott for iOS can be found here and one for OS X by Cory Bohon can be found here. Following these maps or making your own will help prevent you from overlooking functionality you don’t normally use.
|This is really simple. Throw away your code coverage. Toss the automated test suites. Then bring in real customers.
Use craigslist or your favorite testing service to get customers using the product. Not one or two, but dozens. Set up a station in a mall and pay $10 for passers-by to try it on their own phones. And here’s what “try it” means: you tell them the name of the product and nothing else, then you watch. You don’t answer questions or touch the phone.
When they’re done with the experience, ask them if they’re going to be using the app in a week. If most say yes, then your app’s quality is good enough to ship. You see, the only point of testing is to make a great experience for your users. Divorcing QA from UX is just pointless segmentation.
One more thing: if you’re doing this exercise for the first time when you think the product is “done,” you’re probably in for a surprise.
|When it comes to testing, there’s no way to start too early. It’s amazing how much information you can get from users with mockups or prototypes. User testing before full designs are done can lead to a much better app. This is especially informative if it is your first time making an app. And during the process of actually developing the app, user testing can help validate the decisions that have been made.
|The best practice is to have a process. It’s amazing how many developers don’t have a clear process for how their apps should be tested. Defining a QA process and ensuring that apps must pass it before ever reaching a customers hands will catch a huge amount of potential issues.
Our process includes a continuous integration (CI) server which automates building and testing every time a developer changes any piece of code. Only when an app passes all tests and “integrates” successfully will it be allowed to the next phase. From here, the CI server will automatically create beta builds of the app and make it available to all relevant parties. The only way for an app to go from a developers desktop to a testers hands is via our CI environment.
From there our QA department and any other internal or external testers will be able to download and test the app. Our QA team have a thorough list of issues, edge cases and functionality that must be tested. Issues found here are fed back to developers and the cycle begins again. Only when an app makes it through the manual QA process will it be made available for release or submission to the relevant app store.
Do you have any special strategies to make sure your app has been thoroughly tested? Share your questions and comments below or by using #MobileTeam on Twitter.