Tag: A/B Testing

Categories
Tools

Why Developers Should Embrace User Analytics

In our latest Developer Economics survey we were really surprised to see a dramatic fall in the level of adoption of User Analytics tools. A year ago they were the most popular category of third party tool with 38% of mobile developers telling us they used them. This time, only 21% of developers said the same thing. Why has there been such a significant drop? Are the hordes of new developers flocking to mobile just unaware of what’s available? Is all analytics being tainted by the spying scandals? Why does it matter?

user-analytics

The new mobile developer tsunami

At WWDC this June, Apple said that they’d gained almost 50% more developers in the last year. The vast majority of those will be iOS developers (rather than OS X and Safari). Android has been growing even faster in terms of new apps, so it’s likely there was even greater growth in the mobile developer population there. Windows Phone also saw significant growth in developer adoption. With so many new developers getting started building mobile apps we’d expect that they’re not all aware of the tools available to help. However, the reduction in user analytics adoption was greater than for other tools, indeed some tools categories saw increased adoption. So, although lack of awareness is clearly a contributing factor it’s not the whole story.

Tracking behaviours, not people

Information on government spying activities leaked by Edward Snowden has turned sentiment amongst many of the technically aware against digital surveillance. Perhaps some developers feel they shouldn’t be spying on their users’ actions? [tweetable]Does adding analytics to your app violate user privacy? Not usually.[/tweetable] User analytics tools are primarily for examining correlations between user behaviours and changes in the product or its marketing. If you make changes to a sign-up screen you need to see if it improves the fraction of users who sign up. If you’re trying to convert free users into paying users, it’s vital to analyse which changes in design or pricing result in greater revenues. Even to improve a paid up front or entirely free app, understanding what users do when they’re using it is incredibly valuable. As long as you’re not collecting personally identifying data and associating it with activity then it’s hard to see this as spying. Even segmenting users for behavioural targeting of functionality or advertising seems harmless as long as they’re anonymous. When analytics providers aggregate data across apps to build up user profiles for advertising, that may cross the creepy line for some. If this bothers you then choose an analytics provider appropriately, or collect your own data. If it doesn’t then having a privacy policy that clearly states who sees what data and what it’s used for is the responsible thing to do.

Coding blind

[tweetable]Updating and iterating on apps without analytics is apparently incredibly common[/tweetable]. It’s unlikely that all of these developers are regularly running user testing panels to get detailed feedback. Is everyone coding blind? To avoid constant app updates which would annoy users (and not even be possible with the review cycle on iOS) developers need to include multiple changes in every new version. If user sign-ups or revenue goes down after an update, how can you tell what went wrong? If things improve, which changes helped? Which parts of your app do people use the most? Are you working on improving features that no-one is using anyway? Without some kind of analytics, how do you find the answers to these questions? Might you be wasting a lot of effort?

Just integrating a third party analytics tool isn’t enough to answer these questions either. Apps need to be instrumented to log analytics events for every major action in the application and how long users spend on various activities. Beyond that, to really understand the effects of changes, users need to be analysed as cohorts – groups that started using the app around the same time. The value of an app to its users may increase or decrease over time and users who have got used to a UI are more likely to be unhappy with any changes. For an app that’s just starting to grow a user base, a change that causes it to lose 10% of existing users might still be positive if it increases revenues from new users. To know the real effects of some changes, it’s necessary to analyse specific sets of events as funnels. For example, in a registration or purchase process, increasing the number of people that start the process is no good if the number that end up completing it is reduced because they feel misled.

Tool choices

Developers that decide to embrace analytics have a lot of choices, however, the market is quite polarised. By far the most popular tools are Google Analytics and Flurry. These are (mostly) free. Google Analytics has the advantage with Android developers because it is integrated with Google Play, allowing direct tracking through the acquisition and download process into usage. It does have a (very high) limit on usage. If your app gets incredibly successful you’ll have to use the premium tier, which has a flat annual fee of $150k. Flurry on the other hand is free at every scale, although not as advanced as Google Analytics in terms of segmentation, funnels and visualisation of data. Both services are collecting data across apps, effectively to sell to advertisers.

There’s also a premium-only market with better support for cohort analysis, funnels, segmentation and visualisation. Paid offerings also link the analytics direct to actions, allowing automated real-time messaging to users, or the sending of push notifications to specific groups. Mixpanel and Localytics are two examples of tools in this space. These premium tools are also designed to be a lot more usable by non-technical staff, allowing a marketing team to analyse and react to analytics without always needing help from the development team. These tools have a fairly low user or datapoint count for a free tier and then start charging hundreds of dollars per month for successful apps. However, this means that they don’t sell the data from your app to anyone else.

A third option is the relatively little known Amazon Mobile Analytics solution. So far it’s relatively basic on the analysis front and doesn’t have the prettiest interface. However, you can collect 100 million events per month for free and beyond the free tier it’s just $1 per million events. By contrast, Mixpanel charge $2,000 per month for 20 million events. Amazon don’t share your data with anyone or report on it (no word on whether they use it internally for their own purposes).

Give it a try

Thinking about your apps as places to experiment and measure how the users react takes a change of mindset. If you’re not just building apps for fun then this shift can make a massive difference to your success.

If you’ve had a major change in results because of something you discovered using analytics, share your story with us in the comments.

Categories
Business Community Tips

Test Early, Test Often, Test on Everything?

Bugs Everywhere

Testing any mobile app presents a wide range of challenges. The often repeated but rarely followed software best practice of test early, test often is harder to adhere to than usual due to the fragmentation of the target environment and the relative maturity of tools. The increased acceptance of apps by mainstream consumers and intense competition have raised the bars for user experience and quality. There is more to test than ever, yet often very limited budget for doing so. Fortunately every challenge presents an opportunity and a vast array of tools vendors are racing to fill the gaps.

What to test?

Much of the traditional software testing literature focuses on various forms of functional testing – ensuring the system does what it’s meant to do. With a strong trend towards simpler, single purpose apps, this is often the easiest thing to verify in a mobile app project. There is now a much stronger focus on the user experience and this requires testing of an entirely different nature. The most effective way to test that an app is easy (or even fun) to use is to get feedback from real users. Doing that and finding major issues after the app has been built is a very expensive mistake to make, so most developers and designers will want to create mock-ups or prototypes for early feedback. There’s a wide range of tools to help with this task from simple wireframing through to full interactive prototyping. Given the importance of animations within mobile apps to enable users to discover interface interactions and learn to navigate, more complete prototypes are becoming increasingly desirable. As users become more sophisticated and specialist tools reduce the time and effort required to create interactive prototypes this trend is likely to continue.

With the majority of app store revenues coming through in-app purchases, another more specialized form of testing the design of an app is becoming increasingly important – split testing. On the desktop web, tools for trying out design and copy variants to optimize sites for specific user behaviours are very mature and the best of them can be used by staff with no development skills. In the mobile world most of the tools in this space are still very immature and developer-focussed. The responsive design trend on the web and the more restricted deployment options for native apps make this a more challenging problem for mobile devices but we expect the tools in this sector to mature rapidly.

[sectors slugs=’prototyping-mockup’]

When to test?

The earlier you find problems with software, the cheaper it is to fix them. As such, it makes sense to start testing as early as possible. How about testing the idea for the app via a mobile market research service before you even create your first wireframes? It’s worth considering – if you can’t generate interest in your app idea with a simple pitch it’s not going to be easy to get people to download it from the store either.

For most apps (particularly native apps) it’ll be worth using one of the mock-up or prototyping tools mentioned above and test the design before you start coding the real app. It’s much cheaper to iterate a simple design prototype than a native app. However, you’ll still want to try out the actual app with real users before you launch it. To help with that there’s a range of beta testing services that can help you distribute your beta app and find and/or manage testers. There are also services to help you get feedback from your users before and after the app launches. Providing a highly accessible feedback channel for users in the app is your best hope for preventing the inevitable disgruntled few from leaving bad reviews.

Ideally an app will be developed and tested iteratively with functional testing of new features and full regression tests for the existing functionality run for each iteration. This level of testing can get extremely expensive and time consuming unless it is automated. Fortunately there are several tools, open source frameworks and third party services that can help out there too.

[sectors slugs=’beta-testing,feedback-helpdesk,automated-app-testing’]

Where to test?

Another major problem for mobile developers is the scale and fragmentation of the market they’re trying to serve. Collecting a full library of test devices with major firmware variants is way beyond the budget of most developers, let alone the effort that would be required to test manually on all of them. Automated testing solutions can help here also and some services provide access to a large set of devices for remote testing too. However, it’s simply not feasible for most developers to test every version of their apps on all the device and firmware combinations they support. This limitation means some bugs are almost guaranteed to escape into the wild; the important thing then becomes how quickly you discover and fix them. For this reason, crash analytics and bug tracking tools are becoming increasingly important. Another useful weapon in this battle is your usage analytics data – it can enable you to focus testing on the devices which are most popular amongst your user base and also spot changes in use on particular device models that might signal a non-fatal error that’s causing users to abandon the app.

Finally, for some apps, where they are tested geographically may also be important. Do you know what the performance of your app is like for users who are far from your servers? If you use SMS, do you know how long it takes to get to users on different networks around the world (or if it even gets there). Have the localisations for your app been tested by native speakers? Our automated testing and app certification sectors include companies that can crowdsource beta testers or provide access software testing professionals almost anywhere in the world to help you scale globally without leaving your desk.

[sectors slugs=’crash-analytics-bug-tracking,user-analytics,automated-app-testing,app-certification’]

Categories
Business

Prototyping: needless effort or driver of perfection?

Mobile apps are becoming more and more sophisticated every day. They evolve together with mobile devices, giving us even more pleasant, intriguing and unique experiences. Design, usability, functionality accompanied by various touch interactions, animations, and transitions are integral components of apps.

Building an app is not easy. It involves various stages in a long development life cycle. Apps require time to build, time to test, and time to iterate for improvements. Iterations are not easy especially when extensive code changes are required and that’s where usually things get messy.

There is one solution to avoid the trap that lies ahead when developing apps: prototyping. A prototype is an early sample or model built to test a concept or process, or to act as an object to be replicated or learned from. The prototyping step is often skipped due to the extra cost and effort it adds to the lifecycle of a project. However, it is widely accepted in the development world today, that undermining this vital part of the design and build process may lead to miscommunication between developers and clients, pitfalls, over budgeting, and bad quality products. Prototyping allows developers to conduct proper user trials, iterate before coding and send the app to production only when it is perfected.

Graphic designers, user experience designers, usability experts, interaction designers, and developers use different ways and tools to create prototypes. The most popular methods used for prototyping apps are paper prototyping, presentation software, mobile apps (usually for tablets) designed to allow people to prototype mobile applications on actual mobile devices, source code and prototyping applications and tools either web (offered as SaaS) or desktop apps.

In the following section I will briefly describe the prototype tools vendor landscape and the main needs that these tools serve.

Vendor landscape

If we ignore for a moment the paper prototyping and source code, where no specific tool is used, then we are down to three main types of prototyping tools: presentation software, mobile apps and mobile prototyping applications.

prototyping-tools-landscape

Presentation software like Keynote or Powerpoint, are in the market for decades and peeple are well trained to use them in different ways. The way people use them for prototyping, is by linking sketches, or design comps together in a presentation since such tools support animations and screen transitions. Some UI libraries like Keynotopia have UI components of popular mobile operating systems like iOS and Android, designed for presentation software. Presentation software are built for an entire different purpose and they are limited in prototyping functionality but are still valid in some user cases. Diagramming software like Omingraffle and Gliffy are sometimes also used for prototyping.

Mobile apps for iPad and Android tablets designed for mobile prototyping allow real device testing which is their main advantage. Some of these apps provide UI libraries of major mobile UI components like App Cooker and Interface HD. Others, like Popapp allow taking photos of sketches and linking them together. Most of these apps are limited to single device prototyping and usually lack sharing and collaboration tools but are very useful when it comes to quickly validating an idea.

Mobile prototyping applications are web or desktop applications designed specifically for mobile prototyping. These applications vary from simple traditional wireframe applications (including mockups) to advanced prototyping tools that are able to provide a varying degree of mobile-specific functionality such as touch events and gestures, interactions, screen transitions. Most importantly these tools provide the ability to preview a prototype on the actual device.

There are three different types of such mobile prototyping applications:

  1. Hotspot apps are usually web apps that allow you to upload your mobile design comps and link them together usually with a single event (click or touch) without (or at best simple) transition effects. These apps are useful especially for collaboration as most of them allow comments and annotations. Although some of these tools make real device preview possible (i.e. preview on the actual device that the app is built for), they are not really eligible for proper user trials as they do not allow multiple interactions such as touch gestures and other important mobile specific features. Applications in this category are Fieldtest app, Invisionapp, Popap and are usually web apps offered as Software As A Service or mobile apps for tablets and Smartphones.
  2. Wireframe or Mockup tools are tools that allow the development of still wireframes or mockups. Usually these tools have a large number of UI components libraries available. Some of these tools have been in the market for years as they were designed for website mockups or wireframes, but many of them have been changed in an attempt to embrace needs specific to mobile apps. Many of these tools are very advanced in functionality and features, offering a range of useful companion tools for collaboration and more. Most of these tools are limited to single tap interaction (or mouse interactions, as they are designed for web sites) and no or limited animations and transitions. Tools that fall under this category include Balsamiq, MockingBird, UXpin, Pidoco and others and they can be found as web apps available on a subscription basis or as desktop apps.
  3. Prototyping tools are web applications or desktop software designed from the ground up and specifically for mobile (or web) prototyping. These applications go beyond traditional wireframe or mockup applications, to provide functionality for mobile touch events and gestures, interactions, screen transitions and most importantly provide the ability to preview a prototype on the actual device. Many of them come with UI libraries for iOS, Android, Windows mobile and Blackberry and offer collaboration tools and functionality. Tools and software in this category include Axure, Indigo Studio, Proto.io and many more.

[sectors slugs=’prototyping-mockup’]

Current challenges with prototyping tools

Prototyping tools are still in their infancy. They have been around for two years or less must be in sync with the ever changing mobile industry. New mobile devices become available on a daily basis, new versions of mobile operating systems and new functionality that needs to be supported makes the chase even more difficult. The main challenge is the ability to test the prototypes on the real device. In order to achieve this more tools render their prototypes in HTML5 so that they can run on a native mobile browser without the need of installing and maintaining various mobile apps.

Another major challenge is performance and by using HTML5 features such as animations render much slower on a mobile browser than they would on a native app, making the experience a bit far from real and as such defeating the purpose of doing a full prototype in the first place. Nevertheless, some of these tools have reached a maturity level that allow professionals to create fully functional, interactive mobile app prototypes of their apps that look and behave exactly as their app would. This allows the teams to conduct user trials, gather feedback, and iterate for improvements. Furthermore, a proper prototype narrows the communication gap between designers and developers (coders) as well as with the app team and the stakeholders.

Future opportunities with prototyping tools

Prototyping tools gain larger audiences as mobile technologies progress. As mobile apps become more sophisticated, more detailed prototyping is required. The mobile market grows and the prototyping tools market will continue to grow with it as more Operating Systems and newer more capable devices are released. New devices come to life every day, from car navigation systems to refrigerator panels, all having in common touch and interactive interfaces.

Categories
Platforms Tips

A/B Testing on the Amazon Appstore

App Store

In December Amazon launched a new A/B testing service for Android apps on the Amazon Appstore. Integrating A/B testing, particularly for in-app purchase related events, in the store portal is a welcome addition. Slightly disappointing considering this comes from a store provider is that the A/B testing service does not support testing different copy or icons on the storefront itself, purely in-app A/B tests, for which there are already third-party alternatives.

Not a “leaner” alternative store

When we looked at the developer platform “lean factor” for Android previously, we were only considering Google Play. Android came out on top due to the ability to publish new versions to the store almost immediately, enabling much faster iteration. However, the Amazon Appstore does have a review process which introduces significant publishing delay. As such even with the convenient addition of A/B testing baked into Amazon’s store, Google Play still offers a better environment for implementing lean development.

Testing in a smaller market

That said, there is at least one interesting use of this new service. Amazon’s Appstore has historically had a much higher proportion of users paying for downloads and in-app purchases than on Google Play – so much so that in early 2012 many developers made more money on Amazon’s store despite the much smaller audience. A smaller market with a high proportion of paying users is ideal for A/B testing improvements with the most important user group, particularly improvements to in-app purchase flows. Testing new variations on Amazon’s store and then rolling the successful ones out to a Google Play version might be a useful strategy for increasing revenues.

But how much smaller?

A final question is how much smaller is Amazon’s store in terms of downloads? They are notoriously silent when it comes to absolute numbers and even the relative numbers they give are ambiguous. In the press release for the A/B testing service linked above they say that app downloads have grown 500% over the previous year. This was in a release at the beginning of December with no fixed reference point given for the comparison. As can be seen from the relative growth chart produced by Distimo at the same time last year below, the launch of the original Kindle Fire at that time produced similarly massive growth.
Distimo_Amazon_AppStore
Dependending on where the comparison is made, a 500% increase could be anywhere from slightly behind January’s download level to 6 times the December 2011 peak. Until they provide some concrete numbers the best advice is to try it and see what your own results are like. The only major difference we’re certain of between the app buying users on the two stores is a higher proportion of tablet users at Amazon’s.