Testing mobile apps on different OS versions

One of the biggest obstacles in mobile app testing is the wide range of OS versions available. As of 2022, Android has the largest market share globally at over 70%, followed by iOS at around 25%. But both operating systems have highly fragmented version distribution:

• Android - Only about 15% of Android devices are running the latest major version of Android 12 as of May 2022. The rest remain fragmented across older versions like Android 11, 10, 9, 8, and 7.
• iOS - While iOS enjoys faster adoption of latest versions, still around 10% of devices use iOS 13 and earlier.

This fragmentation means developers must test against a matrix of OS variants to account for app behavior differences in everything from UI layout, API compatibility, and performance.

While both are mobile operating systems, iOS and Android have key technical and philosophical differences that impact how you test apps on them:

• Hardware diversity - iOS only runs on Apple devices, while Android has thousands of device types. This means way more hardware-specific bugs to catch on Android.
• OS adoption rates - iOS generally sees faster updates to new OS versions than Android. You can drop iOS 13 support more quickly.
• UI testing - The iOS UI framework is more consistent across versions, while Android UI varied more between OS releases.
• API levels - Android apps specify API level compatibility, while iOS versions have fixed sets of API changes.
• App distribution - The Google Play Store and iOS App Store have different review and publishing processes to factor in.

Understanding these key differences allows you to optimize testing efforts by platform.

A crucial early step is defining the mobile OS versions you will officially support. With limited QA resources, you can’t test every OS variant from the past decade.

For each platform, you should:

• Minimum OS version - The oldest OS release your app will support based on your audience. Balance features with maintaining a large enough user base.
• Target OS version(s) - The OS version(s) you will actively test and optimize the app experience for. Usually 1-2 recent versions.
• Unsupported versions - Older OS releases you will not test on or address issues for. These should not be marketed as supported.

Document these as you would browser or database compatibility. They will guide your testing roadmap and feature development.

To thoroughly test iOS and Android apps, you need a representative set of physical test devices covering different OS versions. Here are some tips on selecting test devices:

• Focus on your target OS versions - Purchase or acquire devices running the 1 or 2 main OS versions you will support. Virtual devices can fill gaps.
• Span mainstream device types - Have phones and tablets from Apple, Samsung, Google, etc. running your target OS.
• Include older devices - Get hands on at least 1 older device with your minimum supported OS to test performance.
• Factor in device memory and specs - Low RAM devices often surface different issues compared to high-end devices.
• Buy used devices - Purchase used devices in good condition rather than new to save budget.

Document your test lab inventory and ensure you can easily restore devices to factory settings for consistent testing.

While physical devices are essential for final validation, you can supplement your test coverage using:

• Android Virtual Devices - Create emulated Android device profiles in the Android SDK to test many OS versions.
• iOS Simulators - Xcode’s iOS Simulator can emulate iPhones, iPads, and app sizes running different iOS versions.
• Third-party emulators - Services like Sauce Labs offer access to hundreds of virtual Android and iOS devices for app testing.
• BrowserStack - Lets you test real iOS and Android devices hosted remotely accessible via browser.

These provide more flexibility but lack the fidelity of real devices. Use them for basic functionality and regressions testing across many configurations.

Automated UI testing is key to catching visual and functional regressions across OS versions efficiently. Consider these tips for building effective automated tests:

• Start early - Build tests as you build features to prevent gaps in test coverage.
• Prioritize flows - Focus automated tests on critical app flows and integrations vs. edge cases.
• Separate test code - Put test code in separate UI test target files from app code for modularity.
• Utilize Page Object Pattern - Create Page Object classes to represent screens to abstract UI details.
• Take screenshots - Capture screenshots of test runs to visually catch regression errors.
• Run tests on commits/nightly - Run automated test suites on version control commits and nightly to catch issues early.
• Parallelize test execution - Leverage CI/CD pipelines tools like Jenkins to run tests in parallel across multiple devices and emulators.

Taking this test automation approach will allow thorough but efficient testing across many permutations of devices and OS versions.

While testing flows should cover functionality across the app, here are some key areas to focus test coverage on when validating each new OS version:

iOS Version Testing

• UI layout and text - Validate screens and UI controls are visually rendered correctly on new iOS sizes and resolutions.
• New API changes - Check any deprecated APIs are removed and new iOS version APIs integrate correctly.
• Permissions - Confirm permission request flows and error handling align with any iOS permission changes.
• First-party apps - Test integration points with updated Apple apps like Safari, Maps, and Siri.
• Performance - Compare app speed on the latest iOS vs older devices to catch any performance regressions.

Android Version Testing

• API Level compatibility - Validate the app runs properly within the expected API levels.
• Screen sizes and ratios - Check layouts adapt well to new popular Android device sizes and aspect ratios.
• OS UI patterns - If Android introduces new navigational patterns, ensure the app adapts accordingly.
• Performance - Regression test rendering performance and network usage on the latest Android system updates.
• Play Store compatibility - Test installs, upgrades, and purchases end-to-end within new Play Store app releases.

Bugs are inevitable even with robust testing. To effectively prioritize and resolve cross-OS issues:

• Reproduce reliably - Ensure the defect can be reproduced consistently on the specific OS version and devices.
• Isolate within OS - Determine if the bug only occurs on one iOS or Android version versus all versions.
• Check compatibility - For functionality issues, check the OS version against deprecated APIs or required API levels.
• Consider workarounds - For minor visual defects on older OS versions, assess if a workaround is viable vs. a true fix.
• Estimate based on impact - Prioritize fixes by the number of your users impacted by the OS/version.

Having clear bug reproduction steps and tracking this info helps triage the severity and priority of cross-OS issues.

New annual OS updates brings extra testing:

• Install early betas - Test your app on initial developer betas of iOS and Android to get a head start on issues.
• Model usage on physical devices - Once stable public betas are available, install them on real test devices and test daily use flows.
• Focus on deprecated APIs - Use API scanning tools to see if any key app integrations will be deprecated or changed in the new OS version.
• Plan for mandatory updates - Once the new OS version is live, assume your users will update and test any remediation and upgrades required for compatibility.

Getting a jumpstart on OS updates will allow you to have updated, compatible app versions ready near the time most of your users migrate.

Here are some best practices to optimize testing mobile apps across many iOS and Android OS versions:

• Automate ruthlessly - Automated testing is required to cost-effectively scale across OS versions.
• Build platform-specific tests - Separate and parallelize iOS and Android tests configurations to maximize test execution.
• Tracker app version coverage - Log OS, device, and app version details for each test run to ensure coverage.
• Shift left on test writing - Prioritize writing test cases earlier in sprints/milestones to allow for iteration.
• Monitor real device usage analytics - Use App Store and Play Store analytics to guide what OS versions to focus testing on based on your live audience.
• Test on the cloud - Leverage cloud testing services to test on hundreds of real device/OS configurations.
• Don't ignore accessibility - Validate accessibility and screen reader support during cross-OS testing.

With a well-planned and optimized test strategy, you can release mobile apps with confidence they deliver an excellent experience across the myriad of modern mobile operating system environments. Proper test coverage is an investment that pays off in happy users and minimal post-release issues down the road.