How We Test Tablets
We test hundreds of tablets in the PC Labs. Here's how we put them through their paces to determine the top performers.
By Sascha Segan and Joel Santo Domingo
June 6, 2014 2:17PM EST
Tablets straddle the line between mobile and home computing. For many people, they're becoming primary computing devices; for others, they're entertainment gadgets to keep the kids quiet in the car. Since we test dozens of tablets on every operating system available in the U.S. each year, we have a few different approaches to tablet testing.
Most notably, we divide our tablets by operating system. Android tablets, Apple iOS tablets, Windows RT tablets, and Windows 8.1 tablets all get somewhat different testing procedures, as all the testing applications and benchmarks won't run on every device. Most notably, Windows 8.1 tablets get much more comprehensive tests of productivity application performance. Here's how we test each type of tablet.
We test Android tablets very similarly to Android smartphones. First, we analyze physical design including display size and quality, controls, ports, and storage.
We run Browsermark and Sunspider against the Chrome browser to check Web browsing speed. We run Antutu (shown at left) and Geekbench to take a look at processor performance. GFXBench and Nenamark look at 2D and 3D graphics performance, and Basemark OS judges the fluidity of the user interface. We also launch and play high-end games (currently, Asphalt 8 and Need for Speed: Most Wanted) to check frame rate, control fluidity, and jitter.
We have a test suite of music files in MP3, AAC, OGG, WAV, and FLAC formats at various bit rates. We play the music through the phone's built-in speaker and both wired and Bluetooth stereo headphones, if possible. Our video test suite consists of MPEG4 and H.264 videos in resolutions up to 1080p.
If the tablet has Wi-Fi, we try to connect to open and WPA2-protected networks. If it specifically promotes its Wi-Fi speed, we check speed using speedtest.net at distances of zero to 100 feet, every 25 feet, using a Meraki router. We check LTE speed by using Speedtest.net at multiple locations outdoors on the street in midtown Manhattan, New York.
If a tablet has GPS, we see if it can capture a GPS signal and show an accurate location both on a busy, crowded street and in an open park.
To test camera capabilities, we use an abbreviated set of our tests for digital cameras. For battery life, we use a rundown test where we play a 12-hour-long video on a loop with the tablet's screen set to maximum brightness and Wi-Fi switched on, until the battery dies.
Tablets with pressure-sensitive pens need specific tests of pen quality. We draw grids of crosshatched lines and grids of stippled dots with the pens, looking for gaps in the lines, uneven pressure sensitivity, or missing dots.
Apple iPads are compared extensively with the previous models of iPad, rather than to other tablets or phones. We start with roughly the same tests we use for Android tablets, above. For benchmarks on iOS, we run Browsermark and Sunspider to test Safari browser performance, Geekbench to check the processor speed and GFXBenchmark to look at 2D and 3D graphics performance.
For media, we play AAC and ALC music transferred from a Mac via iTunes, as well as both a transcoded 1080p H.264 video file and an HD TV show purchased from iTunes.
As each iPad typically comes with new, exclusive apps that show off that year's processor, we test the performance of both that year's featured app and previous years' featured apps on new and previous iPads, including launch time, frame rate, and UI smoothness. To further test the processor, we measure the time it takes to perform large file exports in iMovie on both the current and previous iPads.
We also test Wi-Fi reception and performance, checking it at 25-foot intervals from a Meraki router against previous iPads and other Editors' Choice tablets.
We test Windows tablets in a similar way to how we test laptops, beginning with PCMark 8. It is an overall performance test that supports the Windows 7 and 8 operating systems only. The PCMark 8 score is derived from a compilation of common tasks that users might perform in a typical day. These include photo and 4k UHD video editing, music, gaming, communications, and productivity tasks. We use the Creative test with OpenCL acceleration, which mixes everyday tasks with media and entertainment content creation. PCMark works in both 32-bit and 64-bit environments.
Next, we use a video-based battery rundown test, which supports all operating systems, including Windows, Chrome OS, Linux, Android, and OS X. To this end, we play a 24-hour long MP4 video file compatible with a wide range of mobile devices. We start the test, and run it for as long as a fully charged battery lasts, usually to the point that the system hibernates at under 5 percent battery capacity.
Before we start the test, we crank down the brightness of the screen to 50 percent and disable adaptive screen brightness. Wireless radios are left on, but unconnected to access points. A system qualifies for the "all day computing" designation if it lasts more than eight hours on our battery rundown test.
If the tablet has an internal and an external battery pack or a keyboard dock with a built-in battery, we test both batteries. We will test with the internal battery alone, and then with the internal and external batteries charged and connected to the tablet. These results are reported in our reviews and in our benchmark tables.
For 3D testing, we want to use tests that are challenging to each system. To this end, we chose Unigine's Heaven and Valley tests. Each test simulates movement through highly detailed environments. Heaven navigates a virtual manmade location, while Valley flies through an expansive outdoor landscape. Both tests add atmospherics to each environment, and can stress even multi-GPU gaming rigs.
At the medium settings, we use 1,366 by 768 resolution and set all quality sliders to medium or normal. AA (anti-aliasing) and AF (anisotropic filtering) are disabled, in order to give us an idea of how a system plays games when frame rate is more important to the player than ultimate content quality. For the maximum settings, we set the screen resolution to native (aka max non-scaled resolution for that tablet), and bump the quality settings to extreme or ultra. We choose the setting that is the most strenuous. AA and AF (if present) are set to 4X each. This shows us what happens when the player wants an immersive gaming session with the most eye candy turned on.
Finally, we use an industry-accepted synthetic 3D benchmark that that truly taxes the graphics subsystem. FutureMark's 3DMark, another DirectX benchmark, has built-in Android, iOS, Windows 7, RT, and 8 support. 3DMark has three built-in tests, one made for mobile devices (Ice Storm), one for mainstream systems (Cloud Gate), and one test that truly taxes gaming rigs (Fire Strike). We use Fire Strike in order to put an emphasis on high-end 3D animation, like the stresses that an AAA game would put on your tablet.
Cinebench R15 is a multi-CPU rendering tool that addresses multi-core processing, multi-threading, and OpenGL elements (on the graphics side). CineBench has the benefit of running on 64-bit versions of Windows 7, Windows 8, as well as Mac OS X. We run the CPU test, which measures 3D rendering in software. This test shows the added benefit of multi-threaded multi-core computing.
For video encoding, we're using Handbrake 0.9.9 to transcode a 1080p file ripped from a Blu-ray disc down to a preset suitable for viewing on mobile devices like the iPod touch and the iPhone. Transcoding a video file will save room by shrinking the video file to a size that better fits on your phone's limited internal storage. The test is timed, and faster is better. Handbrake runs on all current flavors of Windows and OS X.
Finally, our Adobe Photoshop CS6 test involves running a 12MP JPEG through a series of 11 filters. The system is timed to see how long each filter takes to manipulate the image, and the total time is compared with times from similarly equipped tablets. The Photoshop CS6 test runs on all x86-compatible versions of Windows and OS X.
For more, see our lists of the 10 Best Tablets, the Best Android Tablets, and our favorite Windows Tablets, as well as our Tablet Product Guide, which is packed with full reviews and tips of all the latest models. Stumped on how to figure out what's best for you? Read Android, Apple, or Windows: How to Choose the Right Tablet.
Get $50 Off a Dell Venue 8 Tablet
How Bad Is it to Use Tech as a Babysitter?
Sascha Segan Lead Analyst, Mobile
PCMag.com's lead mobile analyst, Sascha Segan, has reviewed hundreds of smartphones, tablets and other gadgets in more than 13 years with PCMag. He's the head of our Fastest Mobile Networks project, hosts our One Cool Thing daily Web show, and writes opinions on tech and society. Segan is also a multiple award-winning travel writer. Other than ... See Full Bio
More From Sascha
T-Mobile 5G Is Live in Las Vegas, But Only in a Few Desolate Corners
T-Mobile, Qualcomm Get Closer to Nationwide 5G
In Las Vegas, AT&T Pulls Back the Curtain (Slightly) on Its 5G Strategy
We Visited the First 5G Basketball Arena
Qualcomm's New Cheap Chip Is for Smartphones, Not Flip Phones
Joel Santo Domingo Lead Analyst
Joel Santo Domingo joined PC Magazine in 2000, after 7 years of IT work for companies large and small. His background includes managing mobile, desktop and network infrastructure on both the Macintosh and Windows platforms. Joel is proof that you can escape the retail grind: he wore a yellow polo shirt early in his tech career. Along the way Joel e... See Full Bio
More From Joel
How We Test Desktop PCs
How We Test Laptops
Alienware Area-51 Threadripper Edition
Google Pixelbook
Panasonic Toughbook 33
How to Set Up Your New iPad or iPhone
Save 50 Percent on 8.4-Inch Dell Venue 8 7000 Android Tablet
Dell's Extended Presidents' Day Sale: Save Big on Tablets, HDTVs, More
Family Road Trip Tech Checklist
How Samsung's Galaxy View Could Usher in the Era of Mobile TV