There’s sort of this saying in the UX disciplines, that you never really know what you will learn when you do usability testing with real humans, but you will always learn something new every time you actually put the thing you’re working on in someone else’s hands. I’ve been involved in a fair bit of usability testing in my days, both as a moderator and observer, or hacking together prototypes to be used in testing etc. I can attest that yes, you do learn something every time, and you should try it. What I’ve found since I’ve gotten involved in running an open device lab is that pretty much the same thing is true for doing varied device testing: you never know before you actually pick up a device exactly what you’re going to learn about the design or performance of the thing you’re working on beforehand.

Even if you’re testing a specific thing, there’s always something else that catches your eye, making you go "whoa", either just from not being used to it looking or behaving that way, or just feeling different because of the form factor, or whatever.

I’m working on a site at the moment, where I’d used some javascript to handle toggling a search-box and and the global menu in the header. I figured it was straightforward enough—click a button, something folds out. The menu was absolutely positioned, overlaying the following part of the page, but the search box pushed the rest of the page content down, expanding in place.

Testing the site on slower iOS devices, I found that the performance of the animation was abysmal. A framerate of maybe 2 or 3 when revealing the search box. "Ah", I thought, "I might need to activate the hardware acceleration with one of those 3D-CSS jiggery-pokery-rules". So I tried it, and it was even worse: the page below composited into chunks that moved at different speeds, at about the same frame rate as before, making everything glitchy and horrible. The fascinating thing was that the same component had no problems with performance even on the oldest and weakest of our Android devices.

Whilst I haven’t found a very good solution to that specific problem (other than letting users with old iOS devices get a very janky experience), the moral of the story is that this was not the kind of problem I expected: in hindsight, I think I understand it, but up front, I didn’t think it would be an issue. So just as with testing with users, there is immense value in testing on different devices, even if you don’t expect trouble.