My setup and tips for cross-browser testing

Cross-browser testing can be a pain. Here, I'll try to list some (if maybe not all) of the tools I use to test on as many OS and browser engines and versions as possible.

My main weapon of choice for running other operating systems is Virtualbox. It’s a virtualization software that’s free, Open Source and pretty reliable. It allows you to run a number of "guest" operating systems virtually inside your primary operating system (the "host"). I work on a Mac, and Virtualbox allows me to run various versions of Windows, Linux, Android and even Mac OS itself.

One reason I stick to Virtualbox is that it plays nice with some other tools I use that rely on virtualization software:

  • It works with Vagrant, which is a piece of software that allows you to manage development environments inside virtual machines (more on that in a separate post, perhaps).
  • It works with Genymotion, which is a virtualization system specifically for the Android OS.

Besides, it’s available on Mac, Windows and Linux, so it’s potentially more portable.

Tips and tricks for using virtual machines

As soon as I install a virtual machine, there are some things that I set up.

  1. If Virtualbox Guest Additions are available, I install them. This is a software package that installs extra features specific for the guest OS in question, that eases communication between the Guest and the Host. Stuff like automatically resetting the screen resolution inside the Guest when you resize the virtualbox window inside the Host, automatically releasing the mouse cursor from the Virtual machine when you move the mouse to the edge of the Guest window etc.
  2. Enable bidirectional clipboard—makes it a lot easier to copy and paste URL:s across machines. This is done via the Devices menu, then Clipboard — Bidirectional, or the settings for the virtual machine in the Virtualbox main window.
  3. Set up the Swedish keyboard & Swedish language support—insert your language here.
  4. Install any other browsers and tools that I want available (on Windows, I probably install Firefox & Chrome, at least).
  5. Open a window for each browser, go to "about:blank".
  6. On Windows, I put the IE window in front.
  7. Take a snapshot.

Using snapshots

When you have a virtual machine, you can save the exact state of that machine to disk, known as a snapshot. This means you don’t have to boot the machine from scratch inside Virtualbox each time you want to test something. Booting it from a snapshot takes about 10-15 seconds.

Snapshots mean you could potentially have several different versions of a browser inside a single machine—you’d just install the first version, take a snapshot, upgrade to the next, take a snapshot etc etc. I don’t use this technique a lot: disk space is cheap, and having separate vitual machines for separate IE versions (specifically) makes it possible to run several of them side by side (even if it does slow my machine down quite a bit). It’s also a bit fiddly to switch between snapshots in the Virtualbox UI.

The one thing I do use it for sometimes is to have a separate snapshot of a browser + screen reader software activated.

Windows testing

I use the free virtual machines provided by the modern.ie project from Microsoft: they offer virtual machine disk images of Windows XP, Windows Vista, Windows 7, Windows 8 and Windows 8.1, combined with Internet Explorer 6-11 preconfigured for testing (meaning that the F12 developer tools are installed on the versions that dont’t include them by default). They are fully functional but only valid for 30 days: after that time has expired, the machine will automatically shut down after an hour of use. You can "rearm" them, meaning you get an extra 30 days, by running the command slmgr -rearm in the Windows command prompt. This works 2 times per machine, after that you’ll have to put up with the constant nagging that you’re running an unverified version + the time limit , or reinstall the OS. More about that on Rey Bango’s blog.

Currently, I have IE8 installed on XP, IE9-11 on Windows 7, IE10 Windows 8 and IE11 on Windows 8.1. Like I said, space is cheap so I use separate virtual machines for this.

Linux testing

Not much to say, apart from that I run Ubuntu Linux & test with Firefox and Chromium (Chrome itself isn’t available on Linux, I believe) Update: apparently Chrome is available, but you have to tell Ubuntu where to find it—thanks to Peter Chamberin for pointing this out.

Just download a disk image and install as a virtual machine inside Virtualbox.

Android testing via Genymotion

I stumbled over the Genymotion project recently, which allows you to run Android virtual machines on regular x86 computers. It’s free for non-commercial use, and I’m guessing mostly aimed at Android app developers, so I’m not sure if straight up browser testing counts as commercial usage. The "indie" plan is 99€ however, and could be well worth the money.

The app acts as a shell that downloads and launches virtual Android devices: they have a long list of OS versions and emulated specific devices: you just pick one and it downloads the disk image automatically and installs it behind the scenes as a virtual machine inside Virtualbox. Easy as pie!

As with any device emulation, it’s not the real thing: make sure you test on real devices! Find your nearest Open Device Lab before take-off.

Using the iOS simulator from XCode to test iOS on Mac

You can do some quick iOS emulation on a Mac by using the built-in simulator that comes with XCode (the mac developer tools package that you can install for free from the App Store). The iOS simulator is available when you start a new iOS app project, but not too long ago I found that you can launch the simulator as a standalone app. It’s not visible from the Finder though, so you need to find it from a terminal first — run this command in a terminal window: open /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Applications, and then you can drag the "iOS Simulator" app to the dock or finder sidebar. It seems to disappear from time to time though: probably when XCode updates, for some reason.

Running older versions of Mac OS to test on Safari

Much like the case with IE on Windows, Safari on the Mac is hard to test because it’s tied down to the OS version in some ways: you can’t downgrade it. There is the Multi Safari project, but I’ve found that not many of the Safari binaries there work on Mavericks. However, if you’re running a Mac yourself, you can download and install a virtual copy of older versions inside Virtualbox, which has experimental support for this. Licence-wise, I guess this is a bit of a gray area: first, Apple explicitly disallows you to run their OS on non-apple hardware (which hasn’t stopped the multiple Hackint0sh projects, but that’s beside the point). However, you are allowed to run your legal copy of their OS on 5 machines, I believe, and a virtual machine shoud be no different. Third, it’s an older version of a (now) free OS that we’re talking about, so I wouldn’t fuss too much about it.

I’m running Mountain Lion to be able to test Safari 6. It does require some trickery to get it to work, but here’s how you’d do it (again: this only works on a Mac). These instructions are based on this guide from Robert Setiadi so all the credit goes to him — check his guide if you need more detailed instructions.

  1. Open App Store, go to your purchases. If you’ve previously had Mountain Lion installed via the App Store, you should see it listed, along with a "Download" button. If you’re on Mountain Lion and want to download Lion, I believe you have to jump through some additional hoops to be able to download it (alt-click the "Install" button? Something like that.)
  2. You’ll download the "Install Mountain Lion.app" to your application folder. Leave it there for now.
  3. Next, you’ll need to create a blank virtual machine of type "OS X" inside Virtualbox. Click "New", choose type "Mac OS X", version "Mac OS X 64-bit". Just follow the instuctions in the wizard to the end.
  4. After creating the blank virtual machine, go in to the system tab in its settings and select chipset type PIIX and make sure EFI is enabled.
  5. Next, you need to tweak some settings in the disk image you downloaded from the App Store. This is done via a utility called iesd, which is a small Ruby program that you install as a so-called "gem" on your system (a gem is basically a ruby package or module). Ruby is a programming language that should be preinstalled on Macs, and the commands for installing gems with it. Open up a terminal window (you’ll find the Terminal program inside the "Utilities" folder inside "Applications") and type gem install iesd. If it complains about permissions, try sudo gem install iesd, which runs the command with superuser privileges. In that case, you’ll need to enter your password.
  6. Now you need to download a small file that replaces a system extension: you’ll find it from Robert Setiadi’s site.
  7. Finally, you need to run the iesd command that replaces the original file in the disk image you downloaded from the App Store with the one you just downloaded. Note that you’ll need to replace the file paths in the command so they match your system. Here’s how it looked when I ran it inside the folder where I both had the downloaded .kext-file and wanted the generated disk image to end up: iesd -i /Applications/Install OS X Mountain Lion.app -o MountainLion.dmg --install-extension ’CPUPowerManagement.kext. Same thing here: depending on your file permissions, you might need to put sudo before the command. This is basically the only thing I’ve changed from the original instruction, the flag --install-extension was previously called --install-kexts, it seems.
  8. You have now created a new disk image that you can use to install Mountain Lion inside the blank virtual machine. Start the virtual machine and choose the newly created .dmg file as your startup disk, and continue to install Mountain Lion.

As these things go, instructions tend to miss something or become outdated, mileage may vary yada yada. Check the previously mentioned guide for any updates, or use your favourite search engine to find help.

Testing with screen readers

I try to test on VoiceOver in Mac now and again, which is built in. One tip is that if you’re not working with English language websites, you can install additional language voices for free by going into the speech options in the VoiceOver Utility (in System Settings — Universal Access) and choosing "Customize" in the drop-down for which voice to use. Check the languages/voices you need and you’ll be asked if you want to download them.

There’s also ChromeVox which is a free plugin for Google Chrome on both Windows & Mac, that acts as a screen reader specifically for Chrome. It’s not very popular among blind people, so best just use it just as a quick litmus-test for how your page would be spoken.

For Firefox, there’s a plugin called Screen reader simulator that allows you to see text output of what a screen reader would read when encountering your page—quite handy.

Inside your Windows virtual machines, you can install NVDA, which is a popular and free screen-reader. The other popular but commercial alternatives (JAWS and WindowEyes, for example) have demo versions that you can install as well - if you want to be really sneaky, you can resort to using a snapshot of a time-limited demo that you just reset again and again...

That’s the low-down on the various tools I use for testing in various browser combinations. As I said previously, when it comes to devices like phones or tablets that have radically different hardware, you’d be better off testing on real devices. I’m very lucky that I have an Open Device Lab, MODL, right in the middle of our office.

Hit me up on Twitter if you have any additional advice or comments!