How we test Web browsers

The browser is ubiquitous, contentious, and the one app that everybody uses on every kind of hardware imaginable. Here's how we gauge performance.

Microsoft's FishIE Tank benchmark test.

(Credit: Screenshot by Seth Rosenblatt/CNET)

The Web browser is the most-used kind of software in the world, having become the de facto way that people access the Internet. Today, virtually all computing tasks can be completed in the browser.
Testing browsers can veer from incredibly complex to shockingly simple, depending on what you're looking for and why. At CNET, we prefer a holistic approach to browser benchmarking, looking at a combination of tests that benchmark general browser behavior, as well as several "real-world" tests that look at browser performance in common scenarios. 

Note about mobile testing: We are still finalizing our standards for mobile browser testing, and will update this post as soon as they're ready. For now, the following procedures apply only to desktop browsers.

Is your favorite browser on our test list?

Unless your favorite browser is some obscure remixed version of Netscape, chances are good we test it. However, browser testing is made even more complex than it would otherwise be by the fact that two of the five major browsers, Firefox and Chrome, update on a six-week release cycle. Sometimes those updates bring dramatic changes, but often they don't. Because of the sadly human limitations of your humble editors, CNET will not be testing all browsers simultaneously. Instead, we will conduct quarterly tests for the most-used and best-known browsers, and biannual tests for a wider range of competitors. Also, tests are staggered according to platform, so Windows browsers are not tested simultaneously with Mac, Android, or iOS browsers.

Desktop browsers tested, both Windows and Mac unless otherwise noted:

Chrome

Firefox

Internet Explorer 9 (Windows 7 only)

Internet Explorer 10 (Windows 8 only)

Opera

Safari (Mac only)

Desktop browsers tested biannually will include:

Maxthon

Avant (Windows only)

How we test desktop browsers

We run each of the following three times, and restart the computer before each test, so that the browser is starting "cold." We also wait 30 seconds after booting the browser to ensure that any background processes have been completed. Then we average the three tests.

Google Octane, the successor to Google's V8 test.

(Credit: Screenshot by Seth Rosenblatt/CNET)

Performance benchmarks
The Acid3 test from the Web Standards Project checks browser compliance with accepted standards. Slightly outdated since it doesn't look at HTML5, which has now been finalized, it remains a good way to establish a baseline. Browsers that don't hit 100 out of 100 on the Acid3 are behind the times in a fundamental, crucial way.
Google Octane, the successor to Google's V8 benchmark test, looks at JavaScript performance by testing such areas as code optimization, encryption and decryption, emulation, and array manipulation, and assigning each sub-test a number. The higher the final score, the better.
Mozilla Kraken is another JavaScript performance test, which looks specifically at rendering times for audio, imaging, AI, JSON, and encryption. A smaller number is better for the final score.
The HTML5 Test assigns points for each HTML5 feature that the browser supports, out of a total of 500. This is a rough way to gauge how future-forward the browser is.
JSGameBench, GUImark3 (gaming test, text test), and Microsoft FishIE Tank look at HTML5 Canvas performance in several different game-like environments. Canvas is an important part of HTML5 to test because it creates all the nifty 2D images and shapes that can move across your screen.

Facebook's Ringmark tested can be used on desktop browsers, but it's designed for mobile.

(Credit: Screenshot by Seth Rosenblatt/CNET)

Each of these three tests uses a different standard. FishIE Tank, for example, lets the tester set the number of fish on the screen. The test will then show you in how many frames per second they can be rendered.
Microsoft Chalkboard runs a series of timed tests on HTML5 panning, zooming, and scaling. Faster is better.
Facebook Ringmark checks HTML5 feature support, geared for the needs of the mobile browser. Nevertheless, it works well on desktops and provides a good and rare point of comparison between desktop and mobile.
Real-world tests
We also perform four "real-world" tests to see how the browser performs under in-use conditions. These tests look at specific browser behaviors that you're likely to encounter: startup from cold boot, memory used while open, shutdown time, and wake from sleep.
Like the benchmark testing, each test is performed three times and then averaged. Unlike those tests, which are performed only with the tab running the test opened, our real-world tests are conducted twice: with five tabs open, and with 50 tabs. This is to replicate the real-world scenario of keeping many tabs open simultaneously, something that many people do (even if you don't).
Tabs are chosen according to categories based on realistic use cases: search engine, streaming media, news site, gaming site, and Web mail. The five tabs we open in the less-intensive test are: Google.com, CNET.com, Outlook.com, YouTube.com, and Pandora.com. We're not going to list all 50 sites here for space considerations.

source via cnet