Search Engine Quality Report

The discussion about which search engine produces the best results is a frequent one, and often a heated one.

We determined to develop and test a rigorous methodology for answering this question.

Due to the nature of the problem, this methodology must contain a mix of objective and subjective analysis. We determined to limit subjective analysis as much as possible, but were unable to completely eliminate them.

Our Search Engine Testing Methodology

First, we defined ten general categories of searches.

Category
A band or a musician
A famous athlete
A historical figure
A movie
A place
A politician
A pretty girl
An entertainer
Something to buy

We then identified one randomly chosen example search for each category.

This enables us to execute these tests repetively by choosing a new set of sample search terms each time.

This also enables you to run these tests yourself by following our testing methodology with search phrases of your own choosing.

We then conduct searches for each of these search terms on the following major search engines:

• Google
• Yahoo
• MSN

Our Search Engine Results Scoring System

Finally, we examine the first ten search results from each search engine and score them using the following criteria:

Result	Points
Exactly Relevant	1
Somewhat Relevant	.5
Subscriber-Only	.5
Directory	.5
Non-English	0
Off Topic	0
Spam	0
Duplicate	0
Non-Responsive	0
Search Result	0

We only awarded one-half of a point for results which were only somewhat relevant to our search, for web-sites where we could not receive content without registering, and for directories which offered no content of their own.

We did not award any points for non-English results, off-topic results, spam sites, duplicate pages, non-responsive pages, and web pages that only served more sets of search results.

Note that we did not analyze sponsored results.

Our Results - Which Search Engine Does Return the Best Results?

The results of each round of our tests are available here:

• Search Engine Test Results - Round One
• Search Engine Test Results - Round Two
• Search Engine Test Results - Round Three

The FAQs