What search engine returns the best results?

The discussion about which search engine produces the best results is a frequent one, and often a heated one.

We determined to develop and test a rigorous methodology for answering this question.

This methodology must contain a mix of objective and subjective analysis. We determined to limit subjective analysis as much as possible, but were unable to completely eliminate them.

Our Search Engine Testing Methodology

First, we defined ten general categories of searches.

Category
A band or a musician
A famous athlete
A historical figure
A movie
A place
A politician
A pretty girl
An entertainer
Something to buy

We then identified one randomly chosen example search for each category.

This enables us to execute these tests again by choosing another set of ten sample search terms.

This also enables you to run these tests yourself by following our testing methodology with search phrases of your choice.

We then conducted searches for each of these search terms on the following search engines:

• Google
• Yahoo
• MSN
• AltaVista
• AllTheWeb

Our Search Engine Results Scoring System

Finally, we examined the first ten search results from each search engine and scored them using the following criteria:

Result	Points
Exactly Relevant	1
Somewhat Relevant	.5
Subscriber-Only	.5
Directory	.5
Non-English	0
Off Topic	0
Spam	0
Duplicate	0
Non-Responsive	0
Search Result	0

We only awarded one-half of a point for results which were only somewhat relevant to our search, for web-sites where we could not receive content without registering, and for directories which offered no content of their own.

We did not award any points for non-English results, off-topic results, spam sites, duplicate pages, non-responsive pages, and web pages that only served more sets of search results.

Note that we did not analyze sponsored results.

Our Results - Which Search Engine Does Return the Best Results?

We just finished the second round of our search engine tests. The results of both rounds are available here:

• Search Engine Test Results - Round One
• Search Engine Test Results - Round Two

What Next?

We will periodically redo these tests, with a different set of randomly chosen example searches each time.

We hope to continue to see improvements from each of the search engines we evaluate!

The FAQs