How much are Google, Yahoo and MSN able to index?

We created this SEO experiment to determine how much text the major search engines (Google, Yahoo and MSN) can index. The additional parameter we wish to explore is a maximum search engine cache size.

Of course, you need to keep the size of your pages much smaller than the numbers described below for good rankings, and this experiment aimed just to find out how much disk space search engines reserve for storing their index.

How?

We've created a rather big test file (about 1.3 MB). Every kilobyte of the text on that page contains size marks (2KB, 3KB, 4KB and so on), so we can determine the max text size in SE index performing search for these marks. The test file is based on Leonardo Da Vinci notebooks, and link is here: SEO Experiment: How much are Google, Yahoo and MSN able to index. Sure there is a possibility, that "how much search engine will index" and "how much search engine can index" can be a different parameters, dependent on some factors such as inbound links number. At least, we'll figure out the indexed size of an ordinary page like our test page.

Experiment Results

Indexed size is a size of text in kilobytes, that can be found performing search query on every search engine. Cached size is a size of text in kilobytes, cached by search engine. It is determined from the SERP snippet (Google, Yahoo), or by inspecting the cached page contents (MSN).

Date	Google		Yahoo		MSN
	Indexed size	Cached size	Indexed size	Cached size	Indexed size	Cached size
3 Feb 2007 (1 day online)	~36 KB	977 KB	N/A	N/A	N/A	N/A
23 Feb 2007 (21 day online)	~917 KB	250 KB	~210 KB	525 KB	~1024 KB	~1024 KB

Queries used for this experiment:

• Google: site:wmtips.com inurl:leonardo 916KB
• Yahoo: site:wmtips.com inurl:leonardo 209KB
• MSN: site:wmtips.com "so we can determine the max text size" 1023KB

Conclusions

MSN is able to index more than Google and Yahoo and amount it can index is about 1 MB of text. Google is on the second place with 917 KB of text it's able to index. Yahoo can index about 210 KB.

Google has the best indexing speed. It takes small time after ping to add content to the Google index. Though, it seems that completing the index require some additional time, so we can see the different results in Google on the 1st and 21st day after content publishing.

It seems "Indexed size" and "Cached size" parameters are not equal for Google and Yahoo, and equal for MSN.

Also it's interesting how Google shows its snippet. On the small numbers it shows query keywords directly in the resulting snippet. On the bigger numbers (farther from the beginning), it shows only the beginning of the text in the snippet. So there is a chance that Google has another one smaller "snippet index", used specially for snippets.