Google and Dmoz : Are They in Love?

What is the relationship between the Web's most popular search engine and the largest, most comprehensive human-edited Web directory?

Are they married?

No. The Open Directory Project (a.k.a. Dmoz) is owned by Netscape, which in turn is owned by AOL, which is owned by massive media conglomerate Time Warner. Though Google bought a 5% stake in AOL at the end of 2005, I wouldn't call that a merger.

Are they in bed together?

All the time. But then both parties are shamelessly promiscuous. They aim to be all over the Web and they don't care who knows it. The ODP's data is free to use under license and hundreds of webmasters snap it up. The Google Directory is one of many ODP clones. Meanwhile Google's spider - Googlebot - chases around the Net probing every nook and cranny. Dmoz is just one of millions of sites enjoying his attentions.

So there is no shotgun in the case?

AOL Search is powered by Google. So did AOL force Google to use Dmoz as part of a corporate deal? No. Google created its Dmoz-based directory in 2000. It was not until 2002 that Google began to supply AOL search results.

So what's the big attraction?

Some search engines come right out and admit that they go for Dmoz. Kevin Marcus, creator of Search Hippo, is engagingly frank. His database "is first seeded with sites that are currently listed on the larger directories on the Internet (such as the Open Directory)". The bigger engines may be more coy about it, but are just as keen to run their bots over Dmoz. The major directories - and Dmoz is the biggest of all - are an obvious hangout for link-hungry spiders. A virginal site with no other link to it could be listed in Dmoz on a Monday and find itself in Google and several other engines by the weekend.

So does Dmoz have special pulling power?

Not really. A virginal site listed in Yahoo! should attract the bot swarm just as well. A link from any established site that is constantly spidered by all the major engines would do the same job. Though experimental new engines do seem drawn to Dmoz. It's a useful testing ground.

Is Google losing interest?

a) Updating

Google used to update its directory from the ODP every month and appears to have returned to that schedule, with updates  on 12 December 2007 and 12 January  2008.

However there have been long gaps in the schedule. Those in 2003 and early 2004 were apparently due to technical hitches in the ODP's data. The very few updates in 2006 and 2007 were unexplained, but suggested that Google did not view the directory as a priority.

b) Demotion

On 29 March 2004 Google.com rolled out a new interface without the directory tab. The directory was still there, but you needed to click on 'more' to find it. The reasons for the demotion were twofold, according to Google spokesmen. The directory was not heavily used and Google needed the space.

A further demotion was part of the package announced on 16 May 2007 at Google's Searchology Day. New navigational links were rolled out onto Google.com in the following days and presumably will propagate to other Google search sites. The new interface has no link to the directory.  The directory still exists as I write, but its future must be uncertain if no link to it is intended.

c) Search results

Prior to March 2004 if a site was listed in the Google Directory, then if that site came up in search results, Google would include the site description from its directory and a link to the category where it was listed. Those features were removed in March 2004. However in late February 2005 directory descriptions began to appear occasionally in place of the usual snippet of text from the site, depending on the query and other factors. In addition Google began to sometimes use the directory site title.

So Google is still flirting with the Open Directory, but the uses of ODP data within the search engine seem to be overtaking a simple clone of the directory.    

Sounds like there's a certain rapport?

There were benefits on both sides. Google gained a free directory of hand-selected and ordered links. Google's popularity meant that its directory got noticed, which brought more submissions and editor applications to the ODP - too many, some feel. The ODP groaned under the weight of its admirers. (It's the price of fame.) Google added its own value to the directory. Sites in the ODP are listed in alphabetical order, except for the few that editors pick out for star treatment. The default option in the Google Directory is order by PageRank. That is so useful that the Open Directory makes it easy to go directly from any of its categories to the equivalent category in the Google Directory. Just click on the green ball at the bottom right of the ODP category.

Does Google trust Dmoz?

Google has been hurt by all the people trying to deceive it, faking popularity to rise in the search results. Link-buying and suchlike have undermined PageRank - the very foundation of Google. So when Google trademarked the name TrustRank, it was widely assumed to be a replacement for PageRank.

As it happens, Yahoo personnel were simultaneously playing with a theory they called TrustRank. The idea is to have a secret list of trusted sites. A link from one of them would convey trust. So the two were confused until Matt Cutts of Google revealed in December 2007 that Google's TrustRank was an anti-phishing filter. 

Until this bubble was burst, there was fevered speculation about the sites Google trusts. Many a finger has pointed at Dmoz, though he paper on TrustRank selected seed sites with content firmly under the control of a body such as a government, university or corporation. Dmoz is run by volunteers.

That's not to say that Google can't use Dmoz data in some other way. In its endless battle against spam, Google needs every weapon it can lay hands on. Recently Google patented a method of including an element of editorial opinion in its algorithm. Dmoz was specifically mentioned.

Yet no-one knows better than Google that Dmoz only lists a fraction of the Web. So Google is not about to rely on Dmoz alone for any crucial element of its algorithm. That is a safe bet.

Webmaster worries

Q. When will my Dmoz link appear in Google?

In the Google search?
The busy Googlebots should find and follow a link from Dmoz pretty quickly. But remember that a site may be listed in Dmoz, but banned from Google. Each has its own rules for inclusion.

In the Google Directory?
Whenever the Google Directory is updated from Dmoz. 

Q. My site changed its URL. The Dmoz listing was updated two days ago. Why is Google search still bringing up the old URL?

Although Googlebot should soon pick up the new URL, the Google Directory will still have the old one listed. So will other ODP clones. That is a lot of links to the old location. So your old site will retain PageRank. That could mean that it carries on coming up in Google's results ahead of the new URL for some time. Moving house is always a nuisance. See Moving Internet Address?

Q. Will a Dmoz listing boost my Google PageRank?

Every link helps. But when assigning PageRank, Google treats Dmoz exactly the same as any other site. It does not give any extra weight to a link from the Open Directory. However a link from Dmoz eventually spawns links from its many clones, which adds to PageRank. Since many ODP clones have low PR, the effect may not be as great as is sometimes supposed. The most notable clone is the Google Directory.

Q. My competitor's site is starred in Dmoz. Is this why it does better in Google?

No. ODP editors can mark one or two sites in a category as 'cool'. They then appear at the top of the category, with a star. Editors should only pick out the comprehensive or official site on the topic. But however good the reason, Google pays no attention. It has its own way of ranking in its directory and its own algorithm for search.

Q. Surely having my keywords in the description of my site in Dmoz will help it to be found in Google?

Actually no. To test this for yourself, find a listing in the Open Directory which contains a misspelling, making a complete nonsense word. Search Google for this word. Directory pages containing the listing will appear. But not the site itself.

Q. If Google uses the Dmoz title/description for my page in its search results, will that hurt my site?

It won't have any effect on the ranking of the site/page in Google. Google's algorithm determines ranking. The title/description shown in the search results is just a matter of presentation. That may make a difference to how many searchers click on a particular result. Although webmasters generally feel that the description in their own meta-tags is the best one for the site/page, Google will try to serve up the one that best fits the query, which might be the one from Dmoz.

If you want to ensure that Google does not use Dmoz data in its presentation of results, you can simply add a meta tag to prevent it.

You want more?

The Open Directory has listed many web-sites full of analysis and news of Google at Search Engines: Google.

Questions about the Open Directory will be answered by Dmoz editors at The Open Directory Project Public Forum.

For a  wider picture of the way Open Directory data is used outside the directory itself, see Uses of the Open Directory.