Web directory

Last updated

A web directory or link directory is an online list or catalog of websites. That is, it is a directory on the World Wide Web of (all or part of) the World Wide Web. Historically, directories typically listed entries on people or businesses, and their contact information; such directories are still in use today. A web directory includes entries about websites, including links to those websites, organized into categories and subcategories. [1] [2] [3] Besides a link, each entry may include the title of the website, and a description of its contents. In most web directories, the entries are about whole websites, rather than individual pages within them (called "deep links"). Websites are often limited to inclusion in only a few categories.


There are two ways to find information on the Web: by searching or browsing. Web directories provide links in a structured list to make browsing easier. Many web directories combine searching and browsing by providing a search engine to search the directory. Unlike search engines, which base results on a database of entries gathered automatically by web crawler, most web directories are built manually by human editors. Many web directories allow site owners to submit their site for inclusion, and have editors review submissions for fitness.

Web directories may be general in scope, or limited to particular subjects or fields. Entries may be listed for free, or by paid submission (meaning the site owner must pay to have his or her website listed).

RSS directories are similar to web directories, but contain collections of RSS feeds, instead of links to web sites.


During the early development of the web, there was a list of web servers edited by Tim Berners-Lee and hosted on the CERN webserver. One historical snapshot from 1992 remains. [4] He also created the World Wide Web Virtual Library, which is the oldest web directory. [5]

Scope of listing

Most of the directories are general in on scope and list websites across a wide range of categories, regions and languages. But some niche directories focus on restricted regions, single languages, or specialist sectors. One type of niche directory with a large number of sites in existence is the shopping directory. Shopping directories specialize in the listing of retail e-commerce sites.

Examples of well-known general web directories are Yahoo! Directory (shut down at the end of 2014) and DMOZ (shut down on March 14, 2017). DMOZ was significant due to its extensive categorization and large number of listings and its free availability for use by other directories and search engines. [6]

However, a debate over the quality of directories and databases still continues, as search engines use ODP's content without real integration, and some experiment using clustering.


There have been many attempts to make building web directories easier, such as using automated submission of related links by script, or any number of available PHP portals and programs. Recently, social software techniques have spawned new efforts of categorization, with Amazon.com adding tagging to their product pages.


Directories have various features in their listings, often depending upon the price paid for inclusion:

Human-edited web directories

A human-edited directory is created and maintained by editors who add links based on the policies particular to that directory. Human-edited directories are often targeted by SEOs on the basis that links from reputable sources will improve rankings in the major search engines. Some directories may prevent search engines from rating a displayed link by using redirects, nofollow attributes, or other techniques. Many human-edited directories, including DMOZ, World Wide Web Virtual Library, Business.com and Jasmine Directory, are edited by volunteers, who are often experts in particular categories. These directories are sometimes criticized due to long delays in approving submissions, or for rigid organizational structures and disputes among volunteer editors.

In response to these criticisms, some volunteer-edited directories have adopted wiki technology, to allow broader community participation in editing the directory (at the risk of introducing lower-quality, less objective entries).

Another direction taken by some web directories is the paid for inclusion model. This method enables the directory to offer timely inclusion for submissions and generally fewer listings as a result of the paid model. They often offer additional listing options to further enhance listings, including features listings and additional links to inner pages of the listed website. These options typically have an additional fee associated but offer significant help and visibility to sites and/or their inside pages.

Today submission of websites to web directories is considered a common SEO (search engine optimization) technique to get back-links for the submitted website. One distinctive feature of 'directory submission' is that it cannot be fully automated like search engine submissions. Manual directory submission is a tedious and time-consuming job and is often outsourced by webmasters.

Bid for Position directories

Bid for Position directories, also known as bidding web directories, are paid-for-inclusion web directories where the listings of websites in the directory are ordered according to their bid amount. They are special in that the more a person pays, the higher up the list of websites in the directory they go. With the higher listing, the website becomes more visible and increases the chances that visitors who browse the directory will click on the listing.

See also

Related Research Articles

Meta elements are tags used in HTML and XHTML documents to provide structured metadata about a Web page. They are part of a web page's head section. Multiple Meta elements with different attributes can be used on the same page. Meta elements can be used to specify page description, keywords and any other metadata not provided through the other head elements and attributes.

Spamdexing is the deliberate manipulation of search engine indexes. It involves a number of methods, such as link building and repeating unrelated phrases, to manipulate the relevance or prominence of resources indexed, in a manner inconsistent with the purpose of the indexing system.

Search engine optimization (SEO) is the process of improving the quality and quantity of website traffic to a website or a web page from search engines. SEO targets unpaid traffic rather than direct traffic or paid traffic. Unpaid traffic may originate from different kinds of searches, including image search, video search, academic search, news search, and industry-specific vertical search engines.

Link farm Group of websites that link to each other

On the World Wide Web, a link farm is any group of websites that all hyperlink to other sites in the group for the purpose of increasing SEO rankings. In graph theoretic terms, a link farm is a clique. Although some link farms can be created by hand, most are created through automated programs and services. A link farm is a form of spamming the index of a web search engine. Other link exchange systems are designed to allow individual websites to selectively exchange links with other relevant websites and are not considered a form of spamdexing.

Googlebot Web crawler used by Google

Googlebot is the web crawler software used by Google, which collects documents from the web to build a searchable index for the Google Search engine. This name is actually used to refer to two different types of web crawlers: a desktop crawler and a mobile crawler.

Cloaking is a search engine optimization (SEO) technique in which the content presented to the search engine spider is different from that presented to the user's browser. This is done by delivering content based on the IP addresses or the User-Agent HTTP header of the user requesting the page. When a user is identified as a search engine spider, a server-side script delivers a different version of the web page, one that contains content not present on the visible page, or that is present but not searchable. The purpose of cloaking is sometimes to deceive search engines so they display the page when it would not otherwise be displayed. However, it can also be a functional technique for informing search engines of content they would not otherwise be able to locate because it is embedded in non-textual containers such as video or certain Adobe Flash components. Since 2006, better methods of accessibility, including progressive enhancement, have been available, so cloaking is no longer necessary for regular SEO.

A backlink for a given web resource is a link from some other website to that web resource. A web resource may be a website, web page, or web directory.

Pay-per-click (PPC) is an internet advertising model used to drive traffic to websites, in which an advertiser pays a publisher when the ad is clicked.

Search engine marketing (SEM) is a form of Internet marketing that involves the promotion of websites by increasing their visibility in search engine results pages (SERPs) primarily through paid advertising. SEM may incorporate search engine optimization (SEO), which adjusts or rewrites website content and site architecture to achieve a higher ranking in search engine results pages to enhance pay per click (PPC) listings.

A site map is a list of pages of a web site within a domain.

Web content development is the process of researching, writing, gathering, organizing, and editing information for publication on websites. Website content may consist of prose, graphics, pictures, recordings, movies, or other digital assets that could be distributed by a hypertext transfer protocol server, and viewed by a web browser.

The Sandbox effect is a name given to an observation of the way Google ranks web pages in its index. It is the subject of much debate—its existence has been written about since 2004 but not confirmed, with several statements to the contrary.

The Yahoo! Directory was a web directory which at one time rivaled DMOZ in size. The directory was Yahoo!'s first offering and started in 1994 under the name Jerry and David's Guide to the World Wide Web. When Yahoo! changed its main results to crawler-based listings under Yahoo! Search in October 2002, the human-edited directory's significance dropped, but it was still being updated as of August 19, 2014. Users could browse thousands of listings which were organized in 7 or more tiers. For example, if a user was looking for a site on chess they might follow a path such as: recreation -> games -> board games -> chess.

nofollow is a setting on a web page hyperlink that directs search engines not to use the link for page ranking calculations. It is specified in the page as a type of link relation; that is: <a rel="nofollow" ...>. Because search engines often calculate a site's importance according to the number of hyperlinks from other sites, the nofollow setting allows web site authors to indicate that the presence of a link is not an endorsement of the target site's importance.

Search engine Software system that is designed to search for information on the World Wide Web

A search engine is a software system that is designed to carry out web searches, which means to search the World Wide Web in a systematic way for particular information specified in a textual web search query. The search results are generally presented in a line of results, often referred to as search engine results pages (SERPs) The information may be a mix of links to web pages, images, videos, infographics, articles, research papers, and other types of files. Some search engines also mine data available in databases or open directories. Unlike web directories, which are maintained only by human editors, search engines also maintain real-time information by running an algorithm on a web crawler. Internet content that is not capable of being searched by a web search engine is generally described as the deep web.

In the field of search engine optimization (SEO), link building describes actions aimed at increasing the number and quality of inbound links to a webpage with the goal of increasing the search engine rankings of that page or website. Briefly, link building is the process of establishing relevant hyperlinks to a website from external sites. Link building can increase the number of high-quality links pointing to a website, in turn increasing the likelihood of the website ranking highly in search engine results. Link building is also a proven marketing tactic for increasing brand awareness.

DMOZ Open content directory of Web links established in 1998

DMOZ was a multilingual open-content directory of World Wide Web links. The site and community who maintained it were also known as the Open Directory Project (ODP). It was owned by AOL but constructed and maintained by a community of volunteer editors.

An article directory is a website with collections of articles written about different subjects. Sometimes article directories are referred to as content farms, which are websites created to produce mass content, where some are based on churnalism.

Jasmine Directory Human-edited web directory

Jasmine Directory is a human-edited web directory providing websites and businesses categorized topically and regionally. It offers thirteen topic-based categories and one region-based category with hand-picked and reviewed users' suggested resources. Jasmine Directory was founded in 2009 by Pécsi András and Robert Gomboș and is headquartered in Valley Cottage, New York. It won eight prizes during 2013–2014 for its editorial discretion and manually added resources. Jasmine Directory proved to be useful for SEO Google search results since they manually add about 90% of the resources.


  1. "Web directory". Dictionary. Retrieved 30 August 2013.
  2. Wendy Boswell. "What is a Web Directory". About.com. Retrieved 2010-02-25.
  3. "Web Directory Or Directories". yourmaindomain. Retrieved 30 August 2013.
  4. "World-Wide Web Servers". W3.org. Retrieved 2012-05-14.
  5. Aaron Wall. "History of Search Engines: From 1945 to Google Today". Search Engine History. Retrieved 2017-05-16.
  6. Paul Festa (December 27, 1999), Web search results still have human touch, CNET News.com, retrieved September 18, 2007
  7. Schmitz, Tom (August 2, 2012). "What Everyone Needs To Know About Good, Bad & Bland Links". searchengineland.com. Third Door Media. Retrieved April 21, 2017. Reciprocal links may not help with competitive keyword rankings, but that does not mean you should avoid them when they make sound business sense. What you should definitely avoid are manipulative reciprocal linking schemes like automated link trading programs and three-way links or four-way links.