WebCrawler

Last updated
WebCrawler
Webcrawler logo 2018.png
Logo since 2018
Type of site
Search engine
Available in English
Owner System1
Created byBrian Pinkerton
URL webcrawler.com
CommercialNo
Registrationnone
LaunchedApril 20, 1994;30 years ago (1994-04-20)
Current statusActive

WebCrawler is a search engine, and one of the oldest surviving search engines on the web today. For many years, it operated as a metasearch engine. WebCrawler was the first web search engine to provide full text search. [1]

Contents

History

Screenshot of WebCrawler homepage in September 1995 Webcrawler screenshot 1995.png
Screenshot of WebCrawler homepage in September 1995

Brian Pinkerton first started working on WebCrawler, which was originally a desktop application, on January 27, 1994 at the University of Washington. [2] On March 15, 1994, he generated a list of the top 25 websites. [1]

WebCrawler launched on April 21, 1994, with more than 4,000 different websites in its database [2] and on November 14, 1994, WebCrawler served its 1 millionth search query [2] for "nuclear weapons design and research". [3]

On December 1, 1994, WebCrawler acquired two sponsors, DealerNet and Starwave, which provided money to keep WebCrawler operating. [2] Starting on October 3, 1995, WebCrawler was fully supported by advertising, but separated the adverts from search results. [2]

On June 1, 1995, America Online (AOL) acquired WebCrawler. [2] After being acquired by AOL, the website introduced its mascot "Spidey" on September 1, 1995. [2]

Starting in April 1996, [2] WebCrawler also included the human-edited internet guide GNN Select, which was also under AOL ownership. [4] [5]

On April 1, 1997, Excite acquired WebCrawler from AOL for $12.3 million. [2] [6]

WebCrawler received a redesign on June 16, 1997, adding WebCrawler Shortcuts, which suggested alternative links to material related to a search topic. [7]

WebCrawler was maintained by Excite as a separate search engine with its own database until 2001, when it started using Excite's own database, effectively putting an end to WebCrawler as an independent search engine. [8] Later that year, Excite (then called Excite@Home) went bankrupt and WebCrawler was bought by InfoSpace in 2001. [2]

WebCrawler's homepage (June 2010) WebCrawler Screenshot 6-7-2010.png
WebCrawler's homepage (June 2010)

Pinkerton, WebCrawler's creator, led the Amazon A9.com search division as of 2012. [9] [10]

In July 2016, InfoSpace was sold by parent company Blucora to OpenMail for $45 million, putting WebCrawler under the ownership of OpenMail. [11] OpenMail was later renamed System1. [12]

In 2018, WebCrawler was redesigned from scratch and its logo was changed. [13] [14]

Traffic

WebCrawler was highly successful early on. [15] At one point, it was unusable during peak times due to server overload. [16] It was the second most visited website on the internet in February 1996, but it quickly dropped below rival search engines and directories such as Yahoo!, Infoseek, Lycos, and Excite in 1997. [17]

See also

Related Research Articles

<span class="mw-page-title-main">Lycos</span> Search engine and web portal

Lycos, Inc., is a web search engine and web portal established in 1994, spun out of Carnegie Mellon University. Lycos also encompasses a network of email, web hosting, social networking, and entertainment websites. The company is based in Waltham, Massachusetts, and is a subsidiary of Ybrant Digital.

robots.txt Internet protocol

robots.txt is the filename used for implementing the Robots Exclusion Protocol, a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.

<span class="mw-page-title-main">MapQuest</span> American free Internet mapping service (founded 1996)

MapQuest is an American free online web mapping service. It was launched in 1996 as the first commercial web mapping service. MapQuest vies for market share with competitors such as Apple Maps, Here and Google Maps.

<span class="mw-page-title-main">AltaVista</span> Web search engine

AltaVista was a Web search engine established in 1995. It became one of the most-used early search engines, but lost ground to Google and was purchased by Yahoo! in 2003, which retained the brand, but based all AltaVista searches on its own search engine. On July 8, 2013, the service was shut down by Yahoo!, and since then the domain has redirected to Yahoo!'s own search site.

HowStuffWorks is an American commercial infotainment website founded by professor and author Marshall Brain, to provide its target audience an insight into the way many things work. The site uses various media to explain complex concepts, terminology, and mechanisms—including photographs, diagrams, videos, animations, and articles.

<span class="mw-page-title-main">Excite (web portal)</span> Internet portal

Excite is an American website operated by IAC that provides outsourced internet content such as a metasearch engine, with outsourced weather and news content on the main page. As of 2024, all of Excite's operations are controlled by services outside of the business.

<span class="mw-page-title-main">Dogpile</span> Metasearch engine

Dogpile is a metasearch engine for information on the World Wide Web that fetches results from Google, Yahoo!, Yandex, Bing, and other popular search engines, including those from audio and video content providers such as Yahoo!.

<span class="mw-page-title-main">A9.com</span> Subsidiary of Amazon based in Palo Alto, California

A9.com was a former subsidiary of Amazon that developed search engine and search advertising technology. A9 was based in Palo Alto, California, with teams in Seattle, Bangalore, Beijing, Dublin, Iași, Munich and Tokyo. A9 has development efforts in areas of product search, cloud search, visual search, augmented reality, advertising technology and community question answering.

<span class="mw-page-title-main">AlltheWeb</span>

AlltheWeb was an Internet search engine that made its debut in mid-1999 and was closed in 2011. It grew out of FTP Search, Tor Egge's doctorate thesis at the Norwegian University of Science and Technology, which he started in 1994, which in turn resulted in the formation of Fast Search & Transfer (FAST), established on July 16, 1997.

Yahoo! Japan is a Japanese web portal. Its search engine was the most-visited website in Japan, nearing monopolistic status.

Infoseek was an American internet search engine founded in 1994 by Steve Kirsch.

Yahoo! Search is a search engine owned and operated by Yahoo!, using Microsoft Bing to power results.

<span class="mw-page-title-main">InfoSpace</span> American company

Infospace, Inc. was an American company that offered private label search engine, online directory, and provider of metadata feeds. The company's flagship metasearch site was Dogpile and its other notable consumer brands were WebCrawler and MetaCrawler. After a 2012 rename to Blucora, the InfoSpace business unit was sold to data management company OpenMail.

<span class="mw-page-title-main">Search engine</span> Software system for finding relevant information on the Web

A search engine is a software system that provides hyperlinks to web pages and other relevant information on the Web in response to a user's query. The user inputs a query within a web browser or a mobile app, and the search results are often a list of hyperlinks, accompanied by textual summaries and images. Users also have the option of limiting the search to a specific type of results, such as images, videos, or news.

MetaCrawler is a search engine. It is a registered trademark of InfoSpace and was created by Erik Selberg.

<span class="mw-page-title-main">Waterfox</span> Open-source web browser based on Firefox

Waterfox is a free and open-source web browser and fork of Firefox. It claims to be ethical and user-centric, emphasizing performance and privacy. There are official Waterfox releases for Windows, macOS, Linux and Android. It was initially created to provide official 64-bit support, back when Firefox was only available for 32-bit systems.

Zoo.com is a metasearch engine, which as of 2006, provided results from search engines and other sources, including Google, Yahoo! and Wikipedia. Also as of 2006, Zoo.com provided news content from ABC News, Fox News and Yahoo! News.

System1 is an American Internet advertising company. Formerly known as OpenMail, it was founded in 2013. It describes itself as operating a "Responsive Acquisition Marketing Platform", and cites privacy as one of its principal foci, although it has been criticized for its influence on privacy-focussed properties, including search engine Startpage.com. It is headquartered in Marina del Rey, California.

References

  1. 1 2 "Short History of Early Search Engines". The History of SEO. Archived from the original on 2019-01-21. Retrieved 2019-02-03.
  2. 1 2 3 4 5 6 7 8 9 10 "WebCrawler's History". www.thinkpink.com. Archived from the original on 2005-11-28. Retrieved 2019-01-09.
  3. Lammle, Rob (2012-03-16). "'90s Tech Icons: Where Are They Now?". Mashable. Archived from the original on 2012-03-17. Retrieved 2019-02-18.
  4. "Se-En". searchenginearchive.com. Retrieved 2019-01-25.
  5. "WebCrawler Select: Review Categories". WebCrawler. 1996-10-24. Archived from the original on 1996-10-24. Retrieved 2019-02-03.
  6. Keogh, Garret. "Excite buys WebCrawler from AOL". ZDNet. Retrieved 2019-01-15.
  7. Sullivan, Danny (1997-06-16). "The Search Engine Update, June 17, 1997, Number 7". Search Engine Watch. Archived from the original on 2016-04-14. Retrieved 2019-02-02.
  8. R. Notess, Greg (2002). "On the Net: Dead Search Engines". InfoToday. Archived from the original on 2002-05-25. Retrieved 2019-01-16.
  9. Brid-Aine Parnell (December 18, 2012). "Search engines we have known ... before Google crushed them". The Register. Retrieved November 17, 2016.
  10. "Leading Leaders". A9 Management web page. Archived from the original on November 14, 2016. Retrieved November 15, 2016.
  11. "Blucora to sell InfoSpace business for $45 million". Seattle Times. July 5, 2016.
  12. "System1 raises $270 million for 'consumer intent' advertising". L.A. Biz. Retrieved 2017-12-01.
  13. "WebCrawler Search". WebCrawler. 2018-05-31. Archived from the original on 2018-05-31. Retrieved 2019-02-02.
  14. "WebCrawler Search". WebCrawler. 2018-11-30. Archived from the original on 2018-11-30. Retrieved 2019-02-02.
  15. McGuigan, Brendan (2007). "What was the First Search Engine?". WiseGeek. Archived from the original on 2007-04-27. Retrieved 2019-02-18.
  16. "Search Engine History.com". www.searchenginehistory.com. Retrieved 2019-01-25.
  17. "Infographic: Top 20 Most Popular Websites (1996-2013)". TechCo. 2014-12-26. Retrieved 2019-01-15.