DeepPeep

Last updated

DeepPeep was a search engine that aimed to crawl and index every database on the public Web. [1] [2] Unlike traditional search engines, which crawl existing webpages and their hyperlinks, DeepPeep aimed to allow access to the so-called Deep web, World Wide Web content only available via for instance typed queries into databases. [3] The project started at the University of Utah and was overseen by Juliana Freire, an associate professor at the university's School of Computing WebDB group. [4] [5] The goal was to make 90% of all WWW content accessible, according to Freire. [6] [7] The project ran a beta search engine and was sponsored by the University of Utah and a $243,000 grant from the National Science Foundation. [8] It generated worldwide interest. [9] [10] [11] [12] [13]

Contents

How it works

Similar to Google, Yahoo, and other search engines, DeepPeep allows the users to type in a keyword and returns a list of links and databases with information regarding the keyword.

However, what separated DeepPeep and other search engines is that DeepPeep uses the ACHE crawler, 'Hierarchical Form Identification', 'Context-Aware Form Clustering' and 'LabelEx' to locate, analyze, and organize web forms to allow easy access to users. [14]

ACHE Crawler

The ACHE Crawler is used to gather links and utilizes a learning strategy that increases the collection rate of links as these crawlers continue to search. What makes ACHE Crawler unique from other crawlers is that other crawlers are focused crawlers that gather Web pages that have specific properties or keywords. Ache Crawlers instead includes a page classifier which allows it to sort out irrelevant pages of a domain as well as a link classifier which ranks a link by its highest relevance to a topic. As a result, the ACHE Crawler first downloads web links that has the higher relevance and saves resources by not downloading irrelevant data. [15]

Hierarchical Form Identification

In order to further eliminate irrelevant links and search results, DeepPeep uses the HIerarchical Form Identification (HIFI) framework that classifies links and search results based on the website's structure and content. [14] Unlike other forms of classification which solely relies on the web form labels for organization, HIFI utilizes both the structure and content of the web form for classification. Utilizing these two classifiers, HIFI organizes the web forms in a hierarchical fashion which ranks the a web form's relevance to the target keyword. [16]

Context-Aware Clustering

When there is no domain of interest or the domain specified has multiple types of definition, DeepPeep must separate the web form and cluster them into similar domains. The search engine uses context-aware clustering to group similar links in the same domain by modeling the web form into sets of hyperlinks and using its context for comparison. Unlike other techniques that require complicated label extraction and manual pre-processing of web forms, context-aware clustering is done automatically and uses meta-data to handle web forms that are content rich and contain multiple attributes. [14]

LabelEx

DeepPeep further extracts information called Meta-Data from these pages which allows for better ranking of links and databases with the use of LabelEx, an approach for automatic decomposition and extraction of meta-data. Meta-data is data from web links that give information about other domains. LabelEx identifies the element-label mapping and uses the mapping to extract meta-data with accuracy unlike conventional approaches that used manually specific extraction rules. [14]

Ranking

When the search results pop up after the user has input their keyword, DeepPeep ranks the links based on 3 features: term content, number of backlinks. and pagerank. Firstly, the term content is simply determined by the content of the web link and its relevance. Backlinks are hyperlinks or links that direct the user to a different website. Pageranks is the ranking of websites in search engine results and works by counting the amount and quality of links to website to determine its importance. Pagerank and back link information are obtained from outside sources such as Google, Yahoo, and Bing. [14]

Beta Launch

DeepPeep Beta was launched and only covered seven domains: auto, airfare, biology, book, hotel, job, and rental. Under these seven domains, DeepPeep offered access to 13,000 Web forms. [17] One could access the website at deeppeep.org, but the website has been inactive after the beta version was taken down.

Related Research Articles

Meta elements are tags used in HTML and XHTML documents to provide structured metadata about a Web page. They are part of a web page's head section. Multiple Meta elements with different attributes can be used on the same page. Meta elements can be used to specify page description, keywords and any other metadata not provided through the other head elements and attributes.

<span class="mw-page-title-main">Web crawler</span> Software which systematically browses the World Wide Web

A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing.

Spamdexing is the deliberate manipulation of search engine indexes. It involves a number of methods, such as link building and repeating unrelated phrases, to manipulate the relevance or prominence of resources indexed in a manner inconsistent with the purpose of the indexing system.

robots.txt is the filename used for implementing the Robots Exclusion Protocol, a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.

Search engine optimization (SEO) is the process of improving the quality and quantity of website traffic to a website or a web page from search engines. SEO targets unpaid traffic rather than direct traffic or paid traffic. Unpaid traffic may originate from different kinds of searches, including image search, video search, academic search, news search, and industry-specific vertical search engines.

<span class="mw-page-title-main">Googlebot</span> Web crawler used by Google

Googlebot is the web crawler software used by Google that collects documents from the web to build a searchable index for the Google Search engine. This name is actually used to refer to two different types of web crawlers: a desktop crawler and a mobile crawler.

Internet research is the practice of using Internet information, especially free information on the World Wide Web, or Internet-based resources in research.

<span class="mw-page-title-main">Deep web</span> Content of the World Wide Web that is not indexed by search engines

The deep web, invisible web, or hidden web are parts of the World Wide Web whose contents are not indexed by standard web search-engine programs. This is in contrast to the "surface web", which is accessible to anyone using the Internet. Computer scientist Michael K. Bergman is credited with inventing the term in 2001 as a search-indexing term.

<span class="mw-page-title-main">Dogpile</span> Metasearch engine

Dogpile is a metasearch engine for information on the World Wide Web that fetches results from Google, Yahoo!, Yandex, Bing, and other popular search engines, including those from audio and video content providers such as Yahoo!.

<span class="mw-page-title-main">Metasearch engine</span> Online information retrieval tool

A metasearch engine is an online information retrieval tool that uses the data of a web search engine to produce its own results. Metasearch engines take input from a user and immediately query search engines for results. Sufficient data is gathered, ranked, and presented to the users.

A backlink is a link from some other website to that web resource. A web resource may be a website, web page, or web directory.

Doorway pages are web pages that are created for the deliberate manipulation of search engine indexes (spamdexing). A doorway page will affect the index of a search engine by inserting results for particular phrases while sending visitors to a different page. Doorway pages that redirect visitors without their knowledge use some form of cloaking. This usually falls under Black Hat SEO.

Findability is the ease with which information contained on a website can be found, both from outside the website and by users already on the website. Although findability has relevance outside the World Wide Web, the term is usually used in that context. Most relevant websites do not come up in the top results because designers and engineers do not cater to the way ranking algorithms work currently. Its importance can be determined from the first law of e-commerce, which states "If the user can’t find the product, the user can’t buy the product." As of December 2014, out of 10.3 billion monthly Google searches by Internet users in the United States, an estimated 78% are made to research products and services online.

<span class="mw-page-title-main">Anchor text</span> Visible, clickable text in a hyperlink

The anchor text, link label or link text is the visible, clickable text in an HTML hyperlink. The term "anchor" was used in older versions of the HTML specification for what is currently referred to as the a element, or <a>. The HTML specification does not have a specific term for anchor text, but refers to it as "text that the a element wraps around". In XML terms, the anchor text is the content of the element, provided that the content is text.

<span class="mw-page-title-main">Search engine</span> Software system that is designed to search for information on the World Wide Web

A search engine is a software system that finds web pages that match a web search. They search the World Wide Web in a systematic way for particular information specified in a textual web search query. The search results are generally presented in a line of results, often referred to as search engine results pages (SERPs). The information may be a mix of hyperlinks to web pages, images, videos, infographics, articles, and other types of files. Some search engines also mine data available in databases or open directories. Unlike web directories and social bookmarking sites, which are maintained by human editors, search engines also maintain real-time information by running an algorithm on a web crawler. Any internet-based content that cannot be indexed and searched by a web search engine falls under the category of deep web.

Web archiving is the process of collecting portions of the World Wide Web to ensure the information is preserved in an archive for future researchers, historians, and the public. Web archivists typically employ web crawlers for automated capture due to the massive size and amount of information on the Web. The largest web archiving organization based on a bulk crawling approach is the Wayback Machine, which strives to maintain an archive of the entire Web.

Instant indexing is a feature offered by Internet search engines that enables users to submit content for immediate inclusion into the index.

A focused crawler is a web crawler that collects Web pages that satisfy some specific property, by carefully prioritizing the crawl frontier and managing the hyperlink exploration process. Some predicates may be based on simple, deterministic and surface properties. For example, a crawler's mission may be to crawl pages from only the .jp domain. Other predicates may be softer or comparative, e.g., "crawl pages about baseball", or "crawl pages with large PageRank". An important page property pertains to topics, leading to 'topical crawlers'. For example, a topical crawler may be deployed to collect pages about solar power, swine flu, or even more abstract concepts like controversy while minimizing resources spent fetching pages on other topics. Crawl frontier management may not be the only device used by focused crawlers; they may use a Web directory, a Web text index, backlinks, or any other Web artifact.

<span class="mw-page-title-main">Bing Webmaster Tools</span>

Bing Webmaster Tools is a free service as part of Microsoft's Bing search engine which allows webmasters to add their websites to the Bing index crawler, see their site's performance in Bing and a lot more. The service also offers tools for webmasters to troubleshoot the crawling and indexing of their website, submission of new URLs, Sitemap creation, submission and ping tools, website statistics, consolidation of content submission, and new content and community resources.

<span class="mw-page-title-main">LeapFish</span>

LeapFish.com was a search aggregator that retrieved results from other portals and search engines, including Google, Bing and Yahoo!, and also search engines of blogs, videos etc. It was a registered trademark of Dotnext Inc, launched on 3 November 2008.

References

  1. ↑ Wright, Alex (2009-02-22). "Exploring a 'Deep Web' That Google Can't Grasp". The New York Times . Retrieved 2009-02-23.
  2. ↑ Franke, Susanne (2009-02-24). "DeepPeep: Forscher wollen verborgene Datenbanken im Web zugänglich machen" [DeepPeep: Researchers want to make hidden databases accessible on the web]. Comp. Ztg. Retrieved 2009-02-25– via lanline.de.
  3. ↑ Warwick, Martyn (2009-02-25). "DeepPeep lets light in to the hidden Web". TelecomTV. Retrieved 2009-02-25.[ permanent dead link ]
  4. ↑ Sawant, Nimish (2010-03-09). "Crawling the deep web". LiveMint. Mint . Retrieved 2010-12-13.
  5. ↑ "Main Page". WebDB. University of Utah School of Computing. 2008-10-04. Archived from the original on 2009-02-27. Retrieved 2009-02-23.
  6. ↑ Pichler, Thomas (2009-02-23). "Suchansätze dringen in die Tiefen des Internets: Erforschen von Datenbanken als wichtiger Schritt" [Search phrases penetrate the depths of the Internet: Researching databases as an important step] (in German). Pressetext. Retrieved 2009-02-23.
  7. ↑ "Suchansätze dringen in die Tiefen des Internets" [Search phrases penetrate the depths of the Internet]. nachrichten.ch (in German). 2009-02-24. Archived from the original on 2011-07-07. Retrieved 2010-12-13.
  8. ↑ "Award Abstract #0713637: III-COR: Discovering and Organizing Hidden-Web Sources". NSF Award Search. National Science Foundation. Retrieved 2009-02-23.
  9. ↑ "Esplorando il DeepWeb, i fondali della Rete dove Google non arriva" [Exploring the DeepWeb, the depths of the Net where Google does not arrive]. Liberta di Stampa Diritto all'Informazione (This is an Italian translation of the New York Times article "Exploring a ‘Deep Web’ That Google Can’t Grasp" by Alex Wright) (in Italian). Italy. 2009-04-05. Retrieved 2009-03-05.
  10. ↑ Sándor, Berta (2009-02-24). "Az internet mélyét kutatja a DeepPeep" [The internet researching the depths of DeepPeep]. sg.hu (in Hungarian). SG (Hungary). Retrieved 2009-03-05.
  11. ↑ "Niet alles is te vinden met Google" [Not everything can be found with Google] (in Dutch). Dutch Cowboys. 2009-03-04. Retrieved 2009-03-05.
  12. ↑ "探索谷歌尚未把持的'深层网络'" [Explore Google's not yet dominated 'deep network'] (This is a Chinese translation of the New York Times article "Exploring a ‘Deep Web’ That Google Can’t Grasp" by Alex Wright) (in Chinese). 2006-03-03. Archived from the original on 2011-07-07. Retrieved 2009-03-05.
  13. ↑ "Sfida al deep web: la Kosmix prova a svelare le pagine nascoste di internet" [Challenge to the deep web: Kosmix tries to reveal the hidden pages of the internet]. Messagg. 2009-02-23. Archived from the original on 2012-08-04. Retrieved 2010-12-13.
  14. 1 2 3 4 5 Barbosa, Luciano; Nguyen, Hoa; Nguyen, Thanh; Pinnamaneni, Ramesh; Freire, Juliana (2010-01-01). "Creating and exploring web form repositories". Proceedings of the 2010 ACM SIGMOD International Conference on Management of data. SIGMOD '10. New York, NY, USA: ACM. pp. 1175–1178. doi:10.1145/1807167.1807311. ISBN   9781450300322. S2CID   15471440.
  15. ↑ "ViDA-NYU/ache". GitHub. Retrieved 2016-11-06.
  16. ↑ Duygulu, Pinar (1999-12-22). Lopresti, Daniel P.; Zhou, Jiangying (eds.). "Hierarchical representation of form documents for identification and retrieval" . Proceedings of SPIE. Document Recognition and Retrieval VII. 3967 (1): 128. Bibcode:1999SPIE.3967..128D. doi:10.1117/12.373486. ISSN   0277-786X. S2CID   28128295.
  17. ↑ Beckett, Andy (2009-11-25). "The dark side of the internet". The Guardian. ISSN   0261-3077 . Retrieved 2016-11-06.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.