Knowledge Engine (search engine)

Last updated
The search example of the Knowledge Engine states "Ad-free, secure, non-profit: Make Wikipedia your default search". Wikipedia Search April 2015.png
The search example of the Knowledge Engine states "Ad-free, secure, non-profit: Make Wikipedia your default search".

Knowledge Engine (KE) was a search engine project initiated in 2015 by the Wikimedia Foundation (WMF) to locate and display verifiable and trustworthy information from public-information sources [2] in a way that was less reliant on traditional search engines. [3] It aimed to allow readers to stay on Wikipedia.org and other Wikipedia-related projects when looking for additional information rather than turning to proprietary search engines. [3] Its goal was to protect user privacy, to be open and transparent about how its information originates, and to allow access to related metadata. [4]

Contents

The project was controversial internally, and ended early in 2016. Related ideas were applied to the internal cross-wiki search engine for Wikimedia projects. [1]

History

In 2015, WMF applied for a $250,000 grant from the Knight Foundation to support development of the Knowledge Engine. Its grant proposal noted: "Commercial search engines dominate search-engine use of the Internet, and they're employing proprietary technologies to consolidate channels of access to the Internet's knowledge and information." [5] The project was designed in four stages, each scheduled to take about 18 months. [6]

The project planned to draw information from Wikipedia-related projects and eventually to search other sources of public information such as the U.S. Census Bureau. [5] Leaked internal WMF documents stated the "Knowledge Engine By Wikipedia will democratize the discovery of media, news and information—it will make the Internet's most relevant information more accessible and openly curated, and it will create an open data engine that's completely free of commercial interests. Our new site will be the Internet's first transparent search engine, and the first one that carries the reputation of Wikipedia and the Wikimedia Foundation." [2] The new search engine was not expected to immediately replace a general purpose search engine because at first it would only draw on information from Wikipedia and its other free knowledge projects, [5] though it might in time also have included academic and open access sources in its search results. [7] Matt Southern in Search Engine Journal attributed media confusion about the Knowledge Engine's scope to the fact that later WMF statements clarifying the organization's intentions were "quite a contrast to the original grant application documents". [8]

The project was not discussed publicly with the Wikipedia community while developing the concept, [9] nor part of the existing annual plan. [10] This secrecy was mirrored by a degree of confusion within the organization, and seen as at odds with the goal of transparency. [2] An initial blogpost by WMF Executive Director Lila Tretikov about the project did not address why the original proposal was so much broader than an internal search engine. [11] Some staff and WMF board members felt the WMF was still not being straightforward with the Wikipedia community. [12] This led to a crisis for the organization, [1] leading to Tretikov's resignation in February 2016. [11] [13]

Design

Example of federated data sources potentially used by the Knowledge Engine. Discovery Year 0-1-2 page 9.pdf
Example of federated data sources potentially used by the Knowledge Engine.

The Knowledge Engine had been intended to supplant proprietary search engines, instead showing how a piece of information originated and allowing access to metadata. [4] It would not have had advertisements, and it would have protected users' private data and emphasized collaboration. [4] It would have drawn information from Wikipedia-related projects, as well as potentially other sources of public information such as the U.S. Census Bureau, [5] OpenStreetMap, [14] the Digital Public Library of America, [15] and external sources like Fox News. [1] Jimmy Wales and the WMF stated that the project would focus on improving search on Wikipedia and related Wikimedia projects. [2] The grant application stated that it would "create a model for surfacing high quality, public information on the internet", competing with commercial search engines. [2]

Development timeline

Information about the project was publicized gradually. [2] As early as May 2015, community members asked about the concentration of staff in a new "Search and Discovery" department, though public plans made little or no reference to this work. [2] [15] The grant was applied for in mid-2015 and awarded in September, but only publicly announced in a January 2016 press release. [3]

The project plan had four stages, each scheduled to take about 18 months: Discovery, Advisory, Community and Extension. [6] The initial stage of the project was budgeted to cost $2.5 million, [16] with the whole running to the tens of millions. [1] After a year, the WMF was to evaluate development to date, and at the close of the grant, set plans for the project to continue to the second stage. [6]

Motivation and scope

Since mid-2012, Google Search has included fast facts from Wikipedia articles on its search results pages via the Google Knowledge Graph. Google Knowledge Panel.png
Since mid-2012, Google Search has included fast facts from Wikipedia articles on its search results pages via the Google Knowledge Graph.

A central source of confusion for the project was the extent to which it would directly compete with traditional search engines as a place to search the Web. According to Vice , "the Wikimedia Foundation, the nonprofit that finances and founded Wikipedia, is interested in creating a search engine that appears squarely aimed at competing with Google." [2] According to The Guardian , "there was considerable doubt over what the tool was actually intended to be: a search engine aimed at halting a decline in Wikipedia traffic sent by Google, or simply a service for searching within Wikipedia?" [11]

Since 2012, Google Search and other search engines had started highlighting brief informational summaries from Wikipedia in knowledge panels alongside search results, reducing traffic to Wikipedia from those search engines. [2] According to Search Engine Watch, this led to a battle for attention, [1] and this project could have recouped some of that traffic.

Leaked internal documents from early concepts framed the plan more boldly than the final public description. [18] They said the "Knowledge Engine By Wikipedia will democratize the discovery of media, news and information—it will make the Internet's most relevant information more accessible and openly curated, and it will create an open data engine that's completely free of commercial interests. Our new site will be the Internet's first transparent search engine, and the first one that carries the reputation of Wikipedia and the Wikimedia Foundation." [2]

The apparent contradiction between different descriptions of the purpose led to confusion in the media and in the community. In response to speculation, the WMF published a response clarifying its intentions: "We're not building a global crawler search engine ... Despite headlines, we are not trying to compete with other platforms, including Google. As a non-profit we are noncommercial and support open knowledge. Our focus is on the knowledge contributed on the Wikimedia projects. ... We intend to research how Wikimedia users seek, find, and engage with content. This essential information will allow us to make critical improvements to discovery on the Wikimedia projects." [8] Director of Discovery Tomasz Finc added "we are building an internal search engine, and we are not building a broad one." [1] Jimmy Wales stated that suggestions that the WMF is creating a rival to Google are "trolling", "completely and utterly false", and "a total lie", [2] [19] while allowing that the Knowledge Engine might in time include academic and open access sources in its search results. [7]

Matt Southern in Search Engine Journal attributed media confusion about the KE's scope to the fact that this was "quite a contrast to the original grant application documents", [8] an assessment echoed by James Vincent in The Verge , [9] Matt McGee in Search Engine Land , [20] and Jason Koebler in Vice . [21]

Controversy

Many in the community were furious that details of such a large project had been withheld by an organization that prides itself on radical transparency. Wikimedia's public story—that it was never working on a search engine—was directly contradicted by a grant proposal made to the Knight Foundation and leaked internal documents.

 —Jason Koebler, Vice [12]

Large-scale WMF projects are almost always discussed publicly with the Wikipedia community, but this did not happen with the Knowledge Engine development. [9] Wikipedians were unaware of the existence of the project as a concept, [2] [22] and the KE project was not mentioned in the WMF's annual plan. [10] According to the English Wikipedia's community newsletter, The Signpost , [23] some community members expressed outrage at the perceived secrecy around it and their lack of ability to give input, and this raised questions about WMF's commitment to transparency with the Wikipedia community. [9]

James Heilman, a member of the WMF's Board of Trustees, noted in The Signpost that while on the Board, he had insisted multiple times that the grant documentation be made public, without success. [19] He was dismissed from the Board in December 2015, and it was suggested that his push for transparency concerning the grant had been a factor in his dismissal—a suggestion rejected by Jimmy Wales. [2] The Wikipedia community re-elected Heilman to the Board in 2017. [24]

Ruth McCambridge said in Nonprofit Quarterly, "Wikipedia editors have been requesting from December for the grant proposal and grant letter for a project that many surmise is a bid to remain technologically cutting-edge by the Wikimedia Foundation, but which may divert resources and attention from other pressing needs of the community." [23]

Commenting on the reluctance to share the grant documents with the community, referencing privacy concerns, McCambridge saw "a major difference in culture and values assumptions" compared to previous Wikimedia practice. [23] McCambridge said that "the power of important strategic decisions" here seemed to rest "between funders and the top of the organizational hierarchy" and was "not shared with volunteer editors." [23]

The WMF initially published only portions of the grant documentation, [25] later making the full grant agreement available in February. [19] Further internal documents were leaked shortly after. [2] [9] The full agreement clarified the initial concept for the first stage of the project. [23] Tretikov said she regretted being so late in informing the Wikipedia editing community about the grant. [15]

Longtime Wikipedia editor and journalist William Beutler told Vice Magazine 's Jason Koebler, "Leaving aside whether a search engine is a good idea, let alone feasible, the core issue here is about transparency. The irony is that the Wikimedia Foundation failed to observe one of the movement's own core values ...." [21] UK Wikipedia editor Ashley van Haeften told Ars Technica via e-mail that "Lila, Jimmy, and the rest chose to keep the project and the Knight Foundation application and grant a secret until the projects were underway for six months, and even then this only came to light because it was leaked." [18]

Tretikov's initial public post about the Knowledge Engine project did not explain why the original grant proposal had such a grander vision than the later public plan to develop an internal search engine. [11] Staff who had been uncomfortable about the project's development felt the WMF was not being sufficiently straightforward with the community. [12] According to statements posted of an internal meeting on the WMF's website, [12] a member of the Discovery team member said to Tretikov, "My concern is that we still aren't communicating it clearly enough. This morning's blog post is the truth, but not all of the truth. Namely that we had big plans in the past. It would have been much easier to say that we did have big plans, but they were ditched ... we still haven't acknowledged it. We can't deny it."

Former deputy director of the WMF Erik Möller, up to April 2015, portrayed the events as "very much out of control" and "a crisis." [1] Disagreements about the project, and the response to the resulting controversy, led to many WMF staff members departing, [26] [27] culminating in Tretikov resigning on February 25, 2016. [11] [13]

Related Research Articles

<span class="mw-page-title-main">Wikimedia movement</span> Group of global contributors to Wikimedia projects

The Wikimedia movement is the global community of contributors to the Wikimedia projects, including Wikipedia. This community directly builds and administers these projects with the commitment of achieving this using open standards and software.

<span class="mw-page-title-main">Jimmy Wales</span> Co-founder of Wikipedia (born 1966)

Jimmy Donal Wales, also known as Jimbo Wales, is an Internet entrepreneur, webmaster, and former financial trader. He is a co-founder of the non-profit free encyclopedia, Wikipedia, and the for-profit wiki hosting service Fandom. He has worked on other online projects, including Bomis, Nupedia, WikiTribune, and WT Social.

<span class="mw-page-title-main">Wikia Search</span> Defunct free and open-source web search engine by Wikia

Wikia Search was a short-lived free and open-source web search engine launched by Wikia, a for-profit wiki-hosting company founded by Jimmy Wales and Angela Beesley. Wikia Search followed other experiments by Wikia into search engine technology and officially launched as a "public alpha" on January 7, 2008. The roll-out version of the search interface was widely criticized by reviewers in mainstream media.

<span class="mw-page-title-main">Wikivoyage</span> Free travel guide that anyone can edit

Wikivoyage is a free web-based travel guide for travel destinations and travel topics written by volunteer authors. It is a sister project of Wikipedia and supported and hosted by the same non-profit Wikimedia Foundation (WMF). Wikivoyage has been called the "Wikipedia of travel guides".

<span class="mw-page-title-main">History of wikis</span> History of wiki collaborative platforms

The history of wikis began in 1994, when Ward Cunningham gave the name "WikiWikiWeb" to the knowledge base, which ran on his company's website at c2.com, and the wiki software that powered it. The wiki went public in March 1995, the date used in anniversary celebrations of the wiki's origins. c2.com is thus the first true wiki, or a website with pages and links that can be easily edited via the browser, with a reliable version history for each page. He chose "WikiWikiWeb" as the name based on his memories of the "Wiki Wiki Shuttle" at Honolulu International Airport, and because "wiki" is the Hawaiian word for "quick".

<span class="mw-page-title-main">Erik Möller</span> German journalist and software developer (born 1979)

Erik Möller is a German freelance journalist, software developer, author, and former deputy director of the Wikimedia Foundation (WMF), based in San Francisco. Möller additionally works as a web designer and previously managed his own web hosting service, myoo.de. As of 2022, he was VP of Engineering at the Freedom of the Press Foundation.

<span class="mw-page-title-main">Wikimedia Foundation</span> American charitable organization

The Wikimedia Foundation, Inc., abbreviated WMF, is an American 501(c)(3) nonprofit organization headquartered in San Francisco, California, and registered there as a charitable foundation. It is the host of Wikipedia, the seventh most visited website in the world. In addition, the Foundation hosts fourteen other related content projects. It supports the development of MediaWiki, the wiki software that underpins them all. The Foundation was established in 2003 in St. Petersburg, Florida by Jimmy Wales, as a nonprofit way to fund Wikipedia, Wiktionary, and other crowdsourced wiki projects. Until then, they had been hosted by Bomis, Wales's for-profit company.

<span class="mw-page-title-main">Knol</span> Google publishing project (2007–2011)

Knol was a Google project that aimed to include user-written articles on a range of topics. The lower-case term knol, which Google defined as a "unit of knowledge", referred to an article in the project. Knol was often viewed as a rival to Wikipedia.

<span class="mw-page-title-main">Wikimania</span> Annual conference of the Wikimedia movement

Wikimania is the Wikimedia movement's annual conference, organized by volunteers and hosted by the Wikimedia Foundation. Topics of presentations and discussions include Wikimedia projects such as Wikipedia, other wikis, open-source software, free knowledge and free content, and social and technical aspects related to these topics.

<span class="mw-page-title-main">Sue Gardner</span> Canadian journalist and past executive director of Wikimedia Foundation

Sue Gardner is a Canadian journalist, not-for-profit executive and business executive. She was the executive director of the Wikimedia Foundation from December 2007 until May 2014, and before that was the director of the Canadian Broadcasting Corporation's website and online news outlets.

<span class="mw-page-title-main">Outline of Wikipedia</span> Overview and topical guide to the free online crowdsourced encyclopedia

The following outline is provided as an overview of and a topical guide to Wikipedia:

Conflict-of-interest (COI) editing on Wikipedia occurs when editors use Wikipedia to advance the interests of their external roles or relationships. The type of COI editing that compromises Wikipedia the most is paid editing for public relations (PR) purposes. Several policies and guidelines exist to combat conflict of interest editing, including Wikipedia's conflict of interest guideline and the Wikimedia Foundation's paid-contribution disclosure policy.

<span class="mw-page-title-main">Google Knowledge Graph</span> Knowledge base used by Google to enhance its search engines results

The Google Knowledge Graph is a knowledge base from which Google serves relevant information in an infobox beside its search results. This allows the user to see the answer in a glance, as an instant answer. The data is generated automatically from a variety of sources, covering places, people, businesses, and more.

Wikipedia Zero was a project by the Wikimedia Foundation to provide access to Wikipedia free of charge on mobile phones via zero-rating, particularly in developing markets. The objective of the program was to facilitate access to free knowledge for low-income pupils and students, by means of waiving the network traffic cost. With 97 operators in over 72 countries, it was estimated that access to Wikipedia was provided to more than 800 million people through the program. The program ended in 2018.

<span class="mw-page-title-main">James Heilman</span> Emergency physician and Wikipedia editor

James M. Heilman is a Canadian emergency physician, Wikipedian, and advocate for the improvement of Wikipedia's health-related content. He encourages other clinicians to contribute to the online encyclopedia.

<span class="mw-page-title-main">Lila Tretikov</span> Russian-American engineer and Wikimedian

Lila Tretikov is a Russian-American engineer and manager.

<i>The Signpost</i> English Wikipedias newspaper

The Signpost is the English Wikipedia's online newspaper. Managed by the volunteer community, it is published online with contributions from Wikimedia editors. The newspaper's scope includes the Wikimedia community and events related to Wikipedia, including Arbitration Committee rulings, Wikimedia Foundation issues, and other Wikipedia-related projects. It was founded in January 2005 by Wikipedian Michael Snow, who continued as a contributor until his February 2008 appointment to the Wikimedia Foundation's Board of Trustees.

<span class="mw-page-title-main">Relationship between Google and Wikipedia</span>

The relationship between Google and Wikipedia was collaborative in Wikipedia's early days, when Google helped reduce the pagerank of widespread, uneditable Wikipedia clones that were ostensibly ad farms. In 2007, Google introduced Knol, a direct competitor for community-driven encyclopedia creation, which was subsequently shut down in 2012. Google later supported Wikimedia with numerous grants, and came to rely on Wikipedia for addressing the problem of misinformation on YouTube, providing verifiable and well-sourced information to those seeking it. Google and Wikimedia Enterprise started a partnership in 2021.

References

  1. 1 2 3 4 5 6 7 8 Sentance, Rebecca (March 3, 2016). "Everything you need to know about Wikimedia's 'Knowledge Engine' so far". Search Engine Watch. Archived from the original on January 13, 2017.
  2. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Koebler, Jason (February 16, 2016). "The Secret Search Engine Tearing Wikipedia Apart". Vice . Archived from the original on February 23, 2016.
  3. 1 2 3 McGee, Matt (February 15, 2016). "Wikimedia Foundation Secures $250,000 Grant For Search Engine Development". Search Engine Land . Archived from the original on May 23, 2016.
  4. 1 2 3 Singh, Manish (February 16, 2016). "Wikipedia's Upcoming Search Engine to Rival Google; Offer Full Transparency". Gadgets 360. Archived from the original on February 16, 2016.
  5. 1 2 3 4 Cuthbertson, Anthony (February 16, 2016). "Wikipedia Takes on Google with New 'Transparent' Search Engine". Newsweek . Archived from the original on February 16, 2016.
  6. 1 2 3 Crum, Chris (February 15, 2016). "Wikimedia Works On Search Improvements, Says It's Not Competing with Google [Updated]". WebProNews. Archived from the original on July 3, 2016. Retrieved February 17, 2016.
  7. 1 2 Greis, Friedhelm (February 15, 2016). "Wirbel um angebliche Wikipedia-Konkurrenz zu Google". Golem.de (in German). Archived from the original on February 17, 2016.
  8. 1 2 3 Southern, Matt (February 17, 2016). "Wikimedia Clarifies it is Not Building a Global Web Crawler". Search Engine Journal. Archived from the original on February 18, 2016.
  9. 1 2 3 4 5 Vincent, James (February 17, 2016). "Wikimedia says it's not building a search engine to take on Google". The Verge . Archived from the original on September 5, 2017.
  10. 1 2 McCormick, Rich (February 26, 2016). "Wikimedia head resigns after leak exposed search engine plans". The Verge . Archived from the original on April 14, 2017.
  11. 1 2 3 4 5 Hern, Alex (February 26, 2016). "Head of Wikimedia resigns over search engine plans". The Guardian . Archived from the original on March 28, 2016.
  12. 1 2 3 4 Koebler, Jason (February 25, 2016). "Wikimedia Foundation Executive Director Resigns Amid a Community Revolt". Vice . Archived from the original on February 26, 2016.
  13. 1 2 "Online-Enzyklopädie: Chefin der Wikipedia-Stiftung tritt zurück". Spiegel Online (in German). February 26, 2016. Archived from the original on March 5, 2016.
  14. ↑ Shah, Jaymi (February 16, 2016). "Wikimedia Foundation Secures $250,000 Grant For Search Engine Development". Technoledger. Archived from the original on March 2, 2016.
  15. 1 2 3 Kleinz, Torsten (February 15, 2016). "Wikipedia plant Suchmaschine, aber keinen Google-Konkurrenten". Heinz Heise (in German). Archived from the original on February 17, 2016.
  16. ↑ Orlowski, Andrew (February 12, 2016). "Reluctant Wikipedia lifts lid on $2.5m internet search engine project". The Register . Archived from the original on September 1, 2017.
  17. 1 2 Orlowski, Andrew (January 14, 2014). "Google stabs Wikipedia in the front". The Register . Archived from the original on November 13, 2017.
  18. 1 2 Mullin, Joe (February 29, 2016). "Wikimedia Foundation director resigns after uproar over "Knowledge Engine"". Ars Technica . Archived from the original on March 1, 2016.
  19. 1 2 3 Tual, Morgane (February 16, 2016). "Un projet de moteur de recherche sème la discorde chez Wikipedia". Le Monde (in French).
  20. ↑ McGee, Matt (February 16, 2016). "Wikimedia Foundation: "We're Not Building A Global Crawler Search Engine"". Search Engine Land . Archived from the original on February 17, 2016.
  21. 1 2 Koebler, Jason (February 16, 2016). "Wikimedia: We're Really Really Not Building a Search Engine". Vice . Archived from the original on February 17, 2016.
  22. ↑ Singh, Manish (February 16, 2016). "Knowledge Engine: Wikimedia Foundation takes aim at Google with $3.5m search project". ABC News (Australia) . Archived from the original on February 16, 2016.
  23. 1 2 3 4 5 McCambridge, Ruth (February 16, 2016). "Knight Foundation Grant Request Tears at Wikipedia's Community". Nonprofit Quarterly. Archived from the original on February 24, 2016.
  24. ↑ Andreas Kolbe (7 June 2017). "Golden handshakes of almost half a million at Wikimedia Foundation". The Register. Archived from the original on 10 October 2017.
  25. ↑ Orlowski, Andrew (February 11, 2016). "Move over, Google. Here's Wikipedia's Search Engine – Full of On-Demand Smut". The Register . Archived from the original on July 8, 2017.
  26. ↑ Price, Rob (February 26, 2016). "The executive director of the nonprofit behind Wikipedia just resigned". Business Insider . Archived from the original on February 28, 2016.
  27. ↑ Noisette, Thierry (February 26, 2016). "Crise à la fondation Wikimedia : sa directrice démissionne". L'Obs (in French).
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.