The Loebner Prize was an annual competition in artificial intelligence that awarded prizes to the computer programs considered by the judges to be the most human-like. The format of the competition was that of a standard Turing test. In each round, a human judge simultaneously held textual conversations with a computer program and a human being via computer. Based upon the responses, the judge would attempt to determine which was which.
For the final 2019 competition, the format changed. There was no panel of judges. Instead, the chatbots were judged by the public and there were to be no human competitors.[4] The prize has been reported as defunct as of 2020.[5]
Prizes
Originally, $2,000 was awarded for the most human-seeming program in the competition. The prize was $3,000 in 2005 and $2,250 in 2006. In 2008, $3,000 was awarded.
In addition, there were two one-time-only prizes that have never been awarded. $25,000 is offered for the first program that judges cannot distinguish from a real human and which can convince judges that the human is the computer program. $100,000 is the reward for the first program that judges cannot distinguish from a real human in a Turing test that includes deciphering and understanding text, visual, and auditory input. The competition was planned to end after the achievement of this prize.
Competition rules and restrictions
The rules varied over the years and early competitions featured restricted conversation Turing tests[6] but since 1995 the discussion has been unrestricted.
For the three entries in 2007, Robert Medeksza, Noah Duncan and Rollo Carpenter,[7] some basic "screening questions" were used by the sponsor to evaluate the state of the technology. These included simple questions about the time, what round of the contest it is, etc.; general knowledge ("What is a hammer for?"); comparisons ("Which is faster, a train or a plane?"); and questions demonstrating memory for preceding parts of the same conversation. "All nouns, adjectives and verbs will come from a dictionary suitable for children or adolescents under the age of 12." Entries did not need to respond "intelligently" to the questions to be accepted.
For the first time in 2008 the sponsor allowed introduction of a preliminary phase to the contest opening up the competition to previously disallowed web-based entries judged by a variety of invited interrogators. The available rules do not state how interrogators are selected or instructed. Interrogators (who judge the systems) have limited time: 5 minutes per entity in the 2003 competition, 20+ per pair in 2004–2007 competitions, 5 minutes to conduct simultaneous conversations with a human and the program in 2008–2009, increased to 25 minutes of simultaneous conversation since 2010.
Criticisms
The prize has long been scorned by experts in the field,[8] for a variety of reasons.
It is regarded by many as a publicity stunt.[9][10] Marvin Minsky scathingly offered a "prize" to anyone who could stop the competition. Loebner responded by jokingly observing that Minsky's offering a prize to stop the competition effectively made him a co-sponsor.[11]
The rules of the competition have encouraged poorly qualified judges to make rapid judgements. Interactions between judges and competitors was originally very brief, for example effectively 2.5 mins of questioning, which permitted only a few questions.[9] Questioning was initially restricted to a single topic of the contestant's choice, such as "whimsical conversation",[8][12] a domain suiting standard chatbot tricks.[13]
Competition entrants do not aim at understanding or intelligence but resort to basic ELIZA style tricks,[9][14] and successful entrants find deception and pretense is rewarded.[15]
Contests
2006
In 2006, the contest was organised by Tim Child (CEO of Televirtual) and Huma Shah.[7][16] On August 30, the four finalists were announced:
Rollo Carpenter
Richard Churchill and Marie-Claire Jenkins
Noah Duncan
Robert Medeksza
The contest was held on 17 September in the VR theatre, Torrington Place campus of University College London. The judges included the University of Reading's cybernetics professor, Kevin Warwick, a professor of artificial intelligence, John Barnden (specialist in metaphor research at the University of Birmingham), a barrister, Victoria Butler-Cole and a journalist, Graham Duncan-Rowe. The latter's experience of the event can be found in an article in Technology Review.[17][18] The winner was 'Joan', based on Jabberwacky, both created by Rollo Carpenter.
2007
The 2007 competition was held on October 21 in New York City. The judges were: computer science professor Russ Abbott, philosophy professor Hartry Field, psychology assistant professor Clayton Curtis and English lecturer Scott Hutchins.[19]
No bot passed the Turing test, but the judges ranked the three contestants as follows:
1st: Robert Medeksza, creator of Ultra Hal
2nd: Noah Duncan, a private entry, creator of Cletus
3rd: Rollo Carpenter from Icogno, creator of Jabberwacky
The winner received $2,250 and the annual medal. The runners-up received $250 each.
2008
The 2008 competition was organised by professor Kevin Warwick, coordinated by Huma Shah and held on October 12 at the University of Reading, UK.[20] After testing by over one hundred judges during the preliminary phase, in June and July 2008, six finalists were selected from thirteen original entrant artificial conversational entities (ACEs). Five of those invited competed in the finals:
Eugene Goostman, Vladimir Veselov, Eugene Demchenko and Sergey Ulasen
Jabberwacky, Rollo Carpenter
Ultra Hal, Robert Medeksza
In the finals, each of the judges was given five minutes to conduct simultaneous, split-screen conversations with two hidden entities. Elbot[21] of Artificial Solutions[22] won the 2008 Loebner Prize bronze award, for most human-like artificial conversational entity, through fooling three of the twelve judges who interrogated it (in the human-parallel comparisons) into believing it was human. This is coming very close to the 30% traditionally required to consider that a program has actually passed the Turing test. Eugene Goostman[23] and Ultra Hal[24] both deceived one judge each that it was the human.
Will Pavia, a journalist for The Times, has written about his experience; a Loebner finals' judge, he was deceived by Elbot and Eugene.[25] Kevin Warwick and Huma Shah have reported on the parallel-paired Turing tests.[26]
2009
The 2009 Loebner Prize Competition was held September 6, 2009, at the Brighton Centre, Brighton UK in conjunction with the Interspeech 2009 conference. The prize amount for 2009 was $3,000.
Entrants were David Levy, Rollo Carpenter, and Mohan Embar, who finished in that order.
The writer Brian Christian participated in the 2009 Loebner Prize Competition as a human confederate, and described his experiences at the competition in his book The Most Human Human.
2010
The 2010 Loebner Prize Competition was held on October 23 at California State University, Los Angeles. The 2010 competition was the 20th running of the contest. The winner was Bruce Wilcox with Suzette.
2011
The 2011 Loebner Prize Competition was held on October 19 at the University of Exeter, Devon, United Kingdom. The prize amount for 2011 was $4,000.
The four finalists and their chatterbots were Bruce Wilcox (Rosette), Adeena Mignogna (Zoe), Mohan Embar (Chip Vivant) and Ron Lee (Tutor), who finished in that order.
That year there was an addition of a panel of junior judges, namely Georgia-Mae Lindfield, William Dunne, Sam Keat and Kirill Jerdev. The results of the junior contest were markedly different from the main contest, with chatterbots Tutor and Zoe tying for first place and Chip Vivant and Rosette coming in third and fourth place, respectively.
2012
The 2012 Loebner Prize Competition was held on the 15th of May in Bletchley Park in Bletchley, Buckinghamshire, England, in honor of the Alan Turing centenary celebrations. The prize amount for 2012 was $5,000. The local arrangements organizer was David Levy, who won the Loebner Prize in 1997 and 2009.
The four finalists and their chatterbots were Mohan Embar (Chip Vivant), Bruce Wilcox (Angela), Daniel Burke (Adam), M. Allan (Linguo), who finished in that order.
That year, a team from the University of Exeter's computer science department (Ed Keedwell, Max Dupenois and Kent McClymont) conducted the first-ever live webcast of the conversations.[27]
The four finalists and their chatbots were Steve Worswick (Mitsuku), Dr. Ron C. Lee (Tutor), Bruce Wilcox (Rose) and Brian Rigsby (Izar), who finished in that order.
For the 2013 Junior Loebner Prize Competition the chatbots Mitsuku and Tutor tied for first place with Rose and Izar in 3rd and 4th place respectively.
2014
The 2014 Loebner Prize Competition was held at Bletchley Park, England, on Saturday 15 November 2014. The event was filmed live by Sky News. The guest judge was television presenter and broadcaster James May.
After 2 hours of judging, 'Rose' by Bruce Wilcox was declared the winner. Bruce will receive a cheque for $4000 and a bronze medal. The ranks were as follows:
Rose – Rank 1 ($4000 & Bronze Medal); Izar – Rank 2.25 ($1500); Uberbot – Rank 3.25 ($1000); and Mitsuku – Rank 3.5 ($500).
The Judges were Dr Ian Hocking, Writer & Senior Lecturer in Psychology, Christ Church College, Canterbury; Dr Ghita Kouadri-Mostefaoui, Lecturer in Computer Science and Technology, University of Bedfordshire; Mr James May, Television Presenter and Broadcaster; and Dr Paul Sant, Dean of UCMK, University of Bedfordshire.
2015
The 2015 Loebner Prize Competition was again won by 'Rose' by Bruce Wilcox.[28]
The judges were Jacob Aaron, Physical sciences reporter for New Scientist; Rory Cellan-Jones, Technology correspondent for the BBC; Brett Marty, Film Director and Photographer; Ariadne Tampion, Writer.
2016
The 2016 Loebner Prize was held at Bletchley Park on 17 September 2016. After 2 hours of judging the final results were announced. The ranks were as follows:
The 2017 Loebner Prize was held at Bletchley Park on 16 September 2017. This was the first contest where a new message by message protocol was used, rather than the traditional one character at a time. The ranks were as follows, and were announced by a Nao_(robot):
The 2018 Loebner Prize was held at Bletchley Park on 8 September 2018. This was the final time it would be held in its traditional Turing Test format and its final time at Bletchley Park. The ranks were as follows:
The 2019 Loebner Prize was held at the University of Swansea from 12th–15th September, as part of a larger exhibition which looked at creativity in computers. The format of the contest changed from being a traditional Turing Test, with selected judges and humans, into a 4 day testing session where members of the general public, including schoolchildren, could interact with the bots, knowing in advance that the bots were not humans. Seventeen bots took part instead of the usual 4 finalists. Steve Worswick won for a record 5th time with Mitsuku, which enabled him to be included in the Guinness Book of Records.[30]
A selected jury of judges also examined and voted for the ones they liked best. The ranks were as follows:
Marvin Lee Minsky was an American cognitive and computer scientist concerned largely with research of artificial intelligence (AI). He co-founded the Massachusetts Institute of Technology's AI laboratory and wrote several texts concerning AI and philosophy.
A chatbot is a software application or web interface designed to have textual or spoken conversations. Modern chatbots are typically online and use generative artificial intelligence systems that are capable of maintaining a conversation with a user in natural language and simulating the way a human would behave as a conversational partner. Such chatbots often use deep learning and natural language processing, but simpler chatbots have existed for decades.
Hugh Loebner was an American inventor and social activist, who was notable for sponsoring the Loebner Prize, an embodiment of the Turing test. Loebner held six United States Patents, and was also an outspoken advocate for the decriminalization of prostitution.
Jabberwacky is a chatbot created by British programmer Rollo Carpenter and launched in 1997. Its stated aim is to "simulate natural human chat in an interesting, entertaining and humorous manner". It is an early attempt at creating an artificial intelligence through human interaction.
Ned Joel Block is an American philosopher working in philosophy of mind who has made important contributions to the understanding of consciousness and the philosophy of cognitive science. He has been professor of philosophy and psychology at New York University since 1996.
Robby Garner is an American natural language programmer and software developer. He won the 1998 and 1999 Loebner Prize contests with the program called Albert One. He is listed in the 2001 Guinness Book of World Records as having written the "most human" computer program.
Albert One is an artificial intelligence chatbot created by Robby Garner and designed to mimic the way humans make conversations using a multi-faceted approach in natural language programming.
A.L.I.C.E., also referred to as Alicebot, or simply Alice, is a natural language processing chatterbot—a program that engages in a conversation with a human by applying some heuristical pattern matching rules to the human's input. It was inspired by Joseph Weizenbaum's classical ELIZA program.
The Verbot (Verbal-Robot) was a popular chatbot program and artificial intelligence software development kit (SDK) for Windows and web.
There are a number of competitions and prizes to promote research in artificial intelligence.
Artificial stupidity is a term used within the field of computer science to refer to a technique of "dumbing down" computer programs in order to deliberately introduce errors in their responses.
The computer game bot Turing test is a variant of the Turing test, where a human judge viewing and interacting with a virtual world must distinguish between other humans and video game bots, both interacting with the same virtual world. This variant was first proposed in 2008 by Associate Professor Philip Hingston of Edith Cowan University, and implemented through a tournament called the 2K BotPrize.
The confederate effect is the phenomenon of people falsely classifying human intelligence as machine intelligence during Turing tests. For example, in the Loebner Prize during which a tester conducts a text exchange with one human and one artificial-intelligence chatbot and is tasked to identify which is which, the confederate effect describes the tester inaccurately identifying the human as the machine.
Eugene Goostman is a chatbot that some regard as having passed the Turing test, a test of a computer's ability to communicate indistinguishably from a human. Developed in Saint Petersburg in 2001 by a group of three programmers, the Russian-born Vladimir Veselov, Ukrainian-born Eugene Demchenko, and Russian-born Sergey Ulasen, Goostman is portrayed as a 13-year-old Ukrainian boy—characteristics that are intended to induce forgiveness in those with whom it interacts for its grammatical errors and lack of general knowledge.
The Turing test, originally called the imitation game by Alan Turing in 1949, is a test of a machine's ability to exhibit intelligent behaviour equivalent to, or indistinguishable from, that of a human. Turing proposed that a human evaluator would judge natural language conversations between a human and a machine designed to generate human-like responses. The evaluator would be aware that one of the two partners in conversation was a machine, and all participants would be separated from one another. The conversation would be limited to a text-only channel, such as a computer keyboard and screen, so the result would not depend on the machine's ability to render words as speech. If the evaluator could not reliably tell the machine from the human, the machine would be said to have passed the test. The test results would not depend on the machine's ability to give correct answers to questions, only on how closely its answers resembled those a human would give. Since the Turing test is a test of indistinguishability in performance capacity, the verbal version generalizes naturally to all of human performance capacity, verbal as well as nonverbal (robotic).
Bruce Wilcox is an artificial intelligence programmer.
The Alan Turing Year, 2012, marked the celebration of the life and scientific influence of Alan Turing during the centenary of his birth on 23 June 1912. Turing had an important influence on computing, computer science, artificial intelligence, developmental biology, and the mathematical theory of computability and made important contributions to code-breaking during the Second World War. The Alan Turing Centenary Advisory committee (TCAC) was originally set up by Professor Barry Cooper
Cleverbot is a chatterbot web application. It was created by British AI scientist Rollo Carpenter and launched in October 2008. It was preceded by Jabberwacky, a chatbot project that began in 1988 and went online in 1997. In its first decade, Cleverbot held several thousand conversations with Carpenter and his associates. Since launching on the web, the number of conversations held has exceeded 150 million. Besides the web application, Cleverbot is also available as an iOS, Android, and Windows Phone app.
The Winograd schema challenge (WSC) is a test of machine intelligence proposed in 2012 by Hector Levesque, a computer scientist at the University of Toronto. Designed to be an improvement on the Turing test, it is a multiple-choice test that employs questions of a very specific structure: they are instances of what are called Winograd schemas, named after Terry Winograd, professor of computer science at Stanford University.
1 2 3 Floridi, Luciano; Taddeo, Mariarosaria; Turilli, Matteo (2009). "Turing's Imitation Game: Still an Impossible Challenge for All Machines and Some Judges––An Evaluation of the 2008 Loebner Contest". Minds and Machines. 19 (19): 145–150. CiteSeerX10.1.1.672.7417. doi:10.1007/s11023-008-9130-6. S2CID7888490.
This page is based on this Wikipedia article Text is available under the CC BY-SA 4.0 license; additional terms may apply. Images, videos and audio are available under their respective licenses.
