Chess engine

Last updated

In computer chess, a chess engine is a computer program that analyzes chess or chess variant positions, and generates a move or list of moves that it regards as strongest. [1]

Contents

A chess engine is usually a back end with a command-line interface with no graphics or windowing. Engines are usually used with a front end, a windowed graphical user interface such as Chessbase or WinBoard that the user can interact with via a keyboard, mouse or touchscreen. This allows the user to play against multiple engines without learning a new user interface for each, and allows different engines to play against each other.

Many chess engines are now available for mobile phones and tablets, making them even more accessible.

History

The meaning of the term "chess engine" has evolved over time. In 1986, Linda and Tony Scherzer entered their program Bebe into the 4th World Computer Chess Championship, running it on "Chess Engine," their brand name for the chess computer hardware [2] made, and marketed by their company Sys-10, Inc. [3] By 1990 the developers of Deep Blue, Feng-hsiung Hsu and Murray Campbell, were writing of giving their program a 'searching engine,' apparently referring to the software rather than the hardware. [4] In December 1991, Computer-schach & Spiele referred to Chessbase's recently released Fritz as a 'Schach-motor,' the German translation for 'chess engine.' [5] By early 1993, Marty Hirsch was drawing a distinction between commercial chess programs such as Chessmaster 3000 or Battle Chess on the one hand, and 'chess engines' such as ChessGenius or his own MChess Pro on the other. In his characterization, commercial chess programs were low in price, had fancy graphics, but did not place high on the SSDF (Swedish Chess Computer Association) rating lists while engines were more expensive, and did have high ratings. [6]

In 1994, Shay Bushinsky was working on an early version of his Junior program. He wanted to focus on the chess playing part rather than the graphics, and so asked Tim Mann how he could get Junior to communicate with Winboard. Tim's answer formed the basis for what became known as the Chess Engine Communication Protocol or Winboard engines, originally a subset of the GNU Chess command line interface. [7]

Also in 1994, Stephen J. Edwards released the Portable Game Notation (PGN) specification. It mentions PGN reading programs not needing to have a "full chess engine." It also mentions three "graphical user interfaces" (GUI): XBoard, pgnRead and Slappy the database. [8]

By the mid-2000s, engines had become so strong that they were able to beat even the best human players. Except for entertainment purposes, especially using engines with limited strength, matches between humans and engines are now rare; engines are increasingly regarded as tools for analysis rather than as opponents.

Interface protocol

Common Winboard engines would include Crafty,ProDeo (based on Rebel ), Chenard, Zarkov and Phalanx.

In 1995, Chessbase released a version of their database program including Fritz  4 as a separate engine. This was the first appearance of the Chessbase protocol. Soon after, they added the engines Junior and Shredder to their product line up, including engines in CB protocol as separate programs which could be installed in the Chessbase program or one of the other Fritz style GUI's. Fritz 1-14 were only issued as Chessbase engines, while Hiarcs,Nimzo, Chess Tiger and Crafty have been ported to Chessbase format even though they were UCI or Winboard engines. Recently, Chessbase has begun to include Universal Chess Interface (UCI) engines in their playing programs such as Komodo, Houdini, Fritz 15–16 and Rybka rather than convert them to Chessbase engines.

In 2000, Stefan Meyer-Kahlen and Franz Huber released the Universal Chess Interface, a more detailed protocol that introduced a wider set of features. Chessbase soon after dropped support for Winboard engines, and added support for UCI to their engine GUI's and Chessbase programs. Most of the top engines are UCI these days: Stockfish, Komodo, Leela Chess Zero, Houdini, Fritz 15-16, Rybka, Shredder, Fruit, Critter, Ivanhoe and Ruffian.

From 1998, the German company Millenium 2000 briefly moved from dedicated chess computers into the software market, developing the Millennium Chess System (MCS) protocol for a series of CD's containing ChessGenius or Shredder, but after 2001 ceased releasing new software. [9] A more longstanding engine protocol has been used by the Dutch company, Lokasoft, [10] which eventually took over the marketing of Ed Schröder's Rebel.

Increasing strength

Chess engines increase in playing strength continually. This is partly due to the increase in processing power that enables calculations to be made to ever greater depths in a given time. In addition, programming techniques have improved, enabling the engines to be more selective in the lines that they analyze and to acquire a better positional understanding. A chess engine often uses a vast previously-computed opening "book" to increase its playing strength for the first several moves, up to possibly 20 moves or more in deeply analyzed lines.[ citation needed ]

Chess ability of the best computers.png

Some chess engines maintain a database of chess positions, along with previously-computed evaluations and best moves—in effect, a kind of "dictionary" of recurring chess positions. Since these positions are pre-computed, the engine merely plays one of the indicated moves in the database, thereby saving computing time, resulting in stronger, faster play.

Some chess engines use endgame tablebases to increase their playing strength during the endgame. An endgame tablebase includes all possible endgame positions with a small amount of material. Each position is conclusively determined as a win, loss, or draw for the player whose turn it is to move, and the number of moves to the end with best play by both sides. The tablebase identifies for every position the move which will win the fastest against an optimal defense, or the move that will lose the slowest against an optimal offense. Such tablebases are available for all chess endgames with seven pieces or fewer (trivial endgame positions are excluded, such as six white pieces versus a lone black king). [11] [12]

When the maneuvering in an ending to achieve an irreversible improvement takes more moves than the horizon of calculation of a chess engine, an engine is not guaranteed to find the best move without the use of an endgame tablebase, and in many cases can fall foul of the fifty-move rule as a result. Many engines use permanent brain (continuing to calculate during the opponent's turn) as a method to increase their strength.

Distributed computing is also used to improve the software code of chess engines. In 2013, the developers of the Stockfish chess playing program started using distributed computing to make improvements in the software code. [13] [14] [15] As of June 2017, a total of more than 745 years of CPU time has been used to play more than 485 million chess games, with the results being used to make small and incremental improvements to the chess-playing software. [16] In 2019, Ethereal author Andrew Grant started the distributed computing testing framework OpenBench, based upon Stockfish's testing framework, [17] [18] and it is now the most widely-used testing framework for chess engines.[ citation needed ]

Limiting an engine's strength

By the late 1990s, the top engines had become so strong that few players stood a chance of winning a game against them. To give players more of a chance, engines began to include settings to adjust or limit their strength. In 2000, when Stefan Meyer-Kahlen and Franz Huber released the Universal Chess Interface protocol they included the parameters uci_limitstrength and uci_elo allowing engine authors to offer a variety of levels rated in accordance with Elo rating, as calibrated by one of the rating lists. Most GUIs for UCI engines allow users to set this Elo rating within the menus. Even engines that have not adopted this parameter will sometimes have an adjustable strength parameter (e.g. Stockfish 11). Engines which have a uci_elo parameter include Houdini, Fritz 15–16, Rybka, Shredder, Hiarcs, Junior, Zappa, and Sjeng. GUIs such as Shredder, Chess Assistant, Convekta Aquarium, [19] Hiarcs Chess Explorer, and Martin Blume's Arena [20] have dropdown menus for setting the engine's uci_elo parameter. The Fritz family GUIs, Chess Assistant, and Aquarium also have independent means of limiting an engine's strength apparently based on an engine's ability to generate ranked lists of moves (called multipv for 'principle variation').

Comparisons

Tournaments

The results of computer tournaments give one view of the relative strengths of chess engines. However, tournaments do not play a statistically significant number of games for accurate strength determination. In fact, the number of games that need to be played between fairly evenly matched engines, in order to achieve significance, runs into the thousands and is, therefore, impractical within the framework of a tournament. [21] Most tournaments also allow any types of hardware, so only engine/hardware combinations are being compared.

Historically, commercial programs have been the strongest engines. If an amateur engine wins a tournament or otherwise performs well (for example, Zappa in 2005), then it is quickly commercialized. Titles gained in these tournaments garner much prestige for the winning programs, and are thus used for marketing purposes. However, after the rise of volunteer distributed computing projects such as Leela Chess Zero and Stockfish and testing frameworks such as FishTest and OpenBench in the late 2010s, free and open source programs have largely displaced commercial programs as the strongest engines in tournaments.

List of tournaments

Current tournaments include:

Historic tournaments include:

Ratings

Chess engine rating lists aim to provide statistically significant measures of relative engine strength. These lists play multiple games between engines. Some also standardize the opening books, the time controls, and the computer hardware the engines use, in an attempt to measure the strength differences of the engines only. These lists provide not only a ranking, but also margins of error on the given ratings.

The ratings on the rating lists, although calculated by using the Elo system (or similar rating methods), have no direct relation to FIDE Elo ratings or to other chess federation ratings of human players. Except for some man versus machine games which the SSDF had organized many years ago (when engines were far from today's strength), there is no calibration between any of these rating lists and player pools. Hence, the results which matter are the ranks and the differences between the ratings, and not the absolute values.

Missing from many rating lists are IPPOLIT and its derivatives. Although very strong and open source, there are allegations from commercial software interests that they were derived from a disassembled binary of Rybka. [22] Due to the controversy, all these engines have been blacklisted from many tournaments and rating lists. Rybka in turn was accused of being based on Fruit, [23] and in June 2011, the ICGA formally claimed Rybka was derived from Fruit and Crafty and banned Rybka from the International Computer Games Association World Computer Chess Championship, and revoked its previous victories (2007, 2008, 2009, and 2010). [24] The ICGA received some criticism for this decision. [25] Despite all this, Rybka is still included on many rating lists, such as CCRL and CEGT, in addition to Houdini, a derivative of the IPPOLIT derivative Robbolito, [26] and Fire, a derivative of Houdini. In addition, Fat Fritz 2, a derivative of Stockfish, [27] is also included on most of the rating lists.

Differences between rating lists

There are a number of factors that vary among the chess engine rating lists:

  • Number of games. More games when testing each engine result in higher statistical significance.
  • Formulae used to calculate the elo of each engine.
  • Time control:
    • Longer time controls are better suited for determining tournament play strength, but also either make testing more time-consuming or the results less statistically significant.
    • Increment time controls are better suited for determining tournament play strength since tournaments usually use increment time controls, but many rating lists use cyclic/repeating time controls instead.
    • Consistent time controls throughout the rating list vs different time controls for each test. The latter results in a smaller statistical significance than the former because different time controls is a potential confounder. This is particularly problematic for CCRL because CCRL uses both cyclic/repeating time controls (40/15) and increment time controls (15"+10') in its CCRL 40/15 list yet maintains both time controls on the same list. [28]
  • Opponents used in testing engines.
    • Some rating lists only test an engine against the most recent version of each opponent engine, while other rating lists test an engine against the version(s) of each opponent engine closest in elo to the engine being tested.
    • Most rating lists do not test every engine on the rating list vs every other engine on the rating list in a round-robin tournament format. This causes distortions in the rating lists, especially for CCRL and CEGT. [29]
  • Hardware used:
    • Faster hardware with more memory leads to stronger play.
    • 64-bit (vs. 32-bit) hardware and operating systems favor bitboard-based programs
    • Hardware using modern instruction sets such as AVX2 or AVX512 favor engines using vectors and vector intrinsics in their code, common in neural networks.
    • Graphics processing units favor programs with deep neural networks.
    • Multiprocessor vs. single processor hardware.
    • Consistent hardware throughout the rating list vs different hardware for every test. The latter results in a smaller statistical significance than the former because different hardware is a potential confounder. This is particularly problematic for CEGT because multiple testers each with their own unique hardware are involved in testing each engine in CEGT. [30] The same issue arises in CCRL. [31]
  • Ponder settings (speculative analysis while the opponent is thinking) aka Permanent Brain.
  • Transposition table sizes.
  • GUI settings.
  • Opening book settings.

These differences affect the results, and make direct comparisons between rating lists difficult.

List of rating lists

Current rating lists and rating list organizations include:

Historic rating lists and rating list organizations include:

Test suites

Engines can be tested by measuring their performance on specific positions. Typical is the use of test suites where for each given position there is one best move to find. These positions can be geared towards positional, tactical or endgame play. The Nolot test suite, for instance, focuses on deep sacrifices. [32] The BT2450 and BT2630 test suites measure the tactical capability of a chess engine and have been used by REBEL. [33] [34] There is also a general test suite called Brilliancy which was compiled mostly from How to Reassess Your Chess Workbook. [35] The Strategic Test Suite (STS) tests an engine's strategical strength. [36] Another modern test suite is Nightmare II which contains 30 chess puzzles. [37] [ irrelevant citation ]

Kasparov versus the World (chess game played with computer assistance)

In 1999, Garry Kasparov played a chess game called "Kasparov versus the World" over the Internet, hosted by the MSN Gaming Zone. Both sides used computer (chess engine) assistance. The "World Team" included the participation of over 50,000 people from more than 75 countries, deciding their moves by plurality vote. The game lasted four months, ending after Kasparov's 62nd move when he announced a forced checkmate in 28 moves found with the computer program Deep Junior. The World Team voters resigned on October 22. After the game, Kasparov said: "It is the greatest game in the history of chess. The sheer number of ideas, the complexity, and the contribution it has made to chess make it the most important game ever played." [38]

Engines for chess variants

Some chess engines have been developed to play chess variants, adding the necessary code to simulate non-standard chess pieces, or to analyze play on non-standard boards. ChessV and Fairy-Max, for example, are both capable of playing variants on a chessboard up to 12×8 in size, such as Capablanca Chess (10×8 board).

For larger boards, however, there are few chess engines that can play effectively, and indeed chess games played on an unbounded chessboard (infinite chess) are virtually untouched by chess-playing software, although theoretically a program using a MuZero-derived algorithm could handle an unbounded state space.

Graphical user interfaces

XBoard/Winboard was one of the earliest graphical user interfaces (GUI). Tim Mann created it to provide a GUI for the GNU Chess engine, but after that, other engines such as Crafty appeared which used the Winboard protocol. Eventually, the program Chessmaster included the option to import other Winboard engines in addition to the King engine which was included.

In 1995, Chessbase began offering the Fritz engine as a separate program within the Chessbase database program and within the Fritz GUI. Soon after, they added the Junior and Shredder engines to their product line up, packaging them within the same GUI as was used for Fritz. In the late 1990s, the Fritz GUI was able to run Winboard engines via an adapter, but after 2000, Chessbase simply added support for UCI engines, and no longer invested much effort in Winboard.

In 2000, Stefan Meyer-Kahlen started selling Shredder in a separate UCI GUI of his own design, allowing UCI or Winboard engines to be imported into it.

Convekta's Chess Assistant and Lokasoft's ChessPartner also added the ability to import Winboard and UCI engines into their products. Shane Hudson developed Shane's Chess Information Database, a free GUI for Linux, Mac and Windows. Martin Blume developed Arena, [20] another free GUI for Linux and Windows. Lucas Monge entered the field with the free Lucas Chess GUI. [39] All three can handle both UCI and Winboard engines.

On Android, Aart Bik came out with Chess for Android, [40] another free GUI, and Gerhard Kalab's Chess PGN Master [41] and Peter Osterlund's Droidfish [42] can also serve as GUIs for engines.

The Computer Chess Wiki lists many chess GUIs. [43]

See also

Notes

    Related Research Articles

    <span class="mw-page-title-main">Computer chess</span> Computer hardware and software capable of playing chess

    Computer chess includes both hardware and software capable of playing chess. Computer chess provides opportunities for players to practice even in the absence of human opponents, and also provides opportunities for analysis, entertainment and training. Computer chess applications that play at the level of a chess grandmaster or higher are available on hardware from supercomputers to smart phones. Standalone chess-playing machines are also available. Stockfish, Leela Chess Zero, GNU Chess, Fruit, and other free open source applications are available for various platforms.

    <span class="mw-page-title-main">XBoard</span> Graphical user interface for chess games

    XBoard is a graphical user interface chessboard for chess engines under the X Window System. It is developed and maintained as free software by the GNU project. WinBoard is a port of XBoard to run natively on Microsoft Windows.

    <span class="mw-page-title-main">GNU Chess</span> Free software chess engine

    GNU Chess is a free software chess engine and command-line interface chessboard. The goal of GNU Chess is to serve as a basis for research, and as such it has been used in numerous contexts.

    The Universal Chess Interface (UCI) is an open communication protocol that enables chess engines to communicate with user interfaces.

    ChessBase is a German company that develops and sells chess software, maintains a chess news site, and operates an internet chess server for online chess. Founded in 1986, it maintains and sells large-scale databases containing the moves of recorded chess games. The databases contain data from prior games and provide engine analyses of games. Endgame tablebases are also provided by the company.

    <span class="mw-page-title-main">Crafty</span>

    Crafty is a chess program written by UAB professor Robert Hyatt, with development and assistance from Michael Byrne, Tracy Riegle, and Peter Skinner. It is derived from Cray Blitz, winner of the 1983 and 1986 World Computer Chess Championships. Tord Romstad, co-author of Stockfish, described Crafty as "arguably the most important and influential chess program ever".

    <span class="mw-page-title-main">HIARCS</span>

    HIARCS is a proprietary UCI chess engine developed by Mark Uniacke. Its name is an acronym standing for higher intelligence auto-response chess system. Because Hiarcs is written portable in C, it is available on multiple platforms such as Pocket PC, Palm OS, PDAs, iOS, Microsoft Windows and Mac OS X.

    <i>Fritz</i> (chess) Chess software

    Fritz is a German chess program originally developed for Chessbase by Frans Morsch based on his Quest program, ported to DOS, and then Windows by Mathias Feist. With version 13, Morsch retired, and his engine was first replaced by Gyula Horvath's Pandix, and then with Fritz 15, Vasik Rajlich's Rybka.

    <span class="mw-page-title-main">Zappa (chess)</span> Chess software

    Zappa, Zap!Chess or Zappa Mexico, is a UCI chess engine written by Anthony Cozzie, a graduate student at the University of Illinois at Urbana-Champaign. The program emphasizes sound search and a good use of multiple processors. Earlier versions of Zappa are free and the current version is available at Shredder Computer Chess.

    Fruit is a chess engine developed by Fabien Letouzey. In the SSDF rating list released on November 24, 2006, Fruit version 2.2.1 had a rating of 2842. In the CEGT rating list released on January 24, 2007, Fruit version 2.2.1 had a rating of 2776.

    <span class="mw-page-title-main">Rybka</span> Chess engine

    Rybka is a computer chess engine designed by International Master Vasik Rajlich. Around 2011, Rybka was one of the top-rated engines on chess engine rating lists and won many computer chess tournaments.

    The Swedish Chess Computer Association is an organization that tests computer chess software by playing chess programs against one another and producing a rating list. On September 26, 2008, the list was released with Deep Rybka 3 leading with an estimated Elo rating of 3238. Rybka's listing in June 2006 was the first time a program on the list has passed the 2900 mark. In the year 2000 the ratings of the chess engines in the SSDF rating pool were calibrated with games played against humans.

    This article documents the progress of significant human–computer chess matches.

    <span class="mw-page-title-main">Stockfish (chess)</span> Free and open-source chess engine

    Stockfish is a free and open-source chess engine, available for various desktop and mobile platforms. It can be used in chess software through the Universal Chess Interface.

    <span class="mw-page-title-main">Houdini (chess)</span> UCI chess engine

    Houdini is a UCI chess engine developed by Belgian programmer Robert Houdart. It is influenced by open-source engines IPPOLIT/RobboLito, Stockfish, and Crafty. Versions up to 1.5a are available for non-commercial use, while 2.0 and later are commercial only.

    <span class="mw-page-title-main">Komodo (chess)</span> UCI Chess Engine

    Komodo and Dragon by Komodo Chess are UCI chess engines developed by Komodo Chess, which is a part of Chess.com. The engines were originally authored by Don Dailey and GM Larry Kaufman. Dragon is a commercial chess engine, but Komodo is free for non-commercial use. Dragon is consistently ranked near the top of most major chess engine rating lists, along with Stockfish and Leela Chess Zero.

    Strelka is a computer chess engine for Windows, developed by Yuri Osipov. In total five versions of the program have been developed. The current version, 5.5, runs only on a single processor core. Strelka was one of the strongest programs in the world, according to several blitz rating lists and the strongest in Russia at the time.

    Top Chess Engine Championship, formerly known as Thoresen Chess Engines Competition, is a computer chess tournament that has been run since 2010. It was organized, directed, and hosted by Martin Thoresen until the end of Season 6; from Season 7 onward it has been organized by Chessdom. It is often regarded as the Unofficial World Computer Chess Championship because of its strong participant line-up and long time-control matches on high-end hardware, giving rise to very high-class chess. The tournament has attracted nearly all the top engines compared to the World Computer Chess Championship.

    <span class="mw-page-title-main">CuckooChess</span> Chess engine software

    CuckooChess is an advanced free and open-source chess engine under the GNU General Public License written in Java by Peter Österlund. CuckooChess provides an own GUI, and optionally supports the Universal Chess Interface protocol for the use with external GUIs such as Arena. An Android port is available, where its GUI is also based on Peter Österlund's Stockfish port dubbed DroidFish.

    References

    1. blog.chess.com Creating a chess engine from scratch (Part 1: Basics) Archived 2012-03-08 at the Wayback Machine , Link date 28 June 2012
    2. "4th World Computer Chess Championship - New York 1983 (ICGA Tournaments)". Game-ai-forum.org. Archived from the original on 2018-12-02. Retrieved 2018-12-02.
    3. "New SYS-10". Sys-10.com. Archived from the original on 1 July 2022. Retrieved 26 June 2022.
    4. Feng-hsiung Hsu, Thomas Anantharaman, Murray Campbell and Andreas Nowatzyk. "A Grandmaster Chess Machine." Scientific American. October 1990.
    5. Computer-schach & Spiele would later start using the English word 'engine,' eg. 1997#6 when describing Fritz or Hiarcs.
    6. "The True Value of Chess Software." Computer Chess Reports. 1992-3. vol. 3 no. 2 p. 22
    7. "XBoard Project History". Tim-mann.org. Archived from the original on 2018-11-30. Retrieved 2018-11-29.
    8. "Standard: Portable Game Notation Specification and Implementation Guide" (TXT). Thechessdrum.net. Archived from the original on 2007-08-22. Retrieved 2022-06-26.
    9. Computer-schach & Spiele.
    10. "Home - Lokasoft - Home of ChessPartner". Lokasoft.nl. Archived from the original on 26 June 2022. Retrieved 26 June 2022.
    11. http://tb7.chessok.com Archived 2020-12-01 at the Wayback Machine Lomonosov website allowing registered user to access 7-piece tablebase, and a forum with positions found.
    12. "Who wins from this? (chess puzzle)" Archived 2020-11-11 at the Wayback Machine An example chess position found from the Lomonosov chess tablebase.
    13. "Stockfish Testing Framework". Tests.stockfishchess.org. Archived from the original on 22 March 2014. Retrieved 7 March 2014.
    14. "Get Involved". Stockfishchess.org. Archived from the original on 16 June 2013. Retrieved 8 March 2014.
    15. Costalba, Marco (1 May 2013). "Fishtest Distributed Testing Framework". Talkchess.com. Archived from the original on 19 April 2014. Retrieved 18 April 2014.
    16. "Stockfish Testing Framework - Users". Test.stockfishchess.org. Archived from the original on 25 June 2018. Retrieved 17 June 2017.
    17. "Github - AndyGrant/OpenBench: OpenBench is a Distributed SPRT Testing Framework for Chess Engines". Github.com. Archived from the original on 23 December 2021. Retrieved 23 December 2021.
    18. "OpenBench". Chess.grantnet.us. Archived from the original on 2022-02-16. Retrieved 2022-02-16.
    19. "ChessOK Shop, Software, Training, Equipment, Books". Shop.chessok.com. Archived from the original on 26 June 2022. Retrieved 26 June 2022.
    20. 1 2 "Free chess graphical user interface (GUI) Arena for chess engines". Archived from the original on 2007-03-14. Retrieved 2018-11-29.
    21. "mizarchessengine.com". Archived from the original on 29 January 2013. Retrieved 25 September 2016.
    22. "Chess engine controversy at chessvibes.com, retrieved 28/May/2010". Archived from the original on 2010-05-28. Retrieved 2010-05-28.
    23. "Evaluation". Archived from the original on 2011-08-11. Retrieved 2010-12-18.
    24. "Rybka disqualified and banned from World Computer Chess Championships | ChessVibes". Archived from the original on 2012-03-02. Retrieved 2011-06-30.
    25. Riis, Dr. Søren (January 2, 2012). "A Gross Miscarriage of Justice in Computer Chess (part one)". Chessbase News. Archived from the original on 7 January 2012. Retrieved 19 February 2012.
    26. "Free Houdini beats commercial Rybka 23.5-16.5". Chessvibes. 10 February 2010. Archived from the original on 10 April 2014. Retrieved 25 October 2013.
    27. The Stockfish Team (2021-07-21). "Our lawsuit against ChessBase". Stockfishchess.org. Archived from the original on 2021-07-21. Retrieved 2021-08-18.
    28. "CCRL 40/15, 2m1s and FRC 40/2 lists updated (21-10-2023)". talkchess.com. Archived from the original on 15 June 2024. Retrieved 22 October 2023.
    29. "Experimental testruns of Stockfish / Torch". talkchess.com. Archived from the original on 15 June 2024. Retrieved 29 May 2024.
    30. "CEGT Testers". Cegt.net. Archived from the original on 24 May 2022. Retrieved 26 June 2022.
    31. Ray Banks (Modern Times). "ShashChess". talkchess.com. Archived from the original on 30 April 2024. Retrieved 30 April 2024.
    32. "Nolot test suite". Archived from the original on 2007-08-17. Retrieved 2007-08-04.
    33. "BT2450 test suite". Archived from the original on 2007-09-07. Retrieved 2007-10-10.
    34. Rosenboom, Manfred. "Rebel Century FAQ: 3. Using Rebel". Archived from the original on 3 March 2016. Retrieved 25 September 2016.
    35. "BRILLIANCY TEST SUITE (2 min/move)". Talkchess.com. Archived from the original on 26 June 2022. Retrieved 26 June 2022.
    36. "Strategic Test Suite". Sites.google.com. Archived from the original on 18 January 2017. Retrieved 26 June 2022.
    37. "Uups - Oops: Link-Fehler und URL-Umleitungen". Glarean-magazin.ch. 10 October 2010. Archived from the original on 15 June 2024. Retrieved 26 June 2022.
    38. Harding, T. (2002). 64 Great Chess Games, Dublin: Chess Mail. ISBN   0-9538536-4-0.
    39. "Lucas Chess". Lucaschess.pythonanywhere.com. Archived from the original on 6 July 2022. Retrieved 26 June 2022.
    40. "Aart Bik's Website". Aartbik.com. Archived from the original on 2018-11-20. Retrieved 2018-11-29.
    41. "Chess PGN Master". Pgnmaster.kalab.com. Archived from the original on 21 January 2021. Retrieved 26 June 2022.
    42. "Archived copy". Archived from the original on 2018-12-10. Retrieved 2018-11-29.{{cite web}}: CS1 maint: archived copy as title (link)
    43. "Computer_chess:wiki:lists:gui_protocol_support_list - Computer Chess Wiki". Computer-chess.org. Archived from the original on 2018-11-30. Retrieved 2018-11-29.