PECOTA

Last updated

PECOTA, an acronym for Player Empirical Comparison and Optimization Test Algorithm, [1] is a sabermetric system for forecasting Major League Baseball player performance. The word is a backronym based on the name of journeyman major league player Bill Pecota, who, with a lifetime batting average of .249, is perhaps representative of the typical PECOTA entry. PECOTA was developed by Nate Silver in 2002–2003 and introduced to the public in the book Baseball Prospectus 2003. [2] Baseball Prospectus (BP) has owned PECOTA since 2003; Silver managed PECOTA from 2003 to 2009. Beginning in Spring 2009, BP assumed responsibility for producing the annual forecasts, making 2010 the first baseball season for which Silver played no role in producing PECOTA projections. [3]

Contents

One of several widely publicized statistical systems of forecasts of player performance, PECOTA player forecasts are marketed by BP as a fantasy baseball product. Since 2003, annual PECOTA forecasts have been published both in the Baseball Prospectus annual books and, in more detailed form, on the BaseballProspectus.com subscription-based website. [4] PECOTA also inspired some analogous projection systems for other professional sports: KUBIAK for the National Football League, SCHOENE [5] and CARMELO [6] for the National Basketball Association, and VUKOTA [7] for the National Hockey League.

PECOTA forecasts a player's performance in all of the major categories used in typical fantasy baseball games; it also forecasts production in advanced sabermetric categories developed by Baseball Prospectus (e.g., VORP and EqA). In addition, PECOTA forecasts several summary diagnostics such as breakout rates, improve rates, and attrition rates, as well as the market values of the players. The logic and methodology underlying PECOTA have been described in several publications, but the detailed formulas are proprietary and have not been shared with the baseball research community.

Methodology

Silver described the inspiration for his approach as follows:

The basic idea behind PECOTA is really a fusion of two different things – [Bill] James's work on similarity scores and Gary Huckabay's work on Vlad, [Baseball Prospectus's] previous projection system, which tried to assign players to a number of different career paths. [8] I think Gary used something like thirteen or fifteen separate career paths, and all that PECOTA is really doing is carrying that to the logical extreme, where there is essentially a separate career path for every player in major league history. The comparability scores are the mechanism by which it picks and chooses from among those career paths. [9]

Comparable players

PECOTA relies on fitting a given player's past performance statistics to the performance of "comparable" Major League ballplayers by means of similarity scores. As is described in the Baseball Prospectus website's glossary: [10]

PECOTA compares each player against a database of roughly 20,000 major league batter seasons since World War II. In addition, it also draws upon a database of roughly 15,000 translated minor league seasons (1997–2006) for players that spent most of their previous season in the minor leagues. ... PECOTA considers four broad categories of attributes in determining a player's comparability:

1. Production metrics – such as batting average, isolated power, and unintentional walk rate for hitters, or strikeout rate and groundball rate for pitchers.

2. Usage metrics, including career length and plate appearances or innings pitched.

3. Phenotypic attributes, including handedness, height, weight, career length (for major leaguers), and minor league level (for prospects).

4. Fielding Position (for hitters) or starting/relief role (for pitchers). ... In most cases, the database is large enough to provide a meaningfully large set of appropriate comparables. When it isn't, the program is designed to 'cheat' by expanding its tolerance for dissimilar players until a reasonable sample size is reached.

PECOTA uses nearest neighbor analysis to match the individual player with a set of other players who are most similar to him. Although drawing on the underlying concept of Bill James' similarity scores, PECOTA calculates these scores in a distinct way that leads to a very different set of "comparables" than James' method. [11] Furthermore, Silver describes the following distinct feature:

The PECOTA similarity scores are based primarily on looking at a three-year window of a pitcher’s performance. Thus, we might look at what a pitcher did from ages 35–37, and compare that against the most similar age 35–37 performances, after adjusting for parks, league effects, and a whole host of other things. This is different from the similarity scores you might see at baseball-reference.com or in other places, which attempt to evaluate the totality of a player’s career up to a given age. [12]

Once a set of "comparables" is determined for each player, his future performance forecast is based on the historical performance of his "comparables". For example, a 26-year-old's forecast performance in the coming season will be based on how the most comparable Major League 26-year-olds performed in their subsequent season.

Separate sets of predictions are developed for hitters and pitchers.

Peripheral statistics

PECOTA also relies a lot on the use of peripheral statistics to forecast a given player's future performance. For example, drawing on the insights coming out of the use of defense-independent pitching statistics, PECOTA forecasts a pitcher's future performance in a given area by using information about his past performance in other areas. [13] As baseball analyst and journalist Alan Schwarz writes, "Silver ... designed a sophisticated variance algorithm that has examined every big-league pitcher's statistics since 1946 to determine which numbers best forecast effectiveness, specifically earned run average. His findings are counterintuitive to most fans. 'When you try to predict future E.R.A.'s with past E.R.A.'s, you're making a mistake,' Silver said. Silver found that the most predictive statistics, by a considerable margin, are a pitcher's strikeout rate and walk rate. Home runs allowed, lefty-righty breakdowns and other data tell less about a pitcher's future". [14]

Probability distributions

Instead of focusing on making point estimates of a player's future performance (such as batting average, home runs, and strike-outs), PECOTA relies on the historical performance of the given player's "comparables" to produce a probability distribution of the given player's predicted performance during the next five years. Alan Schwarz has emphasized this feature of PECOTA: "What separates Pecota from the gaggle of projection systems that outsiders have developed over many decades is how it recognizes, even flaunts, the uncertainty of predicting a player's skills. Rather than generate one line of expected statistics, Pecota presents seven – some optimistic, some pessimistic – each with its own confidence level. The system greatly resembles the forecasting of hurricane paths: players can go in many directions, so preparing for just one is foolish". [15] Silver has written,

This procedure requires us to become comfortable with probabilistic thinking. While a majority of players of a certain type may progress a certain way – say, peak early – there will always be exceptions. Moreover, the comparable players may not always perform in accordance with their true level of ability. They will sometimes appear to exceed it in any given season, and other times fall short, because of the sample size problems that we described earlier.

PECOTA accounts for these sorts of factors by creating not a single forecast point, as other systems do, but rather a range of possible outcomes that the player could expect to achieve at different levels of probability. Instead of telling you that it's going to rain, we tell you that there's an 80% chance of rain, because 80% of the time that these atmospheric conditions have emerged on Tuesday, it has rained on Wednesday.

Surely, this approach is more complicated than the standard method of applying an age adjustment based on the 'average' course of development of all players throughout history. However, it is also leaps and bounds more representative of reality, and more accurate to boot. [16]

Team effort

Although Silver was the creator of PECOTA, producing PECOTA forecasts was a team effort: "I might be 'the PECOTA guy,' but it very much is a team effort," Silver has said of the BP staff. "We all do it. It's my baby, but it takes a village to run a PECOTA". [17] For example, PECOTA draws on Clay Davenport's translations (the so-called Davenport Translations or DT's) of minor league and international baseball statistics to estimate the major league equivalent performance of each player. [18] In this way, PECOTA is able to make projections for more than 1,600 players each year, including many players with little or no prior major league experience.

The 2009 preseason forecasts were the last ones for which Silver took primary responsibility. In March 2009, Silver announced that PECOTA's extremely complex and laborious set of database manipulations and calculations would be moving to a different platform. Although Baseball Prospectus had been the owner of PECOTA since Silver sold it to them in 2003 – and Silver stewarded and took responsibility for the forecasts – henceforth PECOTA forecasts would be generated by the Baseball Prospectus team, initially with Clay Davenport in charge of the effort, [19] and later, through the 2013 season, with Colin Wyers heading up both production and improvements in PECOTA. [3]

Alternative forecasting systems

Most of the other popular forecasting systems do not use a "comparable players" approach. Instead most rely on direct projections from a player's past performance to his future performance, typically by using as a baseline a weighted average of a player's performance in his previous three years. Like PECOTA, many of those systems also adjust the projections for aging, park effects and regression toward the mean. Like PECOTA, they may also adjust for the competitive difficulty of each of the two major leagues. [20] The systems differ from one another, however, in the types and intensities of age adjustments, regression-effect estimates, park adjustments, and league-difficulty adjustments that they may make as well as in whether they use similarity scores. [21] PECOTA also makes projections for many more players than do other systems, because PECOTA relies on adjusted minor league statistics as well as major league statistics and tries to make projections for all of the players on major league expanded rosters (40 players per team) as well as other prospects.

Beginning in 2000, the Cleveland Indians developed a proprietary analytical database called DiamondView to evaluate scouting information gathered by the team; this system later incorporated player performance indicators and financial indicators, for purposes of evaluating and projecting the performance of all major league players. [22] During 2008–2009, the Pittsburgh Pirates were in process of developing MITT ("Managing, Information, Tools and Talent"), a proprietary database that integrates scouting reports, medical and contract information, and performance statistics and projections. [23]

Updates and revisions

First introduced in 2003, [24] PECOTA projections are produced each year and published both in the Baseball Prospectus annual monographs and on the BaseballProspectus.com website. PECOTA has undergone several improvements since 2003. The 2006 version introduced metrics for the market valuation of players based on the predicted performance levels. The 2007 version introduced adjustments for league effects, to account for differences in the competitive environment of the two major leagues. [25] The 2008 update took into account differences in players' performance during the first and second halves of the previous season as well as platoon splits (how well a player performed against hitters or pitchers who were left- or right-handed). [26] It also took account of baserunning. [27] In 2009, Baseball Prospectus introduced in-season PECOTA projections, to update and supplement its beginning of the season projections. [28] In 2012, PECOTA substantially changed the way it weighted past years' performance in establishing the baseline for projections. [29] In addition, 10-year forecasts and percentile forecasts were added to the individual player PECOTA cards that are published on-line. [30]

Accuracy

Although Baseball Prospectus promotes PECOTA commercially as "deadly accurate," all projection systems are subject to considerable uncertainty. A comparison found that PECOTA had outperformed several other forecasting systems for the 2006 season in predicting OPS. It performed nearly as well as the best of the other systems in predicting ERA. [31] Although PECOTA projections are made for well over 1000 hitters each season, the evaluation of the system included only slightly over 100 players who had a minimum of 500 major league AB and had also been included in projections by the other systems. Nate Silver's own comparison of the performance of alternative projection systems for hitters in 2007 also showed that PECOTA led the field, though a couple of others were close. [32]

Although designed primarily for predicting individual player performance, PECOTA has been applied also to predicting team performance. For this purpose, projected team depth charts are established with projected playing times for each team member, drawing on the expert advice of the Baseball Prospectus staff. The number of runs a team will score and allow during the coming season is estimated based on the playing times and PECOTA's predicted individual performance of each player, using a "Marginal Lineup Value" algorithm created by David Tate and further developed by Keith Woolner. [33] A team's expected wins is based on applying an improved version of Bill James' Pythagorean Formula to the estimated number of runs scored and allowed by the roster of players under the given playing-time assumptions. [34]

PECOTA has been used in preseason forecasts [35] of how many wins teams will attain and in mid-season simulations of the number of wins each team will attain and its odds of reaching the playoffs. [36] In 2006, PECOTA's preseason forecasts compared favorably to other forecasting systems (including Las Vegas betting line odds) in predicting the number of wins teams would earn during the season. [37] An independent evaluation by the website Vegas Watch showed that PECOTA had the lowest error in predicting Major League team wins in 2008 of all the best known forecasts, both those that were sabermetrically based and those that relied on individual expertise. [38] In 2009, however, PECOTA lagged behind all the well-known forecasters. [39]

A summary for the 2003 through 2007 seasons shows that PECOTA's average error between the predicted and actual team wins declined: [40] 2003 5.91 wins; 2004 7.71 wins; 2005 5.14 wins; 2006 4.94 wins; 2007 4.31 wins. Silver conjectures that the improvement has come in part from taking defense into account in the forecasts beginning in 2005. In 2008 the average error was 8.5 wins. [41]

Notes

  1. "Baseball Prospectus | Glossary". www.baseballprospectus.com. Retrieved 2016-05-05.
  2. Nate Silver, "Introducing PECOTA," in Gary Huckabay, Chris Kahrl, Dave Pease et al., Eds., Baseball Prospectus 2003 (Dulles, VA: Brassey's Publishers, 2003): 507–514.
  3. 1 2 Nate Silver and Kevin Goldstein, "State of the Prospectus: Spring 2009," BaseballProspectus.com, March 24, 2009 Archived March 27, 2009, at the Wayback Machine .
  4. Illustrative PECOTA estimates and "cards" are available for inspection by nonsubscribers here: http://www.baseballprospectus.com/pecota/.
  5. Kevin Pelton, "Introducing SCHOENE: Our NBA Projection System," BasketballProspectus.com (October 20, 2008)
  6. Silver, Nate (October 9, 2015). "We're Predicting The Career Of Every NBA Player. Here's How". FiveThirtyEight . Retrieved February 29, 2016.
  7. Thomas Awad, "Introducing VUKOTA," PuckProspectus.com (July 20, 2009).
  8. Gary Huckabay, "6–4–3: Reasonable Person Standard," BaseballProspectus.com, August 2, 2002.
  9. Rich Lederer, "An Unfiltered Interview with Nate Silver," Baseball Analysts, February 12, 2007.
  10. Baseball Prospectus | Glossary
  11. This difference is explained and illustrated in Nate Silver, "Introducing PECOTA," Baseball Prospectus 2003, cited above.
  12. http://www.baseballprospectus.com/unfiltered/?p=136. Also see Baseball Prospectus' glossary entry for "Comparable Players".
  13. See PERA for an example of the use of peripheral statistics to estimate a performance.
  14. Alan Schwarz, "Numbers Suggest Mets Are Gambling on Zambrano," New York Times, August 22, 2004.
  15. Alan Schwarz, "Predicting Futures in Baseball, and the Downside of Damon," New York Times, November 13, 2005.
  16. Nate Silver, "Baseball Prospectus Basics: The Science of Forecasting," BaseballProspectus.com, March 11, 2004.
  17. William Hageman, "Baseball By the Numbers," Chicago Tribune, January 4, 2006.
  18. See Clay Davenport, "DT's vs. MLEs — A Validation Study," BaseballProspectus.com, January 30, 1998; Clay Davenport, "Winter and Fall League Translations: Just How Good Are These Leagues, Anyway?," BaseballProspectus.com, January 27, 2004; and Clay Davenport, "Over There! A Second Review of Translating Japanese Statistics, and Translating the Mexican League," Baseball Prospectus 2004 (New York: Workman, 2004): 585–590.
  19. See, for example, Clay Davenport, "Depth Charts," BaseballProspectus.com, May 13, 2009.
  20. PECOTA's aging adjustment is implicit in the path of "future" performance of the set of historical "comparable" players.
  21. Among the current major alternative statistically based projection systems are Tom Tango's Marcel projections (available and documented for 2007 at The Hardball Times); Diamond Mind Baseball, also described in an ESPN.com article on 2007 team projections; Ron Shandler's Baseball HQ and his annual book, Baseball Forecaster; The Hardball Times pre-season forecasts, inaugurated with the 2007 season; Chone Smith's "Chone Projections," reported on the website of Fangraphs.com; Baseball Info Solutions – BIS; and Dan Szymborski's "ZiPS" Projections. For a list of well-known forecasting systems as of 2014, including "Steamer," see this summary by Fangraphs: "Projection Systems".
  22. cleveland.com: The Game Plan Archived 2007-12-10 at the Wayback Machine
  23. Pat Mitsch, "Pirates are Hoping 'The MITT' Catches On," Pittsburgh Tribune-Review, July 19, 2009 Archived July 22, 2009, at the Wayback Machine .
  24. Nate Silver, "Introducing PECOTA," Baseball Prospectus 2003, cited previously.
  25. "Baseball Prospectus Chat: Nate Silver," BaseballProspectus.com, January 19, 2007.
  26. Steven Goldman and Christina Kahrl, Eds., Baseball Prospectus 2008 (New York: Plume, 2008), pp. viii–ix.
  27. Nate Silver, "Is Baserunning a Skill?" BaseballProspectus.com, November 29, 2007.
  28. Eric Seidman, "In-Season PECOTAs," BaseballProspectus.com, July 23, 2009.
  29. Colin Wyers, "Reintroducing PECOTA: The Weighting is the Hardest Part," BaseballProspectus.com, February 8, 2012.
  30. Colin Wyers, "Reintroducing PECOTA," BaseballProspectus.com, March 12, 2012.
  31. Chone Smith http://lanaheimangelfan.blogspot.com/2006/12/pecota.html
  32. Nate Silver, "2007 Hitter Projection Roundup," BaseballProspectus.com (October 4, 2007).
  33. Keith Woolner, "Marginal Lineup Value," StatHead.com.
  34. On the Pythagenport formula, see Clay Davenport and Keith Woolner, "Revisiting the Pythagorean Theorem: Putting Bill James' Pythagorean Theorem To the Test," BaseballProspectus.com, June 30, 1999 as well as the Baseball Prospectus glossary entry for "Pythagenport". On the construction of the depth charts for each team and the application of PECOTA to estimating team wins, see Nate Silver, "PECOTA Projects the American League," BaseballProspectus.com, March 21, 2005; and Nate Silver, "PECOTA Breaks Hearts," BaseballProspectus.com, March 29, 2006.
  35. e.g., Clay Davenport, "PECOTA Projected Standings: Pegging the 2009 Season," BaseballProspectus.com, February 19, 2009.
  36. See Clay Davenport, "Playoff Odds Report: The Addition of PECOTA," BaseballProspectus.com, May 3, 2006 and Baseball Prospectus Statistics.
  37. Nate Silver, "Projection Reflection," BaseballProspectus.com, October 11, 2006.
  38. "Evaluating April MLB Predictions (2008)," VegasWatch.net, September 21, 2008 and September 28, 2008.
  39. "Evaluating April MLB Predictions (2009)," VegasWatch, September 28, 2009.
  40. Nate Silver, "Braves, Angels Have Most Heart," BaseballProspectus.com, March 10, 2007.
  41. "Taking the Over on PECOTA," VegasWatch.net, February 8, 2009.

Related Research Articles

Sabermetrics or SABRmetrics is the empirical analysis of baseball, especially baseball statistics that measure in-game activity.

Bill James American baseball writer and statistician

George William James is an American baseball writer, historian, and statistician whose work has been widely influential. Since 1977, James has written more than two dozen books devoted to baseball history and statistics. His approach, which he termed sabermetrics in reference to the Society for American Baseball Research (SABR), scientifically analyzes and studies baseball, often through the use of statistical data, in an attempt to determine why teams win and lose.

Equivalent Average (EqA) is a baseball metric invented by Clay Davenport and intended to express the production of hitters in a context independent of park and league effects. It represents a hitter's productivity using the same scale as batting average. Thus, a hitter with an EqA over .300 is a very good hitter, while a hitter with an EqA of .220 or below is poor. An EqA of .260 is defined as league average.

Clay Davenport is a baseball sabermetrician who co-founded Baseball Prospectus (BP) in 1996. He co-edited several of the Baseball Prospectus annual volumes and is a writer for BaseballProspectus.com. Much of his work for BP was behind the scenes, where he maintained and implemented advanced statistics for the website.

In baseball, value over replacement player is a statistic popularized by Keith Woolner that demonstrates how much a hitter or pitcher contributes to their team in comparison to a replacement-level player who is an average fielder at that position and a below average hitter. A replacement player performs at "replacement level," which is the level of performance an average team can expect when trying to replace a player at minimal cost, also known as "freely available talent."

QuesTec was a digital media company that pioneered virtual replay from real-time measurement data for baseball and tennis. Years before competitors introduced similar products, QuesTec had established a reputation for accuracy and reliability. Although originally based on internally developed technology, in 1998 QuesTec moved to tracking technology provided by engineers at the Atlantic Aerospace Electronics Corporation. QuesTec is best known for its Umpire Information System (UIS), used by Major League Baseball for the purpose of providing feedback and evaluation of Major League umpires.

<i>Baseball Mogul</i>

Baseball Mogul is a series of career baseball management computer games created by game designer Clay Dreslough. The product was first published in 1997. The 22nd and latest installment is Baseball Mogul 2019. A proprietary database, included with the game, permits play in any season of historical baseball from 1901 to the present. The early Baseball Mogul games are considered to be influential works in the baseball management simulation genre.

Baseball Prospectus organization

Baseball Prospectus (BP) is an organization that publishes a website, BaseballProspectus.com, devoted to the sabermetric analysis of baseball. BP has a staff of regular columnists and provides advanced statistics as well as player and team performance projections on the site. Since 1996 the BP staff has also published a Baseball Prospectus annual as well as several other books devoted to baseball analysis and history.

Football Outsiders (FO) is a website started in July 2003 which focuses on advanced statistical analysis of the NFL. The site is run by a staff of regular writers, who produce a series of weekly columns using both the site's in-house statistics and their personal analyses of NFL games.

In sabermetrics and basketball analytics, similarity scores are a method of comparing baseball and basketball players to other players, with the intent of discovering who the most similar historical players are to a certain player.

Tom Tango and "TangoTiger" are aliases used online by a baseball sabermetrics and ice hockey statistics analyst. He runs the Tango on Baseball sabermetrics website and is also a contributor to ESPN's baseball blog TMI . Tango is currently the Senior Database Architect of Stats for MLB Advanced Media.

Nate Silver American pundit and writer

Nathaniel Read Silver is an American statistician and writer who analyzes baseball and elections. He is the founder and editor-in-chief of FiveThirtyEight and a Special Correspondent for ABC News.

Joseph S. (Joe) Sheehan was born in New York City on February 26, 1971, and attended Regis High School. He graduated from the University of Southern California in 1994, with a degree in journalism. Sheehan lives in the New York City area. He is one of the founders and was a co-editor of the first annual book of sabermetric baseball forecasts and analyses by Baseball Prospectus in 1996 as well as several later volumes.

Bill Pecota American baseball player

William Joseph Pecota is a former Major League Baseball infielder. He is the namesake of PECOTA, a sabermetric created by Nate Silver and owned by Baseball Prospectus.

Baseball Think Factory, abbreviated as BTF or BBTF, is a sabermetrically-oriented baseball web site that features daily news stories in baseball, with original content contributed by SABR members such as Dan Szymborski. The site was previously branded as Baseball Primer, and was created in 2001 by the founders of Baseball-Reference. Contributors who have gone on to work for Major League Baseball front offices include Voros McCracken, Carlos Gomez, and Tom Tango. Bill James' Baseball Abstract books published in the 1980s are widely considered to be the modern predecessor to websites using sabermetrics such as baseballthinkfactory and baseballprospectus.

Daniel John Szymborski is an American writer of sabermetrics primarily known for his work with baseball projections and minor league translations.

Earned run average Baseball statistic

In baseball statistics, earned run average (ERA) is the average of earned runs given up by a pitcher per nine innings pitched. It is determined by dividing the number of earned runs allowed by the number of innings pitched and multiplying by nine. Thus, a lower ERA is better. Runs resulting from passed balls or defensive errors are recorded as unearned runs and omitted from ERA calculations.

FanGraphs.com is a website run by Fangraphs Inc., located in Arlington, Virginia, and created and owned by David Appelman that provides statistics for every player in Major League Baseball history.

Wins Above Replacement or Wins Above Replacement Player, commonly abbreviated to WAR or WARP, is a non-standardized sabermetric baseball statistic developed to sum up "a player's total contributions to his team". A player's WAR value is claimed to be the number of additional wins his team has achieved above the number of expected team wins if that player were substituted with a replacement-level player: a player who may be added to the team for minimal cost and effort.

Advanced metrics is the term for the empirical analysis of sports, particularly statistics that measure in-game productivity and efficiency. Advanced metrics were first employed in baseball by Bill James, a pioneer in the field who is considered the father and public face of the practice.

References