Rank reversals in decision-making

Last updated

In decision-making, a rank reversal is a change in the rank ordering of the preferability of alternative possible decisions when, for example, the method of choosing changes or the set of other available alternatives changes. The issue of rank reversals lies at the heart of many debates in decision-making and multi-criteria decision-making, in particular.

Contents

Unlike most other computational procedures, it is hard to tell if a particular decision-making method has derived the correct answer or not. Such methods analyze a set of alternatives described in terms of some criteria. They determine which alternative is the best one, or they provide relative weights of how the alternatives perform, or just how the alternatives should be ranked when all the criteria are considered simultaneously. This is exactly where the challenge with decision making exists. Often it is hard, if not practically impossible, to determine whether a correct answer has been reached or not. With other computational methods, for instance with a job scheduling method, one can examine a set of different answers and then categorize the answers according to some metric of performance (for instance, a project's completion time). But this may not be possible to do with the answers derived by most decision making methods. After all, determining the best decision making method leads to a decision making paradox.

Thus the following question emerges: How can one evaluate decision-making methods? This is a very difficult issue and may not be answered in a globally accepted manner.

A critical part in answering this fundamental question is played by what is known as rank reversals.

Rank reversal

One way to test the validity of decision-making methods is to construct special test problems and then study the solutions they derive. If the solutions exhibit some logic contradictions (in the form of undesirable rank reversals of the alternatives), then one may argue that something is wrong with the method that derived them.

To see the above point more clearly, suppose that three candidates are evaluated for some job opening. Let us designate these candidates as A, B, and C. Suppose that some decision making method has determined that the best candidate for that job is person A, followed by B, who is followed by C. This is the first ranking and it is indicated as follows: A > B > C (where > means better than). Next, suppose that candidate B (who is not the best one) is replaced by an even worse candidate, say person D. That is, now we have B > D, and candidate B is replaced by D while candidates A and C remain in the pool of candidates with exactly the same characteristics as before. When the new set of alternatives (i.e., candidates A, D and C) are ranked together and by assuming that the criteria have exactly the same weights as before, then should not candidate A still be the best one? It turns out that under some decision making methods the best alternative may be different now. [1] This is known as a rank reversal and it is one of the types of rank reversals.

The first type of rank reversal in the above context was observed by Belton and Gear in 1983 as part of a study [2] of the analytic hierarchy process (AHP). [3] They first considered a simple decision problem comprised by 3 alternatives and 2 criteria. Next a copy of a non-optimal alternative was introduced. When the 4 alternatives (i.e., the previous 3 plus the copy) were evaluated, and under the assumption that the criteria weights are exactly the same as before, it was observed that now the indication of the best alternative can change. That is, a rank reversal may occur with the AHP. A few years later it was observed that the AHP, as well as a new variant to it that was introduced by Professor Thomas Saaty (the inventor of the AHP) in response to the previous observation by Belton and Gear, may exhibit rank reversals when a non-optimal alternative is replaced by a worse one (and not a copy of an alternative as in Belton and Gear's experiment). [4]

The issue of rank reversals has captured the interest of many researchers and practitioners in the field of decision-making. It is something that continues to be considered controversial by many and is frequently debated. [5] [6] [7] [8] [9] [10]

Different types of rank reversals

There are many different types of rank reversals, depending on how the alternatives in a problem are defined and evaluated. These types are described next as Type 1, Type 2, Type 3, Type 4, and Type 5.

Rank reversals of Type 1

As stated earlier, one may introduce identical or near-identical copies of non-optimal alternatives and then check to see if the indication of the best alternative changes or not. [2]

Rank reversals of Type 2

Another way is to replace a non-optimal alternative with a worse one and then see if the indication of the best alternative changes or not. [4]

Rank reversals of Type 3

First consider a problem with all the alternatives together and get a ranking. Next, decompose the original problem into a set of smaller problems defined on two alternatives at a time and the same criteria (and their weights) as before. Get the rankings of these smaller problems and check to see if they are in conflict with the ranking of the alternatives of the original (larger) problem. [11]

Rank reversals of Type 4

Type 4 is like Type 3, but ignores the ranking of the original (larger) problem. Instead, check to see if the rankings of the smaller problems are in conflict with each other. For instance, suppose that the following 3 alternatives A, B, and C are considered. Next, suppose that some 2-alternative problems are solved and the rankings A > B, B > C, and C > A, are derived from these 2-alternative problems. Obviously, the above situation indicates a case of non-transitivity (or contradiction) as we get A > B > C > A.

Rank reversals of Type 5

All previous types of rank reversals are known to occur with the analytic hierarchy process (AHP) and its additive variants, the TOPSIS and ELECTRE methods and their variants. [1] [11] [12]

The weighted product model (WPM) does not exhibit the previous types of rank reversals, due to the multiplication formula it uses. [1] [11] However, the WPM does cause rank reversals when it is compared with the weighted sum model (WSM) and under the condition that all the criteria of a given decision problem can be measured in exactly the same unit. [4] The same is true with all the previous methods as well. This is the Type 5 ranking reversal.

It is quite possible to define more types of rank reversals. One only needs to determine ways to alter a test problem and see how the ranking of the alternatives of the new problem differs from the original ranking of the alternatives of the original problem. Furthermore, the difference in rankings, somehow, should indicate the presence of undesirable effects.

Are rank reversals always undesirable?

Decision-making methods are used to make decisions in many aspects of human activity. This is especially true with decisions that involve large amounts of money or decisions that may have huge impact on large numbers of people. Given the well-established fact that different methods may yield different answers when they are fed with exactly the same problem, the question is how to evaluate them. Rank reversals are at the very heart of assessing the merits of such methods. At the same time, they are at the center of many heated debates in this area. Many authors use them as means to criticize decision making methods or to better explain rational behavior. [5] [6] [7] [8] [9] [10]

Consider a simple example of buying a car. Suppose that there are two cars available to the decision maker: Car A and Car B. Car A is much cheaper than Car B but its overall quality is much less when compared to that for Car B. On the other hand, Car B is more expensive than Car A but it is also of better quality. A decision maker who is concerned of the high price issue, may choose Car A over the better quality and more expensive Car B. Next suppose that the car dealer presents to the decision maker a third car, say Car C, which is way more expensive than Car B but now the overall quality of Car C is marginally higher than that of Car B. Under such a scenario, it is quite possible for a decision maker to alter his/her opinion and purchase Car B instead of Car A, even if he/she has not actually seen Car C.

Such events may take place with many rational decision makers.[ dubious – discuss ] In other words, rank reversals may actually be possible in rational decision making. The issue of having rank reversals by rational decision makers has been studied extensively by Amos Tversky. [13] In other words, having rank reversals in certain occasions and of certain types may not be indicative to faulty decision making. However, the key question is how to be able to distinguish when rank reversals indicate that something is wrong or when they do not conflict rational decision making. This is a highly debated issue in the decision making community.

Methods that have been verified to exhibit rank reversals

The following is just a partial list of multi-criteria decision making methods which have been confirmed to exhibit various types of rank reversals: [1] [4] [5] [6] [7] [8] [9] [10] [14] [15] [16] [17]

Related Research Articles

<span class="mw-page-title-main">Thomas L. Saaty</span> Iraqi-American mathematician

Thomas L. Saaty was a Distinguished University Professor at the University of Pittsburgh, where he taught in the Joseph M. Katz Graduate School of Business. He is the inventor, architect, and primary theoretician of the Analytic Hierarchy Process (AHP), a decision-making framework used for large-scale, multiparty, multi-criteria decision analysis, and of the Analytic Network Process (ANP), its generalization to decisions with dependence and feedback. Later on, he generalized the mathematics of the ANP to the Neural Network Process (NNP) with application to neural firing and synthesis but none of them gain such popularity as AHP.

<span class="mw-page-title-main">Multiple-criteria decision analysis</span> Operations research that evaluates multiple conflicting criteria in decision making

Multiple-criteria decision-making (MCDM) or multiple-criteria decision analysis (MCDA) is a sub-discipline of operations research that explicitly evaluates multiple conflicting criteria in decision making. Conflicting criteria are typical in evaluating options: cost or price is usually one of the main criteria, and some measure of quality is typically another criterion, easily in conflict with the cost. In purchasing a car, cost, comfort, safety, and fuel economy may be some of the main criteria we consider – it is unusual that the cheapest car is the most comfortable and the safest one. In portfolio management, managers are interested in getting high returns while simultaneously reducing risks; however, the stocks that have the potential of bringing high returns typically carry high risk of losing money. In a service industry, customer satisfaction and the cost of providing service are fundamental conflicting criteria.

<span class="mw-page-title-main">Analytic hierarchy process</span> Structured technique for organizing and analyzing complex decisions

In the theory of decision making, the analytic hierarchy process (AHP), also analytical hierarchy process, is a structured technique for organizing and analyzing complex decisions, based on mathematics and psychology. It was developed by Thomas L. Saaty in the 1970s; Saaty partnered with Ernest Forman to develop Expert Choice software in 1983, and AHP has been extensively studied and refined since then. It represents an accurate approach to quantifying the weights of decision criteria. Individual experts’ experiences are utilized to estimate the relative magnitudes of factors through pair-wise comparisons. Each of the respondents compares the relative importance of each pair of items using a specially designed questionnaire. The relative importance of the criteria can be determined with the help of the AHP by comparing the criteria and, if applicable, the sub-criteria in pairs by experts or decision-makers. On this basis, the best alternative can be found.

Pairwise comparison generally is any process of comparing entities in pairs to judge which of each entity is preferred, or has a greater amount of some quantitative property, or whether or not the two entities are identical. The method of pairwise comparison is used in the scientific study of preferences, attitudes, voting systems, social choice, public choice, requirements engineering and multiagent AI systems. In psychology literature, it is often referred to as paired comparison.

The analytic network process (ANP) is a more general form of the analytic hierarchy process (AHP) used in multi-criteria decision analysis.

ÉLECTRE is a family of multi-criteria decision analysis (MCDA) methods that originated in Europe in the mid-1960s. The acronym ÉLECTRE stands for: ÉLimination Et Choix Traduisant la REalité.

For group decision-making, the hierarchical decision process (HDP) refines the classical analytic hierarchy process (AHP) a step further in eliciting and evaluating subjective judgements. These improvements, proposed initially by Dr. Jang Ra include the constant-sum measurement scale for comparing two elements, the logarithmic least squares method (LLSM) for computing normalized values, the sum of inverse column sums (SICS) for measuring the degree of (in)consistency, and sensitivity analysis of pairwise comparisons matrices. These subtle modifications address issues concerning normal AHP consistency and applicability in the process of constructing hierarchies: generating criteria, classifying/selecting criteria, and screening/selecting decision alternatives.

Decision-making software is software for computer applications that help individuals and organisations make choices and take decisions, typically by ranking, prioritizing or choosing from a number of options.

This is a worked-through example showing the use of the analytic hierarchy process (AHP) in a practical decision situation.

In decision theory, the weighted sum model (WSM), also called weighted linear combination (WLC) or simple additive weighting (SAW), is the best known and simplest multi-criteria decision analysis (MCDA) / multi-criteria decision making method for evaluating a number of alternatives in terms of a number of decision criteria.

The weighted product model (WPM) is a popular multi-criteria decision analysis (MCDA) / multi-criteria decision making (MCDM) method. It is similar to the weighted sum model (WSM). The main difference is that instead of addition in the main mathematical operation, there is multiplication.

The decision-making paradox is a phenomenon related to decision-making and the quest for determining reliable decision-making methods. It was first described by Triantaphyllou, and has been recognized in the related literature as a fundamental paradox in multi-criteria decision analysis (MCDA), multi-criteria decision making (MCDM) and decision analysis since then.

Potentially All Pairwise RanKings of all possible Alternatives (PAPRIKA) is a method for multi-criteria decision making (MCDM) or conjoint analysis, as implemented by decision-making software and conjoint analysis products 1000minds and MeenyMo.

This is a worked-through example showing the use of the analytic hierarchy process (AHP) in a practical decision situation.

The Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) is a multi-criteria decision analysis method, which was originally developed by Ching-Lai Hwang and Yoon in 1981 with further developments by Yoon in 1987, and Hwang, Lai and Liu in 1993. TOPSIS is based on the concept that the chosen alternative should have the shortest geometric distance from the positive ideal solution (PIS) and the longest geometric distance from the negative ideal solution (NIS). A dedicated book in the fuzzy context was published in 2021

Expert Choice is decision-making software that is based on multi-criteria decision making.

Decision Lens is online decision-making software that is based on multi-criteria decision making.

Criterium DecisionPlus is decision-making software that is based on multi-criteria decision making.

The VIKOR method is a multi-criteria decision making (MCDM) or multi-criteria decision analysis method. It was originally developed by Serafim Opricovic to solve decision problems with conflicting and noncommensurable criteria, assuming that compromise is acceptable for conflict resolution, the decision maker wants a solution that is the closest to the ideal, and the alternatives are evaluated according to all established criteria. VIKOR ranks alternatives and determines the solution named compromise that is the closest to the ideal.

Super Decisions is decision-making software which works based on two multi-criteria decision making methods.

References

  1. 1 2 3 4 Triantaphyllou, E. (2000). Multi-Criteria Decision Making: A Comparative Study. Dordrecht, The Netherlands: Kluwer Academic Publishers (now Springer). p. 320. ISBN   0-7923-6607-7.
  2. 1 2 Belton, V.; A.E. Gear (1983). "On a Short-Coming of Saaty's Method of Analytic Hierarchies". Omega. 11 (3): 228–230. doi:10.1016/0305-0483(83)90047-6.
  3. ↑ Saaty, T.L. (1990). "How to make a decision: The analytic hierarchy process". European Journal of Operational Research. 48 (1): 9–26. doi:10.1016/0377-2217(90)90057-I. hdl: 10338.dmlcz/143540 .
  4. 1 2 3 4 Triantaphyllou, E.; S.H. Mann (1989). "An Examination of the Effectiveness of Multi-Dimensional Decision-Making Methods: A Decision-Making Paradox". International Journal of Decision Support Systems. 5 (3): 303–312. doi:10.1016/0167-9236(89)90037-7 . Retrieved 2010-06-25.
  5. 1 2 3 Leskinen, P.; J. Kangas (2005). "Rank reversals in multi-criteria decision analysis with statistical modeling of ratio-scale pairwise comparisons". Journal of the Operational Research Society. 56 (7): 855–861. doi:10.1057/palgrave.jors.2601925. S2CID   33411976.
  6. 1 2 3 Ishizaka, A.; M. Lusti (2006). "How to derive priorities in AHP: a comparative study". Central European Journal of Operations Research. 14 (4): 387–400. doi:10.1007/s10100-006-0012-9. S2CID   18717451.
  7. 1 2 3 Kujawski, E. (2005). "A reference-dependent regret model for deterministic tradeoff studies". Systems Engineering. 8 (2): 119–137. doi:10.1002/sys.20027. S2CID   41318639.
  8. 1 2 3 Saaty, T.L. (2005). "Making and validating complex decisions with the AHP/ANP". Journal of Systems Science and Systems Engineering. 14 (1): 1–36. doi:10.1007/s11518-006-0179-6. S2CID   122736296.
  9. 1 2 3 Bevilacquaa, M.; M. Braglia (2000). "The analytic hierarchy process applied to maintenance strategy selection". Reliability Engineering & System Safety. 70 (1): 71–83. doi:10.1016/S0951-8320(00)00047-8.
  10. 1 2 3 Zahir, S. (2009). "Normalisation and rank reversals in the additive analytic hierarchy process: a new analysis". International Journal of Operational Research. 4 (4): 446–467. doi:10.1504/IJOR.2009.023538.
  11. 1 2 3 Triantaphyllou, E. (2001). "Two new cases of rank reversals when the AHP and some of its additive variants are used that do not occur with the multiplicative AHP". Multi-Criteria Decision Analysis. 10: 11–25. doi:10.1002/mcda.284.
  12. ↑ Wang, X.; E. Triantaphyllou (2008). "Ranking irregularities when evaluating alternatives by using some ELECTRE methods". Omega. 36: 45–63. doi:10.1016/j.omega.2005.12.003. S2CID   26349552.
  13. ↑ Tversky, A. (1969). "Intransitivity of Preferences". Psychological Review. 76: 31–48. doi:10.1037/h0026750. S2CID   144609998.
  14. ↑ Zanakis, S.H.; A. Solomon; N. Wisharta; S. Dublish (1998). "Multi-attribute decision making: A simulation comparison of select methods". European Journal of Operational Research. 107 (3): 507–529. doi:10.1016/S0377-2217(97)00147-1.
  15. ↑ Ertugrul, I.; N. Karakasoglu (2008). "Comparison of fuzzy AHP and fuzzy TOPSIS methods for facility location selection". The International Journal of Advanced Manufacturing Technology. 39 (7–8): 783–795. doi:10.1007/s00170-007-1249-8. S2CID   111003263.
  16. ↑ Zhang, K.; C. Kluck; G. Achari (2009). "A Comparative Approach for Ranking Contaminated Sites Based on the Risk Assessment Paradigm Using Fuzzy PROMETHEE". Environmental Management. 44 (5): 952–967. Bibcode:2009EnMan..44..952Z. doi:10.1007/s00267-009-9368-7. PMID   19763684. S2CID   42902838.
  17. ↑ Olson, D.; H.M. Moshkovich; R. Schellenberger; A.I. Mechitov (1995). "Consistency and Accuracy in Decision Aids: Experiments with Four Multiattribute Systems". Decision Sciences. 26 (6): 723–748. doi:10.1111/j.1540-5915.1995.tb01573.x.
  18. ↑ Norton EC, Miller MM, Wang JJ, Coyne K, Kleinman LC. Rank reversal in indirect comparisons. Value Health. 2012 Dec;15(8):1137-40. doi: 10.1016/j.jval.2012.06.001. Epub 2012 Sep 7. PMID: 23244817; PMCID: PMC3527821
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.