Optimal computing budget allocation

In Computer Science, Optimal Computing Budget Allocation (OCBA) is a simulation optimization method designed to maximize the Probability of Correct Selection (PCS) while minimizing computational costs. First introduced by Dr. Chun-Hung Chen in the mid-1990s, OCBA determines how many simulation runs (or how much computational time) or the number of replications each design alternative needs to identify the best option while using as few resources as possible. ^{[1] [2]}

OCBA has also been shown to enhance partition-based random search algorithms for solving deterministic global optimization problems. ^[3] Over the years, OCBA has been applied in manufacturing systems design, healthcare planning, and financial modeling. It has also been extended to handle more complex scenarios, such as balancing multiple objectives, ^[4] feasibility determination, ^[5] and constrained optimization. ^[6]

Intuitive Explanation

The goal of OCBA is to provide a systematic approach to efficiently run a large number of simulations by focusing only on the critical alternatives, in order to select the best alternative.

In other words, OCBA prioritizes only the most critical alternatives, minimizing computation time and reducing the variances of these critical estimators. The expected outcome is maintaining the required level of accuracy while requiring fewer computational resources. ^[7]

Figure 1: Preliminary simulation results show alternatives 2 and 3 have lower average delay times. OCBA suggests focusing further simulation resources on alternatives 2 and 3 while stopping simulations for alternatives 1, 4, and 5 to save costs without compromising accuracy.

Core Optimization Problem

The problem is mathematically formulated as:

$${\displaystyle \max _{\tau _{1},\tau _{2},\ldots ,\tau _{k}}\mathrm {PCS} }$$

Subject to:

${\displaystyle \sum _{i=1}^{k}\tau _{i}=\tau ,\quad \tau _{i}\geq 0,;i=1,2,...,k}$

where:

${\displaystyle k}$: Total number of design alternatives

${\displaystyle \tau _{i}}$: Number of simulation replications allocated to the ${\displaystyle i}$-th design

${\displaystyle \tau }$: Total computational budget

OCBA optimizes the allocation of simulation replications by focusing on alternatives with higher variances or smaller performance gaps relative to the best alternative. The ratio of replications between two alternatives, such as ${\displaystyle N_{2}}$ and ${\displaystyle N_{3}}$, is determined by the following formula:

$${\displaystyle {\frac {N_{2}}{N_{3}}}={\frac {\left({\frac {\sigma _{2}}{\delta _{1,2}}}\right)^{2}}{\left({\frac {\sigma _{3}}{\delta _{1,3}}}\right)^{2}}}}$$

Here:

${\displaystyle \sigma _{i}}$: The variance of the performance of alternative ${\displaystyle i}$.

${\displaystyle \delta _{1,i}}$: The performance gap between the best alternative (${\displaystyle 1}$) and alternative ${\displaystyle i}$.

${\displaystyle N_{i}}$: The number of simulation replications allocated to alternative ${\displaystyle i}$.

This formula ensures that alternatives with smaller performance gaps (${\displaystyle \delta _{1,i}}$) or higher variances (${\displaystyle \sigma _{i}}$) receive more simulation replications. This maximizes computational efficiency while maintaining a high Probability of Correct Selection (PCS), ensuring computational efficiency by reducing replications for non-critical alternatives and increasing them for critical ones. ^[8] Numerical results show that OCBA can achieve the same simulation quality with only one-tenth of the computational effort compared to traditional methods. ^[2]

Some extensions of OCBA

According to Szechtman and Yücesan (2008), ^[9] OCBA is also helpful in feasibility determination problems. This is where the decisions makers are only interested in differentiating feasible alternatives from the infeasible ones. Further, choosing an alternative that is simpler, yet similar in performance is crucial for other decision makers. In this case, the best choice is among top-r simplest alternatives, whose performance rank above desired levels. ^[10]

In addition, Trailovic ^[11] and Pao ^[12] (2004) demonstrate an OCBA approach, where we find alternatives with minimum variance, instead of with best mean. Here, we assume unknown variances, voiding the OCBA rule (assuming that the variances are known). During 2010 research was done on an OCBA algorithm that is based on a t distribution. The results show no significant differences between those from t-distribution and normal distribution. The above presented extensions of OCBA is not a complete list and is yet to be fully explored and compiled. ^[2]

Multi-Objective OCBA

Multi-Objective Optimal Computing Budget Allocation (MOCBA) is the OCBA concept that applies to multi-objective problems. In a typical MOCBA, the PCS is defined as

$${\displaystyle \Pr\{CS\}\equiv \Pr \left\{\left(\bigcap _{i\in S_{p}}E_{i}\right)\bigcap \left(\bigcap _{i\in {\overline {S}}_{p}}E_{i}^{c}\right)\right\},}$$

in which

• ${\displaystyle S_{p}}$ is the observed Pareto set,
• ${\displaystyle {\overline {S}}_{p}}$ is the non-Pareto set, i.e., ${\displaystyle {\overline {S}}_{p}=\Theta \backslash S_{p}}$,
• ${\displaystyle E_{i}}$ is the event that design ${\displaystyle i}$ is non-dominated by all other designs,
• ${\displaystyle E_{i}^{c}}$ is the event that design ${\displaystyle i}$ is dominated by at least one design.

We notice that, the Type I error ${\displaystyle e_{1}}$ and Type II error ${\displaystyle e_{2}}$ for identifying a correct Pareto set are respectively

${\displaystyle e_{1}=1-\Pr \left\{\bigcap _{i\in {\overline {S}}_{p}}E_{i}^{c}\right\}}$ and ${\displaystyle e_{2}=1-\Pr \left\{\bigcap _{i\in S_{p}}E_{i}\right\}}$.

Besides, it can be proven that

${\displaystyle e_{1}\leq ub_{1}=H\left|{\overline {S}}_{p}\right|-H\sum _{i\in {\overline {S}}_{p}}{\max _{j\in \Theta ,j\neq i}\left[\min _{l\in {1,\ldots ,H}}\Pr \left\{{\tilde {J}}_{jl}\leq {\tilde {J}}_{il}\right\}\right]}}$

and

${\displaystyle e_{2}\leq ub_{2}=(k-1)\sum _{i\in S_{p}}\max _{j\in \Theta ,j\neq i}\left[\min _{l\in {1,\ldots ,H}}\Pr \left\{{\tilde {J}}_{jl}\leq {\tilde {J}}_{il}\right\}\right],}$

where ${\displaystyle H}$ is the number of objectives, and ${\displaystyle {\tilde {J}}_{il}}$ follows posterior distribution ${\displaystyle Normal\left({\bar {J}}_{il},{\frac {\sigma _{il}^{2}}{N_{i}}}\right).}$ Noted that ${\displaystyle {\bar {J}}_{il}}$ and ${\displaystyle \sigma _{il}}$ are the average and standard deviation of the observed performance measures for objective ${\displaystyle l}$ of design ${\displaystyle i}$, and ${\displaystyle N_{i}}$ is the number of observations.

Thus, instead of maximizing ${\displaystyle \Pr\{CS\}}$, we can maximize its lower bound, i.e., ${\displaystyle APCS{-}M\equiv 1-ub_{1}-ub_{2}.}$ Assuming ${\displaystyle \tau \rightarrow \infty }$, the Lagrange method can be applied to conclude the following rules:

${\displaystyle \tau _{i}={\frac {\beta _{i}}{\sum _{j\in \Theta }\beta _{j}}}\tau ,}$

in which

• for a design ${\displaystyle h\in S_{A}}$, ${\displaystyle \beta _{h}={\frac {\left({\hat {\sigma }}_{hl_{j_{h}}^{h}}^{2}+{\hat {\sigma }}_{j_{h}l_{j_{h}}^{h}}^{2}/\rho _{h}\right)/{\delta _{hj_{h}l_{j_{h}}^{h}}^{2}}}{\left({\hat {\sigma }}_{ml_{jm}^{m}}^{2}+{\hat {\sigma }}_{j_{m}l_{jm}^{m}}^{2}/\rho _{m}\right)/{\delta _{mj_{m}l_{j_{m}}^{m}}^{2}}}}}$,
• for a design ${\displaystyle d\in S_{B}}$, ${\displaystyle \beta _{d}={\sqrt {\sum _{i\in \Theta _{d}^{*}}{\frac {\sigma _{dl_{d}^{i}}^{2}}{\sigma _{il_{d}^{i}}^{2}}}\beta _{i}^{2}}}}$,

and ${\displaystyle \delta _{ijl}={\bar {J}}_{jl}-{\bar {J}}_{il},}$

${\displaystyle j_{i}\equiv \arg \max _{j\in \Theta ,j\neq i}\prod _{l=1}^{H}{\Pr \left\{{\tilde {J}}_{jl}\leq {\tilde {J}}_{il}\right\}},}$

${\displaystyle l_{j_{i}}^{i}\equiv \arg \min _{l\in {1,\ldots ,H}}\Pr \left\{{\tilde {J}}_{jl}\leq {\tilde {J}}_{il}\right\},}$

${\displaystyle S_{A}\equiv \left\{design\;h\in S\mid {\frac {\delta _{hj_{h}l_{j_{h}}^{h}}^{2}}{{\frac {{\hat {\sigma }}_{hl_{j_{h}}^{h}}^{2}}{\alpha _{h}}}+{\frac {{\hat {\sigma }}_{j_{h}l_{j_{h}}^{h}}^{2}}{\alpha _{j_{h}}}}}}<\min _{i\in \Theta _{h}}{\frac {\delta _{ihl_{h}^{i}}^{2}}{{\frac {{\hat {\sigma }}_{il_{h}^{i}}^{2}}{\alpha _{i}}}+{\frac {{\hat {\sigma }}_{hl_{h}^{i}}^{2}}{\alpha _{h}}}}}\right\},}$

${\displaystyle S_{B}\equiv S\backslash S_{A},}$

${\displaystyle \Theta _{h}={i|i\in S,j_{i}=h},}$

${\displaystyle \Theta _{d}^{*}={h|h\in S_{A},j_{h}=d},}$

${\displaystyle \rho _{i}=\alpha _{j_{i}}/\alpha _{i}.}$

Constrained optimization

The primary performance measure can be called the main objective while the secondary performance measures are referred as the constraint measures. This falls into the problem of constrained optimization. When the number of alternatives is fixed, the problem is called constrained ranking and selection where the goal is to select the best feasible design given that both the main objective and the constraint measures need to be estimated via stochastic simulation. The OCBA method for constrained optimization (called OCBA-CO) can be found in Pujowidianto et al. (2009) ^[13] and Lee et al. (2012). ^[14]

Feasibility determination

Define

• ${\displaystyle k}$: total number of designs;
• ${\displaystyle m}$: total number of performance measure constraints;
• ${\displaystyle c_{j}}$: control requirement of the ${\displaystyle j}$th constraint for all the designs, ${\displaystyle j=1,2,...,m}$;
• ${\displaystyle S_{A}}$: set of feasible designs;
• ${\displaystyle S_{B}}$: set of infeasible designs;
• ${\displaystyle \mu _{i,j}}$: mean of simulation samples of the ${\displaystyle j}$th constraint measure and design ${\displaystyle i}$;
• ${\displaystyle \sigma _{i,j}^{2}}$: variance of simulation samples of the ${\displaystyle j}$th constraint measure and design ${\displaystyle i}$;
• ${\displaystyle \alpha _{i}}$: proportion of the total simulation budget allocated to design ${\displaystyle i}$;
• ${\displaystyle {\bar {X}}_{i,j}}$: sample mean of simulation samples of the ${\displaystyle j}$th constraint measure and design ${\displaystyle i}$.

Suppose all the constraints are provided in form of ${\displaystyle \mu _{i,j}\leq c_{j}}$, ${\displaystyle i=1,2,...,k,j=1,2,...,m}$. The probability of correctly selecting all the feasible designs is

${\displaystyle {\begin{aligned}\mathrm {PCS} =\mathbb {P} \left(\bigcap _{i\in S_{A}}{\Big (}\bigcap _{j=1}^{m}({\bar {X}}_{i,j}\leq c_{j}){\Big )}\cap \bigcap _{i\in S_{B}}{\Big (}\bigcup _{j=1}^{m}({\bar {X}}_{i,j}>c_{j}){\Big )}\right),\end{aligned}}}$

and the budget allocation problem for feasibility determination is given by Gao and Chen (2017) ^[15]

${\displaystyle {\begin{aligned}\max _{\alpha _{1},\alpha _{2},\ldots ,\alpha _{k}}&\mathrm {PCS} \\{\text{subject to }}&\sum _{i=1}^{k}\alpha _{i}=1,\\&\alpha _{i}\geq 0,i=1,2,...,k.\end{aligned}}}$

Let ${\displaystyle I_{i,j}(x)={\frac {(x-\mu _{i,j})^{2}}{2\sigma _{i,j}^{2}}}}$ and ${\displaystyle j_{i}\in \mathrm {argmin} _{j\in \{1,...,m\}}I_{i,j}(c_{j})}$. The asymptotic optimal budget allocation rule is

${\displaystyle {\begin{aligned}{\frac {\alpha _{i}}{\alpha _{i'}}}={\frac {I_{i',j_{i'}}(c_{j_{i'}})}{I_{i,j_{i}}(c_{j_{i}})}},i,i'\in \{1,2,...,k\}.\end{aligned}}}$

Intuitively speaking, the above allocation rule says that (1) for a feasible design, the dominant constraint is the most difficult one to be correctly detected among all the constraints; and (2) for an infeasible design, the dominant constraint is the easiest one to be correctly detected among all constraints.

OCBA with expected opportunity cost

Specifically, the expected opportunity cost is

${\displaystyle {\begin{aligned}EOC=\mathbb {E} _{\mathcal {T}}[\mu _{\mathcal {T}}-\mu _{t}]=\sum _{i=1,i\neq t}^{k}\delta _{i,t}\mathbb {P} ({\mathcal {T}}=i),\end{aligned}}}$

where,

• ${\displaystyle k}$ is the total number of designs;
• ${\displaystyle t}$ is the real best design;
• ${\displaystyle {\mathcal {T}}}$ is the random variable whose realization is the observed best design;
• ${\displaystyle \mu _{i}}$ is the mean of the simulation samples of design ${\displaystyle i}$, ${\displaystyle i=1,2,...,k}$;
• ${\displaystyle \delta _{i,j}=\mu _{i}-\mu _{j}}$.

The budget allocation problem with the EOC objective measure is given by Gao et al. (2017) ^[16]

${\displaystyle {\begin{aligned}\min _{\alpha _{1},\alpha _{2},\ldots ,\alpha _{k}}&\mathrm {EOC} \\{\text{subject to }}&\sum _{i=1}^{k}\alpha _{i}=1,\\&\alpha _{i}\geq 0,i=1,2,...,k,\end{aligned}}}$

where ${\displaystyle \alpha _{i}}$ is the proportion of the total simulation budget allocated to design ${\displaystyle i}$. If we assume ${\displaystyle \alpha _{t}\gg \alpha _{i}}$ for all ${\displaystyle i\neq t}$, the asymptotic optimal budget allocation rule for this problem is

${\displaystyle {\begin{aligned}&{\frac {\alpha _{t}^{2}}{\sigma _{t}^{2}}}=\sum _{i=1,i\neq t}^{k}{\frac {\alpha _{i}^{2}}{\sigma _{i}^{2}}},\\&{\frac {\alpha _{i}}{\alpha _{j}}}={\frac {\sigma _{i}^{2}/\delta _{i,t}^{2}}{\sigma _{j}^{2}/\delta _{j,t}^{2}}},i\neq j\neq t,\end{aligned}}}$

where ${\displaystyle \sigma _{i}^{2}}$ is the variance of the simulation samples of design ${\displaystyle i}$. This allocation rule is the same as the asymptotic optimal solution of problem (1). That is, asymptotically speaking, maximizing PCS and minimizing EOC are the same thing.

OCBA with input uncertainty

Assuming that the uncertainty set contains a finite number of scenarios for the underlying input distributions and parameters, Gao et al. (2017) ^[17] introduces a new OCBA approach by maximizing the probability of correctly selecting the best design under a fixed simulation budget, where the performance of a design is measured by its worst-case performance among all the possible scenarios in the uncertainty set.

Recent Applications of OCBA

• A 2023 study introduced a budget-adaptive allocation rule for OCBA, dynamically adjusting simulation budgets to maximize the probability of correct selection. This approach was validated through synthetic examples and case studies, showcasing its efficiency in identifying optimal system designs. ^[18]
• In 2022, researchers developed a data-driven OCBA method that addresses input uncertainty by updating distribution estimates with streaming data. This method ensures consistent and asymptotically optimal selection of the best design, enhancing decision-making in dynamic environments. ^[2]
• A 2020 study proposed an OCBA-based tree policy for Monte Carlo tree search, optimizing computational resource allocation to maximize the probability of correct action selection. This approach maximizes the probability of correct action selection under limited sampling budgets by dynamically balancing exploration of less-sampled actions and exploitation of promising ones. ^[19]
• The Distributed Asynchronous Optimal Computing Budget Allocation (DA-OCBA) framework applies OCBA principles in a cloud computing environment for simulation optimization. By utilizing idle docker containers and enabling asynchronous execution of simulation tasks, DA-OCBA improves the efficiency of simulation optimization in large-scale systems. The framework demonstrates significant computational savings and scalability, making it particularly useful in applications such as cloud-based resource allocation systems. ^[20]
• A 2021 paper proposes OCBA as a means of ensuring the successful allocation of resources to reduce the impact of hazards and environmental factors that may block the timely transfer of cargo across ports. By implementing OCBA within a digital twin-based framework, decision makers will be able to use real-time data to optimize the number of simulation runs for each recovery alternative. ^[21]

Emerging Research Area: Integration of Machine Learning with OCBA

Predictive Multi-Fidelity Models: Gaussian mixture models (GMMs) predict relationships between low- and high-fidelity simulations, enabling OCBA to focus on the most promising alternatives. Multi-fidelity models combine insights from low-fidelity simulations, which are computationally inexpensive but less accurate, and high-fidelity simulations, which are more accurate but computationally intensive. The integration of GMMs into this process allows OCBA to strategically allocate computational resources across fidelity levels, significantly reducing simulation costs while maintaining decision accuracy. ^[22]

Dynamic Resource Allocation in Healthcare: A Bayesian OCBA framework has been applied to allocate resources in hospital emergency departments, balancing service quality with operational efficiency. By minimizing expected opportunity costs, this approach supports real-time decision-making in high-stakes environments. ^[23] Additionally, the integration of OCBA with real-time digital twin-based optimization has further advanced its application in predictive simulation learning, enabling dynamic adjustments to resource allocation in healthcare settings. ^[24] Furthermore, a contextual ranking and selection method for personalized medicine leverages OCBA to optimize resource allocation in treatments tailored to individual patient profiles, demonstrating its potential in personalized healthcare. ^[25]

Sequential Allocation using Machine-learning Predictions as Light-weight Estimates (SAMPLE): SAMPLE is an extension of OCBA that presents a new opportunity for the integration of machine learning with digital twins for real-time simulation optimization and decision-making. Current methods for applying machine learning on simulation data may not produce the optimal solution due to errors encountered during the predictive learning phase since training data can be limited. SAMPLE overcomes this issue by leveraging lightweight machine learning models, which are easy to train and interpret, then running additional simulations once the real-world context is captured through the digital twin. ^[26]

References

1. Chen, Chun-Hung (1995). "An Effective Approach to Smartly Allocate Computing Budget for Discrete Event Simulation". Proceedings of the 34th IEEE Conference on Decision and Control. IEEE. pp. 2598–2605.
2. Chen, Chun-Hung; Lee, Loo H. (2011). Stochastic Simulation Optimization: An Optimal Computing Budget Allocation. World Scientific Series on Nonlinear Science Series A. Vol. 82. World Scientific. doi:10.1142/7437. ISBN   978-981-4282-64-2.
3. Chen, Wei; Gao, Siyang; Chen, Chun-Hung; Shi, Lei (2014). "An Optimal Sample Allocation Strategy for Partition-Based Random Search". IEEE Transactions on Automation Science and Engineering. 11 (1). IEEE: 177–186. Bibcode:2014ITASE..11..177C. doi:10.1109/TASE.2013.2251881.
4. Lee, Loo Hay; Li, Li Wei; Chen, Chun-Hung; Yap, C. M. (2012). "Approximation Simulation Budget Allocation for Selecting the Best Design in the Presence of Stochastic Constraints". IEEE Transactions on Automatic Control. 57 (12): 2940–2945. doi:10.1109/TAC.2012.2204478 (inactive 28 January 2026).
5. Szechtman, R.; Yücesan, E. (2008). "A New Perspective on Feasibility Determination" (PDF). Proceedings of the 2008 Winter Simulation Conference. pp. 273–280.
6. Gao, Shu; Xiao, Hongsheng; Zhou, Enlu; Chen, Wei (2017). "Robust Ranking and Selection with Optimal Computing Budget Allocation". Automatica. 81: 30–36. doi:10.1016/j.automatica.2017.03.015.
7. Chen, Chun-Hung. "Optimal Computing Budget Allocation (OCBA) for Simulation-based Decision Making Under Uncertainty". Archived from the original on 1 October 2013. Retrieved 9 July 2013.
8. Chen, C. H., J. Lin, E. Yücesan, and S. E. Chick, "Simulation Budget Allocation for Further Enhancing the Efficiency of Ordinal Optimization," Journal of Discrete Event Dynamic Systems, 2000.
9. Szechtman R, Yücesan E (2008) A new perspective on feasibility determination. Proc of the 2008 Winter Simul Conf 273–280
10. Jia QS, Zhou E, Chen CH (2012). efficient computing budget allocation for finding simplest good designs. IIE Trans, To Appear.
11. Trailovic Tekin E, Sabuncuoglu I (2004) Simulation optimization: A comprehensive review on theory and applications. IIE Trans 36:1067–1081
12. Trailovic L, Pao LY (2004) Computing budget allocation for efficient ranking and selection of variances with application to target tracking algorithms, IEEE Trans Autom Control 49:58–67.
13. Pujowidianto NA, Lee LH, Chen CH, Yap CM (2009) Optimal computing budget allocation for constrained optimization. Proc of the 2009 Winter Simul Conf 584–589.
14. Lee LH, Pujowidianto NA, Li LW, Chen CH, Yap CM (2012) Approximation simulation budget allocation for selecting the best design in the presence of stochastic constraints, IEEE Trans Autom Control 57:2940–2945.
15. Gao, S. and W. Chen, "Efficient feasibility determination with multiple performance measure constraints," IEEE Transactions on Automatic Control, 62, 113–122, 2017.
16. Gao, Siyang; Chen, Weiwei; Shi, Leyuan (2017). "A New Budget Allocation Framework for the Expected Opportunity Cost" . Operations Research. 65 (3): 787–803. doi:10.1287/opre.2016.1581.
17. Gao, S., H. Xiao, E. Zhou and W. Chen, "Robust Ranking and Selection with Optimal Computing Budget Allocation," Automatica, 81, 30–36, 2017.
18. Chen, Chun-Hung (1995). "An Effective Approach to Smartly Allocate Computing Budget for Discrete Event Simulation". Proceedings of the 34th IEEE Conference on Decision and Control. IEEE. pp. 2598–2605.
19. Li, Yunchuan; Fu, Michael C.; Xu, Jie (2020). "An Optimal Computing Budget Allocation Tree Policy for Monte Carlo Tree Search". IEEE Transactions on Automatic Control. 67 (6): 2685. arXiv: 2009.12407 . Bibcode:2022ITAC...67.2685L. doi:10.1109/TAC.2021.3088792.
20. Wang, Yu; Tang, Wei; Yao, Yan; Zhu, Fang (2019). "DA-OCBA: Distributed Asynchronous Optimal Computing Budget Allocation Algorithm of Simulation Optimization Using Cloud Computing". Symmetry. 11 (10): 1297. Bibcode:2019Symm...11.1297W. doi: 10.3390/sym11101297 .
21. Zhou, Chenhao; Xu, Jie; Miller-Hooks, Elise; Zhou, Weiwen; Chen, Chun-Hung; Lee, Loo Hay; Chew, Ek Peng; Li, Haobin (2021). "Analytics with digital-twinning: A decision support system for maintaining a resilient port". Decision Support Systems. 143 113496. doi:10.1016/j.dss.2021.113496.
22. Peng, Y.; Xu, J.; Lee, L. H.; Hu, J.; Chen, C. H. (2019). "Efficient Simulation Sampling Allocation Using Multifidelity Models". IEEE Transactions on Automatic Control. 64 (8): 3156–3169. Bibcode:2019ITAC...64.3156P. doi:10.1109/TAC.2018.2886165.
23. Chen, Weiwei; Gao, Siyang; Chen, Wenjie; Du, Jianzhong (2023). "Optimizing Resource Allocation in Service Systems via Simulation: A Bayesian Formulation". Production and Operations Management. 32: 65–81. doi: 10.1111/poms.13825 .
24. Goodwin, Timothy; Xu, Jie; Celik, Niyazi; Chen, Chun-Hung (2024). "Real-Time Digital Twin-Based Optimization with Predictive Simulation Learning" . Journal of Simulation. 18 (1): 47–64. doi:10.1080/17477778.2022.2046520.
25. Gao, Siyang; Du, Jianzhong; Chen, Chun-Hung (2024). "A Contextual Ranking and Selection Method for Personalized Medicine". Manufacturing and Service Operations Management. 26 (1): 167–181. arXiv: 2206.12640 . doi:10.1287/msom.2022.0232.
26. Goodwin, Travis; Xu, Jie; Celik, Nurcin; Chen, Chun-Hung (2024). "Real-time digital twin-based optimization with predictive simulation learning". Journal of Simulation. 18: 47–64. doi:10.1080/17477778.2022.2046520.

External links

• Optimal Computing Budget Allocation (OCBA) for Simulation-based Decision Making Under Uncertainty (Simulation Optimization)