Distributional Soft Actor Critic

Distributional Soft Actor Critic (DSAC) is a suite of model-free off-policy reinforcement learning algorithms, tailored for learning decision-making or control policies in complex systems with continuous action spaces ^[1] . Distinct from traditional methods that focus solely on expected returns, DSAC algorithms are designed to learn a Gaussian distribution over stochastic returns, called value distribution. This focus on Gaussian value distribution learning notably diminishes value overestimations, which in turn boosts policy performance. Additionally, the value distribution learned by DSAC can also be used for risk-aware policy learning ^{[2] [3] [4]} . From a technical standpoint, DSAC is essentially a distributional adaptation of the well-established soft actor-critic (SAC) method ^[5] .

To date, the DSAC family comprises two iterations: the original DSAC-v1 and its successor, DSAC-T (also known as DSAC-v2), with the latter demonstrating superior capabilities over the Soft Actor-Critic (SAC) in Mujoco benchmark tasks. The source code for DSAC-T can be found at the following URL: Jingliang-Duan/DSAC-T.

Both iterations have been integrated into an advanced, Pytorch-powered reinforcement learning toolkit named GOPS ^[6] : GOPS (General Optimal control Problem Solver).

References

1. Duan, Jingliang; et al. (2021). "Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors". IEEE Transactions on Neural Networks and Learning Systems. 33 (11): 6584-6598. doi:10.1109/TNNLS.2021.3082568.
2. Yang, Qisong; et al. (2021). "WCSAC: Worst-case soft actor critic for safety-constrained reinforcement learning". AAAI.
3. Wu, Jingda; et al. (2022). "Uncertainty-aware model-based reinforcement learning: Methodology and application in autonomous driving". IEEE Transactions on Intelligent Vehicles.
4. Yang, Qisong; et al. (2023). "Safety-constrained reinforcement learning with a distributional safety critic". Machine Learning.
5. Haarnoja, Tuomas; et al. (2018). "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor". ICML.
6. Wang, Wenxuan; et al. (2023). "GOPS: A general optimal control problem solver for autonomous driving and industrial control applications". Communications in Transportation Research.