Bilkent University
Department of Computer Engineering
M.S.THESIS PRESENTATION
Uncertainty-Aware Safety Propagation Critics for Safe Reinforcement Learning
Kutay Demiray
Master Student
(Supervisor: Asst. Prof. Özgür Salih Öğüz)
Computer Engineering Department
Bilkent University
Abstract: Safe reinforcement learning (RL) seeks to optimize long-term performance while ensuring that specified safety constraints are satisfied. This requirement is important in many real-world applications, but becomes difficult to meet when cost estimates are inaccurate or the available data are limited. In model-free actorcritic algorithms, learned cost critics may be unreliable in insufficiently explored regions, which can result in constraint violations during both training and deployment. This thesis introduces Uncertainty-Aware Safety Propagation Critics (USPC), a safe RL approach that uses epistemic uncertainty to construct conservative estimates of future cost. USPC maintains an ensemble of cost critics whose disagreement is used to quantify uncertainty and form an upper confidence bound on the predicted cost. Inspired by safety propagation in safe Bayesian optimization, USPC further introduces a safe set network that learns a pessimistic surrogate of the cost action-value function and enables safety information to be propagated across continuous action spaces. This surrogate can replace the standard cost critic in existing off-policy safe RL algorithms, making policy updates less likely to rely on uncertain or overly optimistic cost estimates. Experiments on several Safety Gymnasium benchmark tasks show that USPC reduces the frequency and magnitude of constraint violations in most environments while preserving competitive reward performance relative to the underlying baselines. This thesis also provides theoretical analysis of the proposed approach, including a conservativeness result for the safe set network and approximation guarantees for the finite anchor-based target construction.
DATE: August 31, Monday @ 10:00
Place: EA 516