Ph.D. Defense

Nathan Williams

(Faculty Advisor: Professor Dimitri Mavris)

"Graph-ReCon: A Graph-Based Framework for Induced Relational Context in Model-Free Reinforcement Learning"

Wednesday, April 20

12:30 p.m.

Weber, CoVE

Abstract:

Model-Free Reinforcement Learning (RL) has shown incredible success across many fields, including automation, finance, and healthcare. However, despite these achievements, Model-Free RL agents continue to face two critical challenges that inhibit their utility: (1) the Performance Challenge, how well the agent works, characterized by poor sample efficiency, instability in training, and mediocre generalization; and (2) the Interpretability Challenge, how well the agent can be understood, as agents remain difficult to understand and explain to stakeholders. Thus, this dissertation is guided by a singular purpose – to advance Model-Free Reinforcement Learning by improving agent performance and interpretability.

Although these challenges may seem distinct, they share a common dependency – the state representation, i.e., how the agent views and encodes its environment. State Representation Learning (SRL), a fundamental component of Model-Free RL, offers a variety of methods that seek to do just that – enhance the state representation. However, existing SRL approaches primarily focus on performance and do not explicitly address interpretability. Moreover, across SRL classes, there exists no unified approach that enables online, structured, relational state representation learning in a manner that is both environment-agnostic and algorithm-agnostic. This deficiency constitutes the central technical gap addressed in this work.

To address this gap, this dissertation introduces Graph-ReCon: a graph-based framework for induced relational context in Model-Free RL. Graph-ReCon extends the traditional Model-Free RL pipeline by incorporating a Relational Context component, consisting of three core parts: (1) an experience graph that captures agent-environment interactions; (2) a Graph Neural Network (GNN) to process the experience graph; and (3) two auxiliary losses to further refine the state representation space.
The Graph-ReCon framework is evaluated in two complementary parts. Part I, consisting of three experiments, demonstrates consistent improvements in sample efficiency, training stability, generalization, and interpretability across multiple environments and algorithms. Part II extends this evaluation to cybersecurity, highlighting Graph-ReCon’s practical utility in a high-stakes domain. Collectively, these results demonstrate that incorporating relational context leads to more effective and more interpretable agents. Ultimately, this work is grounded in a simple, but elegant principle: decisions are important, but the relationships between decisions are no less important than the decisions themselves.

Committee:

Dr. Dimitri Mavris (advisor), School of Aerospace Engineering
Dr. Woong-Je Sung, School of Aerospace Engineering
Dr. Dalton Lin, Sam Nunn School of International Affairs 
Dr. Anqi Wu, School of Computational Science and Engineering
Dr. Xiuwei Zhang, School of Computational Science and Engineering
,