Full read through and edit, sending to Mykel.

algorithms/cem-variants-usage-basis.tex

@@ -2,5 +2,5 @@
 S = paraboloid # objective function
 𝐌 = MvNormal([0, 0], [200 0; 0 200]) # proposal distribution
 
-(𝐌, bestₓ, bestᵥ) = ce_surrogate(S, 𝐌; basis=:squared)
+(𝐌, bestₓ, bestᵥ) = ce_surrogate(S, 𝐌; basis=:squared) # squared radial basis surrogate
 \end{lstlisting}

appendices/episodic_ast_appendix.tex

@@ -1,5 +1,5 @@
-\chapter{Alternative FMS Failure Events}\label{sec:application_events}
-This section details alternative failure events investigated and their associated miss distance calculations when stress testing the trajectory predictions in a flight management system (FMS).
+\chapter{Alternative FMS Failure Events}\label{cha:fms_events}
+This section details alternative failure events investigated and their associated miss distance calculations when stress testing the trajectory predictions in a flight management system (FMS) performed in \cref{cha:episodic_ast}.
 Ultimately, analysis of these failure events showed their inadequacy and arc length failures were selected as the primary event.
 
 \section{Tangency Kinks}
@@ -19,33 +19,33 @@ \section{Tangency Kinks}
 \phantom{}
 
 \vspace{-6mm}
-\section{Course Directions}
-In-bound and out-bound course directions may deviate from one another and can be classified as a failure event.
-Closely related to the \textit{disconnection} failure event, the course direction failures can arise when two sequential lateral packets are disconnected. This failure specifically looks for angle differences between the course-out $\theta_{\text{out}}$ and course-in $\theta_\text{in}$ directions of the sequential lateral packets. If this angle difference $\omega = \abs{\theta_\text{out} - \theta_\text{in}}$ is above the threshold $\tau_c = 1\si{\degree}$ then it is classified as a failure.
+\section{Disconnections}
+Disconnected lateral packets occur when two sequential lateral packets are not connected end-to-start, thus leaving a distance $\delta$ between them. A failure occured if this geodesic distance $\delta = \lVert e_i - p_{i+1} \rVert$ is above the threshold $\tau_d = 10$ \si{ft}.
 
 \begin{figure}[!ht]
 \centering
-\resizebox{0.6\columnwidth}{!}{\input{diagrams/course-direction.tex}}
-\caption{Course direction failure event and miss distance.}
-\label{fig:course_direction}
+\resizebox{0.6\columnwidth}{!}{\input{diagrams/disconnection.tex}}
+\caption{Disconnected failure event and miss distance.}
+\label{fig:disconnection}
 \end{figure}
 
-Miss distance is calculated as how close to the threshold $\tau_c$ is the maximum wrapped angle difference between the course-out angle and course-in angle denoted by $\omega$, namely
+Miss distance is calculated as the threshold $\tau_d$ minus the maximum distance between the end points $e_i$ (or $s_i$ if no arc is provided) and the initial point $p_{i+1}$ of the next lateral packet:
 \begin{equation*}
-    d = \tau_c - \max\limits_{\substack{\theta_\text{out} \in L_i\\\theta_\text{in} \in L_{i+1}}}\abs{\theta_\text{out} - \theta_\text{in}}.
+    d = \tau_d \; - \max\limits_{\substack{e_i \in L_i\\p_{i+1} \in L_{i+1}}}\lVert e_i - p_{i+1} \rVert
 \end{equation*}
 
-\section{Disconnections}
-Disconnected lateral packets occur when two sequential lateral packets are not connected end-to-start, thus leaving a distance $\delta$ between them. If this geodesic distance $\delta = \lVert e_i - p_{i+1} \rVert$ is above the threshold $\tau_d = 10$ \si{ft} then a failure occurred.
+\section{Course Directions}
+In-bound and out-bound course directions may deviate from one another and can be classified as a failure event.
+Closely related to the \textit{disconnection} failure event, the course direction failures can arise when two sequential lateral packets are disconnected. This failure specifically looks for angle differences between the course-out $\theta_{\text{out}}$ and course-in $\theta_\text{in}$ directions of the sequential lateral packets. If this angle difference $\omega = \abs{\theta_\text{out} - \theta_\text{in}}$ is above the threshold $\tau_c = 1\si{\degree}$ then it is classified as a failure.
 
 \begin{figure}[!ht]
 \centering
-\resizebox{0.6\columnwidth}{!}{\input{diagrams/disconnection.tex}}
-\caption{Disconnected failure event and miss distance.}
-\label{fig:disconnection}
+\resizebox{0.6\columnwidth}{!}{\input{diagrams/course-direction.tex}}
+\caption{Course direction failure event and miss distance.}
+\label{fig:course_direction}
 \end{figure}
 
-Miss distance is calculated as the threshold $\tau_d$ minus the maximum distance between the end points $e_i$ (or $s_i$ if no arc is provided) and the initial point $p_{i+1}$ of the next lateral packet:
+Miss distance is calculated as how close to the threshold $\tau_c$ is the maximum wrapped angle difference between the course-out angle and course-in angle denoted by $\omega$, namely
 \begin{equation*}
-    d = \tau_d - \max\limits_{\substack{e_i \in L_i\\p_{i+1} \in L_{i+1}}}\lVert e_i - p_{i+1} \rVert
+    d = \tau_c - \max\limits_{\substack{\theta_\text{out} \in L_i\\\theta_\text{in} \in L_{i+1}}}\abs{\theta_\text{out} - \theta_\text{in}}.
 \end{equation*}

chapters/abstract.tex

@@ -1,25 +1,23 @@
-Before safety-critical autonomous systems are deployed into the real-world, we first must validate their safety.
-One common approach is to stress test the systems in simulation.
+Before safety-critical autonomous systems are deployed into the real-world, we must first ensure their validity.
+One common approach for validation is to stress test these systems in simulation.
 This thesis proposes several techniques to aid in efficient stress testing of black-box systems, especially when those systems are computationally expensive to evaluate.
 We first introduce novel variants of the cross-entropy method for stochastic optimization used to find rare failure events.
-The original cross-entropy method relies on enough objective function calls to accurately estimate the optimal parameters of the underlying distribution and may get stuck in local minima.
+The original cross-entropy method relies on enough objective function calls to accurately estimate the optimal parameters of the proposal distribution and may get stuck in local minima.
 The variants we introduce attempt to address these concerns and the primary idea is to use every sample to build a surrogate model to offload computation from an expensive system under test.
 To test our approach, we created a parameterized test objective function with many local minima and a single global minimum, where the test function can be adjusted to control the spread and distinction of the minima.
 % Experiments were run to stress the cross-entropy method variants and results indicate that the surrogate model-based approach reduces local minima convergence using the same number of function evaluations.
 
-To find failure events and their likelihoods in computationally expensive sequential decision making systems, we propose a modification to the black-box stress testing approach called \textit{adaptive stress testing}.
+To find failure events and their likelihoods in computationally expensive open-loop systems, we propose a modification to the black-box stress testing approach called \textit{adaptive stress testing}.
 This modification generalizes adaptive stress testing to be broadly applied to episodic systems, where a reward is only received at the end of an episode.
 To test this approach, we analyze an aircraft trajectory predictor from a developmental commercial flight management system.
 The intention of this work is to find likely failures and report them back to the developers so they can address and potentially resolve shortcomings of the system before deployment. 
-We use a modified Monte Carlo tree search algorithm with progressive widening as our adversarial reinforcement learner.
- % and compared performance to direct Monte Carlo simulations and to the cross-entropy method as an alternative importance sampling baseline.
-The goal is to find potential problems otherwise not found by traditional requirements-based avionics testing.
+We use a modified Monte Carlo tree search algorithm with progressive widening as our adversarial reinforcement learner with the goal of finding potential problems otherwise not found by traditional requirements-based avionics testing.
 % Results indicate that our adaptive stress testing approach finds more failures with higher likelihoods relative to the baselines.
 
-When validating a system that relies on a static validation dataset, one could exhaustively evalute the entire dataset, yet that process may be computationally intractable especially when testing minor modification to the system under test.
+When validating a system that relies on a static validation dataset, one could exhaustively evalute the entire dataset, yet that process may be computationally intractable especially when validating minor modification to the system under test.
 To address this, we reformulate the problem to intelligently select candidate validation data points that we predict to likely cause a failure, using knowledge of the system failures experienced so far.
 We propose an adaptive black-box validation framework that will learn system weaknesses over time and exploit this knowledge to propose validation samples that will likely result in a failure.
-To further reduce computational load, we use a low-dimensional encoded representation of inputs to train the adversarial failure classifier, which selects candidate failures to evaluate.
+To further reduce computational load, we use a low-dimensional encoded representation of inputs to train the adversarial failure classifier, which will select candidate failures to evaluate.
 % Experiments were run to test our approach against a random candidate selection process and we also compare against full knowledge of the true system failures.
 % We stress test a black-box neural network classifier trained on the MNIST dataset,
 % and results show that using our framework, the adversarial failure classifier selects failures about $3$ times more often than random.

chapters/acknowledgments.tex

@@ -15,16 +15,16 @@
 I have also had the pleasure to work with James G. O'Brien as an undergraduate researcher while at Wentworth and I'm grateful for his inspiration and trust he provided me. I owe an enormous debt to everyone I have interacted with while at MIT Lincoln Laboratory over the years---they each played a role in shaping me into a better scientist, engineer, analyst, and friend. This includes Ted Londner, Michael Owen, Wes Olson, Ian Jessen, Adam Panken, Tomas Elder, Cindy McLain, Luis Alvarez, Tan Trinh, Robert Klaus, Justin MacKay, Jared Wikle, Emilie Cowen, Tom Teller, Adam Gjersvik, Sam Wu, Anshu Das, Jack Lepird, Charles Leeper, and Dan Griffith. Particular appreciation goes to Jeff Bezanson for Julia and showing me its divinity. Also a special thanks to Neal Suchy from the Federal Aviation Administration for his admiration and his faith in my work.
 
 Part of this work had the support from GE's Global Research Center and GE Aviation through the Stanford Center for AI Safety.
-I want to thank each of these organizations for their fascinating problems and allowing me to explore research ideas that fit not only my interests but had large industrial impacts.
+I want to thank each of these organizations for their fascinating challenges and allowing me to explore research ideas that fit not only my interests but had large industrial impacts.
 I also want to thank the NASA AOSP System-Wide Safety Project for partially supporting this work and Jerry Lopez, Nicholas Visser, and Joachim Hochwarth for their engineering guidance.
 
 My family and friends have always been there for me, even as we are physically distant.
-My Mom, Dad, brothers Travis and Jake, and sister Emily are a big reason I have core values that have helped me succeed.
+My Mom, Dad, and siblings---Travis, Emily, and Jake---are a big reason I have core values that have helped me succeed.
 Their love and support is infinite and I could not thank them enough for the life they've provided me.
 My wife's family have also been incredibly supportive and I am thankful for their immense encouragement.
 To everyone back in Rockport, MA and beyond, you've seen me grown through every phase in my life, and that bond is irreplaceable; so thank you.
 
-Lastly---but most importantly---I want to thank my wife, Eva Moss, for always being supportive and growing with me during my graduate studies.
+Lastly, but most importantly, I want to thank my wife, Eva Moss, for always being supportive and growing with me during my graduate studies.
 Eva, you always make me laugh, smile, and think deeply, which has shaped me into a better person because of it.
 Your logical thinking helps me check my opinions at the door.
 Your flexibility in leaving our home back in Massachusetts and moving out to California tremendously helped in reducing the stress of graduate school---I love you and I am forever grateful.