Search Space
We reference our team's earlier research on scenario description languages (SDL's) presented in https://ieeexplore.ieee.org/document/8088654. In our view a scenario is a meta-simulation which encodes a set of possible realizations of a driving task.
Scenario Components
A scenario, \(\mathbf{S}\), is a tuple which consists of a set of agents, \(A\), that includes the ego-vehicle, pedestrians, and other vehicles, a world, \(W\), which captures static features of the environment like the road, a set of traffic laws \(\mathcal{L}\), a goal \(\Phi\) to be achieved by the ego-vehicle in this scenario, a set of initial states \(\text{Init}\) that captures the initial states of all vehicle agents, and exit conditions \(\mathcal{E}\) that define when a scenario is over.
Agents \((A)\)
There are 2 agents: the ego-vehicle (\(a_1\)) and an environment-vehicle \(a_2\). It is unrealistic to assume perfect knowledge of the environment vehicle's intentions and dynamics. Thus, the latter should be modeled non-deterministically, e.g. \(\dot{x}=v_x, \dot{v}_x \in [0,1]\frac{m}{s^2}\) or probabilistically, e.g. \(\dot{x}=v_x, \dot{v}_x \sim P\) for some distribution \(P\).
World \((W)\)
Static aspects of an AVs environment can be treated as a special kind of agent. While they do not evolve over time, they can still be initialized from a distribution or interval. For example: roads can be described as a graph of lanes where each lane is a finitely-parameterized curve or sequence of curves: e.g. straight lines (with parameter: \(\text{length}\)), cubic splines (parameters: \(\text{length},\text{curvatures}\)), or polylines. In this sense even the road geometry can be made a searchable parameter of the scenario.
Laws \((\mathcal{L})\)
The laws are fixed in a given scenario, common encodings could include Metric Temporal Logic (MTL) (https://link.springer.com/article/10.1007/BF01995674) or Signal Temporal Logic (STL) (https://link.springer.com/chapter/10.1007/978-3-540-30206-3_12). Note that the use of a temporal logic is not required; most reinforcement-learning reward functions work. In the example, one law imposes a speed limit of 50 (in whatever desired units of length and time you prefer) for the duration \(T\) of the scenario: \(l = \text{Always}_{[0,T]} (v < 50)\).
Goals \((\Phi)\)
The goal \(\Phi\) of the ego vehicle is of the form "Ego must reach some region within \(N\) time units". In the example scenario, the goal region is the green rectangle, expressing that the vehicle should turn onto the main road within some bounded time horizon.
Initial States (\(\text{Init}\))
An AV \(a\) estimates its state \(x_{0}\) to within some bounded error (due to measurement noise), so it only knows that \(x_0\) is in some set \(Init_{a}\); moreover there may be a variety of valid starting conditions for both the world and agents. Thus, a finite number of scenario objects may characterize the system-under-test's operational domain, but for each scenario instance there are an infinite number of variations. The set \(\text{Init}\) is the product of all vehicles' and world parameters initial sets. The initial sets of the 2 cars are shown in light grey in the example scenario. When using the TrustworthySearch API we consider the function \(\text{Init}\) to sample \(X\), a random variable, from a distribution \(P_{0}\); if the distribution is unknown it may be appropriate to instead utilize a uniform distribution over the interval of valid initial conditions.
Exit Conditions \((\mathcal{E})\)
Finally, the scenario ends either when the ego vehicle leaves the region defined by the T-Junction (the green region in the diagram) and proceeds to the next navigation task, or a timeout occurs (the red region in the diagram). Other exit conditions could include a "handover" signal initiated by any part of the stack or an accident.
From Scenarios to Simulations
In the following sections of the integration guide we will treat each scenario component in detail. First, however, lets understand how to generate simulations from the components of a scenario object.
Simulation parameters and their distributions
Below we describe a possible parametrization for a highway scenario, including vehicle variables for vehicle poses/behaviors and weather. This description is an excerpt from the paper https://arxiv.org/abs/1912.03618, where we use the open-source CARLA simulator to test Comma AI's OpenPilot on highway scenarios.
Agent Variables
For the behaviors of the environment vehicles we follow the parametrization of https://arxiv.org/abs/1811.00145. Namely, we search over the last layer of a neural network GAIL generator model. We use \(\xi \in \mathbb{R}^{404}\) to denote the weights of the last layer of the neural network trained to imitate human-like driving behavior with \(\xi \sim \mathcal{N}(\mu_0, \Sigma_0)\), with the mean and covariance matrices learned via empirical bootstrap. The input for this GAIL generator is described in https://arxiv.org/abs/1811.00145. The single network defined by \(\xi\) drives the behavior for all environment agents in a particular simulation rollout.
Initialization Variables
Initial positions for each vehicle are independently given by the pair \((S,T)\), where \(S\sim 500 \text{Beta}(2,2) + 200\) denotes the longitudinal position along the stretch of road with respect to the start of the road, and \(T\sim0.5 \text{Beta}(2,2)-0.25\) denotes the lateral distance from the lane center with left being positive. Both have units in meters. Initial velocities are independently given by \(V \sim 10 \text{Beta}(2,2)+15\) with units in meters/second.
Perception variables
If the scenario uses a rendering engine or sensor simulation tool in the loop there will be additional factors which can be used to parameterize the scenario. In what follows we describe parameters for the CARLA simulator; for other proprietary simulators a similar methodology may be applied. In order to define the weather and lighting conditions 4 parameters are defined: precipitation on the ground (\(P_g\)), the altitude angle of the sun (\(A\)), cloudiness (\(C\)), and precipitation in the air (\(P_a\)). Precipitation on the ground follows the distribution \(P_g\sim 50\text{Uniform}(0,1)\), where the units are proprietary for the CARLA simulator. Sun altitude follows the distribution \(A\sim 90\text{Uniform}(0,1)\), whose units are in degrees. Letting \(C_b\sim\text{Beta}(2,2)\) and \(C_u\sim\text{Uniform}(0,1)\), cloudiness has a distribution given by the following mixture: \begin{equation} C \sim 30C_b \mathbf{1}{C_u < 0.5} + (40C_b+60)\mathbf{1}{C_u \ge 0.5}, \end{equation} where \(\mathbf{1}\{\cdot\}\) is the indicator function and the units of \(C\) are in proprietary units for CARLA. Precipitation in the air is a deterministic function of cloudiness, \(P_a = C\mathbf{1}\{C \ge 70\},\) where the units are again proprietary CARLA units.
Practical Considerations
The following aspects of setting up a scenario for use with TrustworthySearch are important to consider in order to reliably interpret the results.
Semantics of Exit Conditions
Simulations can "complete" for a variety of reasons. In some cases the simulation may timeout or crash. Determining how such issues are reported back to the module responsible for proposing the next simulations to run (e.g. the TrustworthySearch API) is important. Should those samples be rejected? Should they receive a large penalty? Should they include a binary indicator function? Any of the decisions may be valid for your specific use case, and these choices must be made judiciously before starting a test.
Determinism
Knowing whether your simulator is deterministic or not can change the way you search for failures (or use the TrustworthySearch API). If the simulator is deterministic but the real system is not, is there value in supporting non-deterministic hardware-in-the-loop executions? In our opinion, determinism is better, but we can support non-deterministic simulations if the proper variable is set in the job request.
Time
The perception and planning elements may use a real-time clock to stamp vehicle-state messages. However, the simulation clock does not necessarily match wall-clock time. To address this problem, the simulator should supply the system under test with a clock signal.