Overview
Design Principles
Testing a whole system as a black box maintains its integrity and confidentiality. Integrity ensures we capture as accurately as possible the true risk of an AV policy. Combining tests of individual components does not yield this result, and inserting a scaffolding for testing interfaces between components can fundamentally change the nature of failures. Confidentiality of internal system behavior ensures there is no leakage of proprietary knowledge about the AV policy’s design; this is especially important to establish cooperation of AV manufacturers with regulators or third-party testers.
The TrustworthySearch API is designed so that AV software can be tested as a black box. For each evaluation job the TrustworthySearch API sends a stream of simulation parameters to be evaluated, to your simulation workers.
When a simulation worker recieves a set of parameters it evaluates the system-under-test in closed loop with a model of the environment.
The details of each simulation remain opaque to the search mechanism, only the evaluation of the safety metric is returned.
gRPC and Protocol Buffers
Our API, trustworthy search, uses remote procedure calls (RPCs) implemented using gRPC (link).
Like many RPC systems, gRPC is based around the idea of defining a service, specifying the methods that can be called remotely with their parameters and return types.
gRPC uses protocol buffers (link) as an interface definition language. In the following section(Messages) we have built up a number of objects which can transmit information regarding jobs (search queries) including their initialization, status, and results. Given these messages we then define services which implement RPCs to enable the setup and execution of evaluation jobs.
A new evaluation job is started by a Unary RPC.
Unary RPCs where the client sends a single request to the server and gets a single response back, just like a normal function call.
Given the job id a server streaming RPC sends a sequence of simulation parameters to be evaluated.
Server streaming RPCs where the client sends a request to the server and gets a stream to read a sequence of messages back. The client reads from the returned stream until there are no more messages. gRPC guarantees message ordering within an individual RPC call.
Simulation parameters are distributed to workers and individual simulation results (safety metric evaluations) are uploaded to the search server both via Unary RPCs. When the job is completed the broker or other client with the job id can request a stream of the job results and analysis.
In general the TrustworthySearch service is fixed (the implementation is on the server side); however, you may choose to implement the client side (the running of simulations) in any manner you see fit.