Messages

The message definitions here are based on the Protobuf files available at https://protos.trustworthy.ai.

Distribution

The probabilistic distribution primitives to search over. For simplicity, this demo just searches over standard Gaussians with mean 0, variance 1 or standard Uniform distributons between 0 and 1. The full API exposes a more sophisticated interface with more continuous and discrete distribution primitives (e.g. UNIFORM, EXP, BINOMIAL, MULTINOMIAL).

Parameters

• Type (enum Type)
  • GAUSSIAN
  • UNIFORM

Definition

message Distribution {
  enum Type {
    GAUSSIAN = 0;
    UNIFORM = 1;
  }
}

Example

Create a 2D standard normal distribution:

import trustworthy_search_pb2 as ts

dist_types=[ts.Distribution.GAUSSIAN]*2

JobStyle

Styles of jobs.

Parameters

• Type (enum Type)

  • RISK - find the coverage of all failures as well as return the relative importance of the failure modes.
  • GRID - do grid search over the parameter space.
  • MONTECARLO - do a naive Monte Carlo search for falures over the parameter space
  • STRESSTEST - look for worst-case failures in the parameter space

• Mode (enum Mode)

  • MINIMIZE - try to make the objective function go below a threshold
  • MAXIMIZE - try to make the objective function go above a threshold

Definition

message JobStyle {
  enum Type {
    RISK = 0;
    GRID = 1;
    MONTECARLO = 2;
    STRESSTEST = 3;
  }
  enum Mode {
    MAXIMIZE = 0;
    MINIMIZE = 1;
  }
}

Example

Set the job type to RISK:

import trustworthy_search_pb2 as ts

job_type = ts.JobStyle.Type.RISK

JobRequest

Parameters

• threshold (double threshold) - the threshold above (or below) which you want to find events
• dimension (int32 dimension) - the size of the search space
• dist_types (repeated dist_types) - a list of each dimension's distribution type
• JobStyle.Type (enum JobStyle.Type) - search mode
• JobStyle.Mode (enum JobStyle.Mode) - minimization or maximization of metric
• num_evals (int32 num_evals) - the number of simulations you want to run (not used for RISK - see stop_prob)
• grid_density (repeated int32 grid_density) - how many grid points along each dimension (only used for GRID job_type)
• stop_prob (double stop_prob) - the RISK job will stop at events with likelihood stop_prob or with value treshold, whichever comes first
• stop_prob_confidence (double stop_prob_confidence) - the accuracy with which to measure events with probability stop_prob confidence is defined as 1-coefficient of variation

Definition

message JobRequest {
  double threshold = 1;
  int32 dimension = 2;
  repeated Distribution.Type dist_types = 3;
  JobStyle.Type job_type = 4;
  JobStyle.Mode job_mode = 5;
  int32 num_evals = 6;
  repeated int32 grid_density = 7;
  double stop_prob = 8;
  double stop_prob_confidence = 9;
}

Example

Define the parameters of a job to be sent to the TrustworthySearch service:

import trustworthy_search_pb2 as ts

job_params = ts.JobRequest(threshold=threshold,
                           dimension=2,
                           dist_types=[ts.Distribution.GAUSSIAN]*2,
                           job_type=ts.JobStyle.Type.RISK,
                           job_mode=ts.JobStyle.Mode.MAXIMIZE,
                           num_evals=num_evals,
                           grid_density=grid_density,
                           stop_prob=stop_prob,
                           stop_prob_confidence=stop_prob_confidence
                           )

JobResult

This API returns the parameters and corresponding objective values of all simulations that were run. The 'completed' field indicates whether an objective was sent back to the TrustworthySearch server for a set of 'params' that were sent by the server to the client. This may be false if the client killed the job prematurely before completing all simulations. The full API also gives access to further analysis of the job, including dimensionality-reduced visualizations and access to query importance-samplers. For the RISK job type, the probabilties of various thresholds are also given.

Parameters

• Result (message Result)
  • simid (int32 simid) -
  • params (repeated double params) -
  • objective (double objective) -
  • completed (bool completed) -
• RiskResult (message RiskResult)
  • threshold (repeated double threshold) -
  • probability (repeated double probability) -
  • confidence (repeated double confidence) -
• results (repreated Result results) -
• riskresult (repeated RiskResult riskresults) -

Definition

message JobResult {
  message Result {
    int32 simid = 1;
    repeated double params = 2;
    double objective = 3;
    bool completed = 4;
  }
  message RiskResult {
    repeated double threshold = 1;
    repeated double probability = 2;
    repeated double confidence = 3;
  }
  repeated Result results = 1;
  RiskResult riskresult = 2;
}

Job

Parameters

• jobid (int32 jobid) - each job is uniquely defined by its id
• info (string info) - information from the server to notify the client about the job

Definition

message Job {
  int32 jobid = 1;
  string info = 2;
}

BudgetRequest

Parameters

• stop_prob (double stop_prob) - the RISK job will stop at events with likelihood stop_prob or with value treshold, whichever comes first
• stop_prob_confidence (stop_prob_confidence) - the accuracy with which to measure events with probability stop_prob_confidence is defined as 1-coefficient of variation

Definition

message BudgetRequest {
  double stop_prob = 1;
  double stop_prob_confidence = 2;
}

Budget

Parameters

• num_evals (int32 num_evals) - estimated number of evaluations needed for a RISK job defined by a BudgetRequest

Definition

message Budget {
  int32 num_evals = 1;
}

SimParams

Parameters

• jobid (int32 jobid) - identifies the job
• simid (int32 simid) - is a hash that identifies the simulation
• params (repeated double params) - are the scenario parameters to use

Definition

message SimParams {
  int32 jobid = 1;
  int32 simid = 2;
  repeated double params = 3;
}

SimResult

Parameters

• jobid (int32 jobid) - identifies the job
• simid (int32 simid) - is a hash that identifies the simulation
• objective (double objective) - the scalar result of the simulation

Definition

message SimResult {
  int32 jobid = 1;
  int32 simid = 2;
  double objective = 3;
}

BrokerPort

Parameters

• port (int32 port) -

Definition

message BrokerPort {
  int32 port=1;
}

BrokerSimParams

Parameters

• simparams (SimParams simparams) -
• port (int32 port) -

Definition

message BrokerSimParams {
  SimParams simparams=1;
  int32 port=2;
}

BatchBrokerSimParams

Collects all of the parameters of many jobs into one object

Parameters

• param_list (repeated BrokerSimParams param_list)

Definition

message BatchBrokerSimParams {
  repeated BrokerSimParams param_list=1;
}