Algorithms for blackbox falsification of convolutional neural networks

testing, safety, deep, learning, computer, vision
pip install SafeCV==0.0.3



Vision based algorithms for black-box falsification and safety testing of convolutional neural networks

SafeCV is mainly concerned with the falsification of deep, feed-forward convolutional neural networks. The package requires openCV, Keras, numpy and pomegranate. Running the examples requires matplotlib in addition.

Installation with:

pip install SafeCV

As of right now, the package contains two main algorithms:

  • DFMCS [Depth First Monte-Carlo Search] - A single monte-carlo based manipulation simulation based on human perception
  • MCTS [Monte-Carlo Tree Search] - A monte-carlo tree search method for creating robust adversarial examples

Later, we will include a two-player game formulation for studying MNIST and CIFAR10 networks.


Each run of DFMCS and MCTS must first initialize parameters:

params_for_run = MCTS_Parameters(image, class, model)

These parameters can be changed to fit the desired performance of the algorithm. Then, the algorithm can be run with:

best_image, sev, prob, statistics = MCTS(params_for_run)


  • best_image is the best adversarial example that was found,
  • sev is the L0 Severity of the adversarial example,
  • prob is the softmax output corresponding to the best adversarial example,
  • statistics is a tuple of different runtime statistics that help illucidate perfomance

Runtime Parameters

Finally, we give a brief documentation of what each of the parameters controls

  • model - The Neural Network model to be queried
  • ORIGINAL_IMAGE - The unmodified copy of the image (implicitly protected)
  • TRUE_CLASS - The expected classification
  • manip_method - a method that takes in two variables (pixel value and a constant) and dictates how the input will be manipulated
  • VISIT_CONSTANT - Number of manipulations to make per time step
  • SIGMA_CONSTANT - Varience to use when formulating the saliency distribution
  • X_SHAPE - size of the X dimension of the input
  • Y_SHAPE - size of the Y dimension of the input
  • predshape - how to reshape the input before feeding it to the network
  • kp, des, r - Keypoint values returned from an OpenCV feature detector
  • EPSILON - Constant to be fed into the manipulation method
  • verbose - Determines if the user wants to see all of the runtime outputs in the console
  • preprocess - User defined method to say how an image should be preprocessed (default is to reshape to predshape and return)
  • predict - Method for predicting the class and probability of an input
  • small_image - if image is less than 50x50
  • inflation_constant - When small_image is true, how much should we inflate the input to get a good saliency distribution
  • backtracking_constant - How many pixels to remove at each backtracking step