Basic neural network library

pip install milligrad==1.2.0



This repo is a learning exercise for me to get a deeper understanding in deep learning and especially in dense neural networks. Basically I took Andrej Karpathy's micrograd and added just some extra functions in order to train a neural network with just a bunch of lines of code.

DenseNN initialization

To create a neural network in milligrad you have to call a DenseNN object. You have to specify the following parameters:

  • nin: number of inputs of the first layer of the neural network. It is an integer and it has to match the size of inputs (a parameter of nn.Train() method).

  • nouts: a list (or tuple) specifiying the number of neurons of each neural network layer. Note that it must be a 1-dimensional array and its length must match the one of activations array.

  • activations: a list (or tuple) of strings specifiying the activations of each neural network layer. Note that it must be a 1-dimensional array and its length must match the one of nouts array. Right now only 5 activations are available: linear, ReLU, sigmoid, tanh, softmax.

One useful DenseNN method is nn.parameters() with which you can access DenseNN's parameters (weights and biases). Every parameter is a Value object containing 2 attributes:

  • data: the numerical value of that neural network weight or bias.

  • grad: the gradient that has been computed by backward propagation.

By printing a DenseNN you get a syntethic representation. It has to be read from left to right and it shows the structure of every layer of the neural network. For example:

[Layer of [ReLUNeuron(3), ReLUNeuron(3)], Layer of [TanhNeuron(2)]]

means that the first layer is made of:

2 neurons with ReLU activation, each one with 3 input neurons

and the second layer is made of:

1 neuron with tanh activation and it takes 2 neurons (previous layer dimension) as inputs

Training and Predictions

After initializing the neural network, essentially you have to use only 2 other functions: nn.Train() and nn.Predict():

  • nn.Train() takes as inputs:

    • inputs: a list containing inputs value to feed in the nn neural network in order to train it. Note that inputs must be a 2-dimensional array (i.e. [[1, 2, 3, ...], [4, 5, 6, ...], [7, 8, 9, ...], ...]).

    • predictions: a list containing predictions to feed in the nn neural network in order to train it. Note that inputs must be a 1-dimensional array if you want binary or linear classification (i.e. [1, 2, 3, ...]) or a 2-dimensional array if you want a multi-class classification (i.e. [[1, 0, 0, ...], [0, 1, 0, ...], [0, 0, 1, ...], ...]). Note that in the latter case predictions array must be one-hot encoded.

    • loss: a string containing the name of the loss function that has to be used in order to train the neural network. Currently available loss functions are: MeanSquaredError and BinaryCrossEntropy, CategoricalCrossEntropy. By default loss is set to ''. Since no loss function is applied by default, make sure to set properly this parameter.

    • epochs_number: an integer that specifies the number of training steps.

    • batch_size: an integer that specifies the batch size for gradient descent (hence how many examples the training function has to take before updating the weights and biases). For example batch_size=1 corresponds to stochastic gradient descent, while batch_size=len(inputs) corresponds to batch gradient descent (every batch size between 1 and len(inputs) corresponds to mini batch gradient descent).

    • learning_rate: a float containing the learning rate for the weights update (param = param - learning_rate * gradient).

    • decay_rate: a float specifying the decay rate for learning rate decay (learning_rate = learning_rate/(1 + decay_rate * epoch)). By default is set to 0, so no learning rate decay is done without setting this parameter. Note that high learning_rate_decay makes learning rate decays faster.

    • L2_regularization: a float specifying the regularization factor for L2 regularization (loss = loss + L2_regularization * sum(weights^2)). Note that by default L2_regularization is set to 0 so no regularization is done.

  • nn.Train() gives as output: a DenseNN object

  • nn.Predict() takes as inputs:

    • input: a list containing one training set sample that will be fed in the neural network. Note that input length must match the size of nn zero-th layer.
  • nn.Predict() gives as output the final node (a Value object) of the nn neural network containing the prediction value (nn.Predict(input).data).