isanet.optimizer.optimizer

Optimizer Module. This module the module provides a basic optimizer class that can be extended to implement an optimizer class by specifying the operations to be performed during each step.

The backpropagation method compute the gradient on the following objective function (Loss)

Loss = 1/2 sum_k (y_i -y_i(w)')^2

and the step method must be implemented.

Moreover it’s provides an EarlyStopping callback that can be used during the fitting phase a model.

class isanet.optimizer.optimizer.EarlyStopping(monitor='val_loss_mse', eps=1e-13, patience=0, verbose=False)

Bases: object

Stop training when a the MSE (Mean squared error) on validation (generalization error) is increasing and exceeds a certain threshold for a finite number of epochs.

Notes

The class define the generalization loss at epoch t to be the relative increase of the validation error over the minimum-so-far (in percent):

GL(t) = 100*(mse_val(t)/min_mse_val(t)-1)

Then it will stop the optimization when the generatilization loss exceeds a certain thresold for a finite number of epochs:

G(t) > eps
Parameters

  • monitor (String) – Quantity to be monitored.

  • eps (float - E.g. 'val_loss_mse') – Threshold that is used to decide whether to stop the fitting of the model (it stops if this is true and after a number of epoch > ‘patience’).

  • patience (integer) – Number of epochs that mse should be worse after which training will be stopped.

  • verbose (boolean, default=False) – Whether to print progress messages to stdout.

check_early_stop(model, epoch, history)

Check if Early Stopping criteria has occurred.

Parameters

  • model (isanet.model.MLP) –

  • epoch (integer) – Current number of epoch (epoch == 0 is the first epoch).

  • history (dict) – Contains, for the current epoch, the values of mse, mee and accuracy for training and validation and the time taken to compute that epoch. This parameter is used to check the value of current MSE (Mean squared error) on validation (generalization error).

Returns

True if the Early stopping has occurred, else False

Return type

boolean

get_min_val()

Returns the min mse on validation if a minimum of the generalization has been reached after overfitting.

Returns

Min of the mse on validation.

Return type

float

get_weights_backup()

Returns the weights’s backup if a minimum of the generalization has been reached after overfitting.

Returns

Weights’s backup.

Return type

list of arrays-like

class isanet.optimizer.optimizer.Optimizer(loss=None, tol=None, n_iter_no_change=None, norm_g_eps=None, l_eps=None, debug=False)

Bases: object

This class implemets the general optimizer.

It must be extended to be used, since method ‘step’ must be implemented.

Parameters

  • loss (String, e.g. 'loss_mse' or 'loss_mse_reg') – When implement this class, a loss to monitor must be specified: MSE or MLE+REG

  • epoch (integer, default=0) – Total number of iterations performed by the optimizer.

  • model (isanet.model.MLP) – Specify the Multilayer Perceptron object to optimize

  • tol (float, default=None) – Tolerance for the optimization. When the loss on training is not improving by at least tol for ‘n_iter_no_change’ consecutive iterations convergence is considered to be reached and training stops.

  • n_iter_no_change (integer, default=None) – Maximum number of epochs with no improvements > tol.

  • norm_g_eps (float, optional) – Threshold that is used to decide whether to stop the fitting of the model (it stops if the norm of the gradient reaches ‘norm_g_eps’).

  • l_eps (float, optional) – Threshold that is used to decide whether to stop the fitting of the model (it stops if the loss function reaches ‘l_eps’).

  • debug (boolean, default=False) – If True, allows you to perform iterations one at a time, pressing the Enter key.

backpropagation(model, weights, X, Y)

Computes the derivative of 1/2 sum_n (y_i -y_i’)

Parameters

  • model (isanet.model.MLP) – Specify the Multilayer Perceptron object to optimize

  • weights (list) – List of arrays, the ith array represents all the weights of each neuron in the ith layer.

  • X (array-like of shape (n_samples, n_features)) – The input data.

  • Y (array-like of shape (n_samples, n_output)) – The target values.

Returns

contains the gradients for each layer to be used in the delta rule. Each index in the list represents the ith layer. (from the first hidden layer to the output layer).:

E.g. 0 -> first hidden layer, ..., n+1 -> output layer
where n is the number of hidden layer in the net.

Return type

list

forward(weights, X)

Uses the weights passed to the function to make the Feed-Forward step.

Parameters

  • weights (list) – List of arrays, the ith array represents all the weights of each neuron in the ith layer.

  • X (array-like of shape (n_samples, n_features)) – The input data.

Returns

Output of all neurons for input X.

Return type

array-like

get_batch(X_train, Y_train, batch_size)
Parameters

  • X_train (array-like of shape (n_samples, n_features)) – The input data.

  • Y_train (array-like of shape (n_samples, n_output)) – The target values.

  • batch_size (integer) – Size of minibatches for the optimizer.

Returns

Each key of the dictionary is a integer value from 0 to number_of_batch -1 and define a batch. Each element is a dictionary and has two key: ‘batch_x_train’ and ‘batch_y_train’ and refer to the portion of data and target respectively used for the training.

Return type

dict of dict

optimize(model, epochs, X_train, Y_train, validation_data=None, batch_size=None, es=None, verbose=0)
Parameters

  • model (isanet.model.MLP) – Specify the Multilayer Perceptron object to optimize.

  • epochs (integer) – Maximum number of epochs.

  • X_train (array-like of shape (n_samples, n_features)) – The input data.

  • Y_train (array-like of shape (n_samples, n_output)) – The target values.

  • validation_data (list of arrays-like, [X_val, Y_val], optional) – Validation set.

  • batch_size (integer, optional) – Size of minibatches for the optimizer. When set to “none”, the optimizer will performe a full batch.

  • es (isanet.callbacks.EarlyStopping, optional) – When set to None it will only use the epochs to finish training. Otherwise, an EarlyStopping type object has been passed and will stop training if the model goes overfitting after a number of consecutive iterations. See docs in optimizier module for the EarlyStopping Class.

  • verbose (integer, default=0) – Controls the verbosity: the higher, the more messages.

Returns

Return type

integer

step(model, X, Y, verbose)

It must be implemented by the derived class (SGD/NCG/LBFGS).

Parameters

  • model (isanet.model.MLP) –

    Specify the Multilayer Perceptron object to optimize

    Xarray-like of shape (n_samples, n_features)

    The input data.

  • Y (array-like of shape (n_samples, n_output)) – The target values.

  • verbose (integer, default=0) – Controls the verbosity: the higher, the more messages.

Raises

NotImplementedError –