Search in a Third is a Python package designed for optimization of hyperparameters in neural networks with an emphasis on moderate computational usage. It utilizes a greedy algorithm guided by heuristic directions that avoid traversing the entire multidimensional space of hyperparameters to achieve an optimal configuration of models efficiently.
- Efficient Hyperparameter Optimization: Focuses on reducing the number of models trained while still achieving optimal results.
- Greedy Algorithm with Heuristic Guidance: Narrows down the search space intelligently to find the best model configurations without exhaustive search.
- Optimized Computational Use: Designed to make the most out of available computational resources, avoiding unnecessary model training.
You can install the package using pip:
pip install search_in_a_third
This implementation guide is designed to quickly guide users through the implementation of Search in a Third.
To install the library, execute the following command:
pip install search_in_a_third
To import the library, you should include in your code:
import search_in_a_third as siat
The first thing you need to do is process the data. For this, I will first give you some tips about the source files that will facilitate your work with search_in_a_third:
- Prioritize the use of files in csv format.
- Do not include the column names in your file.
- Be careful that the attribute separator is not included in the content of the columns of the file.
- Manage nulls before loading the file, the library executes a very basic management of nulls.
Once you have your file ready, you can include the following code in your program:
data_model = siat.build_data_model(url, ['col1', 'col2', 'class'], 'class', 0.3, 'csv', ',')
Preparing the configuration object is the most tedious part that involves the use of Search in a Third. You must be very careful in this step so that the algorithm understands what you want it to do.
The configuration object is a Python dictionary that must have the following mandatory elements:
- loss which must be a list with at least one valid loss function.
- optimize which must be a list with at least one valid optimization function.
- output_layer which must be a list with at least one valid activation function.
- learning_rate which must be a list of two numbers between 0 and 1.
- middle_layers_configuration with at least one Layer type element.
In turn, each element in middle_layers_configuration is a list that must have the following mandatory elements:
- At index 0 a guid type identifier.
- At index 1 a list of two integers that represent the minimum and maximum number of neurons to search for.
- At index 2 a list of two decimal numbers between 0 and 1 that represent the minimum and maximum dropout value to search for.
- At index 3 a list of valid activation functions.
An example of a configuration object is as follows:
configuration_json = {'loss': ['categorical_crossentropy'], 'optimizer': ['Adam'], 'output_layer': ['sigmoid'], 'learning_rate': [0.0001, 0.1], 'middle_layers_configuration': [['51e2ebaa-1f44-41cd-b057-d56aaa42a13c', [1, 100], [0, 0.25], ['relu', 'sigmoid']], ['cdec3ae7-cd55-42bb-8894-2f11952488fc', [1, 100], [0, 0.25], ['relu', 'sigmoid']]]}
To execute the search for optimal configurations for your model's hyperparameters, you should include the following line of code in your program, where configuration_json is the configuration object you generated earlier, data_model is the data object you also generated earlier, and n is the number of iterations you want the algorithm to perform (I recommend starting with 3):
iterations = siat.search(configuration_json, data_model, n)
In the iterations variable, you will have a dictionary where each element represents each of the iterations. Each element will be named after the iteration index converted to String, for example: '0', '1', '2', ...
And each iteration will be a dictionary that in its 'results' element will have a list of result objects as follows:
[{'id': '831865d9-cd6a-492b-a461-c823cce8644e', 'loss': 'categorical_crossentropy', 'model': '', 'result': 0.976190447807312, 'optimizer': 'Adam', 'output_layer': 'sigmoid', 'learning_rate': 0.06670000000000001, 'middle_layers': [{'id': '87b14a0a-eb15-42b5-a998-5f452d1ae622', 'units': 34, 'dropout': 0, 'activation': 'relu'}], 'x_characteristics': 4}, {…….]
In the example, it is a model with an accuracy result of 0.976, using a Categorical Crossentropy loss function, an Adam optimization function with a learning rate of 0.0667, output activated by the Sigmoid function, and a single hidden layer with 34 neurons, without dropout, activated by the Relu function.
If you want to find in this list the element that corresponds to the best-performing model, you can use the following line of code where i corresponds to the integer index of the iteration:
best_performer = iterations['i']['best_performer']
And if you want to discover the most promising search space offered by the iteration for a next iteration, you can run the following code where i corresponds to the integer index of the iteration:
next_configuration = iterations['i']['next_configuration']
This project is licensed under the MIT License - see the LICENSE file for details.
- Diego Larriera
For more information, please contact proflarriera@gmail.com.