Guide to use trained model for predictions

This guide explain how to use a trained model to make predictions of new data set

Note: This notebook is binder ready, please uncomment the following cell to install ``HARDy`` in the binder environment

 #!: $(pip install ../../)

1. Preparing transformation configuration file

Just like training, predictions using HARDy requires a transformation configuration .yaml file. In this example, only best performing transformation \(log(q)\) vs. \(d(I(q))/d(q)\) is considered. The configuration file is shown in image below:

The instructions for building transformation configuration file are available at How to write configuration files

Extracting new data into the binder directory

!tar -xzf new_data_set.tar.gz

2. Importing required modules

import hardy
import os

Using TensorFlow backend.

3. Preparing the new data set to be fed into trained model

Applying transformations to the data

• Defining the location for transformation configuration

transformation_config_path = './scattering_tform_config.yaml'

• Collecting the filenames of new data set having only .csv file extension

new_data_file_list = [item for item in os.listdir('./new_data_set/') if item.endswith('.csv')]

• Loading transformation information from the configuration file

tform_command_list, tform_command_dict = hardy.arbitrage.import_tform_config(transformation_config_path)

Successfully Loaded 1 Transforms to Try!

• Defining variables for transforming images

run_name = 'log_q_der_I'
new_datapath = './new_data_set/'
classes = ['sphere', 'cylinder', 'core-shell', 'ellipsoid']
project_name = 'new_data_set'
scale = 0.2
target_size = (100, 100)

Please note that the order of classes must be same as used for training of Machine Learning model. Moreover, the scale and target_size must also be the same as used for training.

• Using data wrapper function to generate the rgb images

hardy.data_wrapper(run_name=run_name, raw_datapath=new_datapath, tform_command_dict=tform_command_dict,
                   classes='d', scale=0.2)

Loaded  44 of 44        Files    at rate of 370 Files per Second
         Success!        About 0.0 Minutes...
Making rgb Images from Data...  Success in 2.12seconds!
That Took 2.29 Sec !

0

Please note that the value for classes used in data_wrapper module is a string rather than classes. This is being done to use same module for different functionalites i-e for training and predictions. Since, the new data set doesn’t have the labels, the class ‘d’ is used as assumed class

• The data_rapper will apply the numerical and visual transformations and pickle the data into the new_datapath folder

• To load the transformed data, following code is used:

transformed_data = hardy.handling.pickled_data_loader(new_datapath, run_name)

• The data now needs to be converted into iterator acceptable to tensorflow. This can be done by

new_data_set = hardy.to_catalogue.test_set(image_list=transformed_data, target_size=target_size,
                                           classes=classes, color_mode='rgb',
                                           iterator_mode='arrays', batch_size=len(new_data_file_list),
                                           training=False)

The number of unique labels was found to be 1, expected 4

The argument for training is kept false, to avoid tagging classes in data set. During training, it is kept as true so that model can seek validation of predicted outcomes

The data is now ready to be used for predictions

4. Making predictions

• Loading the model

trained_model = hardy.cnn.save_load_model('./best_model.h5', load=True)

• Making predictions

hardy.reporting.model_predictions(trained_model, new_data_set, classes, transformed_data)

	Filenames	Predicted_Labels	Probabilities
0	12ab	cylinder	[0.164, 0.745, 0.019, 0.072]
1	36ab	cylinder	[0.146, 0.738, 0.019, 0.097]
2	28ab	core-shell	[0.687, 0.086, 0.06, 0.167]
3	24ab	cylinder	[0.096, 0.809, 0.001, 0.094]
4	7ab	cylinder	[0.272, 0.597, 0.032, 0.099]
5	5ab	cylinder	[0.163, 0.77, 0.001, 0.066]
6	9ab	core-shell	[0.816, 0.023, 0.056, 0.106]
7	10ab	cylinder	[0.15, 0.794, 0.0, 0.056]
8	38ab	cylinder	[0.388, 0.477, 0.049, 0.086]
9	26ab	cylinder	[0.204, 0.666, 0.021, 0.109]
10	34ab	core-shell	[0.82, 0.035, 0.026, 0.119]
11	1ab	core-shell	[0.646, 0.226, 0.021, 0.107]
12	30ab	cylinder	[0.114, 0.677, 0.033, 0.176]
13	41ab	cylinder	[0.18, 0.54, 0.084, 0.195]
14	22ab	core-shell	[0.802, 0.107, 0.069, 0.023]
15	14ab	cylinder	[0.192, 0.493, 0.102, 0.213]
16	18ab	cylinder	[0.132, 0.686, 0.029, 0.153]
17	20ab	cylinder	[0.171, 0.703, 0.021, 0.104]
18	43ab	cylinder	[0.354, 0.545, 0.015, 0.086]
19	32ab	core-shell	[0.725, 0.175, 0.056, 0.044]
20	16ab	cylinder	[0.126, 0.693, 0.041, 0.14]
21	3ab	cylinder	[0.16, 0.567, 0.061, 0.212]
22	6ab	cylinder	[0.157, 0.759, 0.001, 0.084]
23	29ab	cylinder	[0.072, 0.831, 0.001, 0.097]
24	37ab	cylinder	[0.141, 0.669, 0.038, 0.152]
25	25ab	cylinder	[0.315, 0.57, 0.032, 0.083]
26	13ab	cylinder	[0.026, 0.862, 0.001, 0.111]
27	27ab	core-shell	[0.832, 0.111, 0.032, 0.024]
28	39ab	core-shell	[0.696, 0.056, 0.055, 0.193]
29	35ab	cylinder	[0.453, 0.51, 0.011, 0.026]
30	11ab	core-shell	[0.805, 0.118, 0.048, 0.029]
31	4ab	core-shell	[0.782, 0.044, 0.038, 0.136]
32	8ab	core-shell	[0.458, 0.431, 0.021, 0.09]
33	15ab	core-shell	[0.82, 0.03, 0.122, 0.028]
34	19ab	core-shell	[0.712, 0.156, 0.088, 0.044]
35	31ab	cylinder	[0.392, 0.494, 0.031, 0.084]
36	23ab	cylinder	[0.158, 0.794, 0.0, 0.047]
37	40ab	cylinder	[0.137, 0.701, 0.027, 0.135]
38	0ab	cylinder	[0.19, 0.557, 0.056, 0.197]
39	2ab	cylinder	[0.182, 0.507, 0.062, 0.248]
40	17ab	cylinder	[0.168, 0.54, 0.069, 0.223]
41	42ab	cylinder	[0.164, 0.792, 0.0, 0.044]
42	21ab	core-shell	[0.819, 0.097, 0.064, 0.02]
43	33ab	ellipsoid	[0.111, 0.025, 0.68, 0.183]

This will generate a pandas dataframe outlining the file names, their predicted labels, and probabilities of predictions for each class. It is shown in image below:

---