Handling

Submodules

handling module

hardy.handling.handling.ask_file_list()

Alternative to get_file_list, just makes a tkinter window and asks the user to select the files. Written easiy so we don’t have to remember tkinter

hardy.handling.handling.check_dir_path(dir_path, files_contain=['.csv'], n_required=1, raise_err=False)

Check if a directory contains the files you want:

Parameters:

dir_path: str

Path to check (required)

files_contain: list

list of strings to check. Files must contain ALL strings to pass.

(Default is set to look for .csv)

n_required: int

Number of successful files required to pass the test

(Default is 1 file)

raise_err: bool

Failure Handling. whether to Return Failure or raise an Error.

(Default is FALSE, which will not raise errors.)

Returns

——-

BOOLEAN (T/F), did we find all the required files?

Error Message, to use in selecting a folder if we failed.

hardy.handling.handling.classes_from_fnames(file_list=None, path=None, expect=2, print_ok=True, from_serials=False)

Given a list of file names, determine if there are classifying endings that split the data into “expect” (default 2) Groups.

Returns:

classification_list: list

list containing the classes/labels to separate the data in.

hardy.handling.handling.get_file_list(dir_path='../local_data', str_has=['.'], str_inc=['.'], ftype='csv', interact=False)

Get a list of file paths to open, which fulfill certain criteria. (Alternative to single/multiselect File Dialog, or hard-coded file names

Parameters:

dir_path: str

Path to check initially. (hard-coded default for now. can we globally config?)

str_has: numpy.array

Basic filter parameters, defaults just for testing now. str_has: AND filter (Must contain all in list)

str_inc: numpy.array

OR filter (Must contain at least one)

ftype: str

File type to check for. Default csv

interact: bool

whether to use file dialogs if the path fails, or just error out.

Returns:

files_wanted: tuple

TUPLE of file names that pass the tests (built as list)

dir_path: str

Final (successful) directory path used. (from cwd, or from base)

hardy.handling.handling.pickled_data_loader(raw_datapath, run_name)

Loads the pickled data

Parameters:

raw_datapath: str

location of raw data

run_name: str

transformation name from the dictionary

hardy.handling.handling.read_csv(full_fname, skiprows=0, last_skiprows=None, maxskip=100)

Function to loop through pandas read_csv, checking the data

and trying again if it’s bad.

Note:

Will Return ONLY columns which are interger or floats.

No lists, no strings, no silly things.

Parameters:

full_fname: str

joined path and file name so that we can load the file

try_skiprows: int

this replaces the hard “skiprows” in the old functions. It’ll be the first we try.

last_skiprows: int (optional)

Function Output of the successful skiprows #. To be re-fed into the function on the next loop occurance to speed up.

max_skip: int

loop size. Will error if you skip this many rows.

Returns:

fdata : Pandas DataFrame

The dataframe obtained from teh csv file

last_skiprows : int

the value fo the last row skipped

pre_processing module

hardy.handling.pre_processing.classes_folder_split(path, classes=['noise', ''], class_folder=['noisy', 'not_noisy'], file_extension='.png')

Functions that separates the files into folders representing each class

Parameters:

path : str

string containing the path to the files where to create the training and validation sets folders

classes: list

A list containing strings of the classes the data is divided in. The classes are contained in the filename as labels.

class_folder: list

A list of string containing the name of the folders to be create to split the files into the right classes.

file_extension: str

the extension of the file to be moved. The default value is .png

Returns:

list_of_folders: list

A list of stings representing the path of the new folders created while splitting the data into classes

hardy.handling.pre_processing.hold_out_test_set(path=None, number_of_files_per_class=100, seed=None, classes=['noise', ''], file_extension='.csv')

Functions that returns a list of filenames of the randomly selected files to compose the test set

Parameters:

path : str

string containing the path to the files to select from the test set from.

number_of_files_per_class: int

The number of files to select from each class.

classes: list

a list containing strings of the classes the data is divided in. The classes are contained in the filename as labels.

file_extension: str

the extension of the file to read. The default value is .csv

image_list: np.array

numpy array representing file names, image data and labels

iterator_mode: str

string representing if the data provided is in arrays

Returns:

test_set_serialnumbers : list

A list containig the strings of filenames randomly selected to be part of the test set.

hardy.handling.pre_processing.save_to_folder(input_path, project_name, run_name)

Function that creates a new path to the folder for a specific transformation. The transformation folder will be nested in a run folder named using the run date and the project name

Parameters:

input_path : str

String containing the path to the .csv files

project_name : str

String representing the project name. This will be used to name the folder containing the results from the hardy run

run_name : str

String representing the transformation applied to the data

Returns:

transformation_folder_path : str

String representing the path to the newly generated path

hardy.handling.pre_processing.test_set_folder(path, test_set_filenames)

Functions that removes the files randomly chosen to be part of the test set and saves them intothe test_set folder

Parameters:

path : str

string containing the path where to create a test set folder

test_set_filenames: list

The list containig the strings of filenames randomly selected to be part of the test set.

Returns:

test_set_folder : str

A string containging the path to the test set folder.

to_catalogue module

hardy.handling.to_catalogue.data_set_split(image_list, test_set_filenames)

Function that splits the list of image arrays into a test set and a learning setto use for the classification step

Parameters:

image_list: list

A list of tuples containing the filenames, the arrays reoresenitng the images and their labels

test_set_filenames: list

List of strings containig the filename of the datasets selected to the be in the test set

Returns:

test_set_list : list

A list of tuples containing the filenames, the arrays reoresenitng the images and their labels to be used as the test set

learning_set_list : list

A list of tuples containing the filenames, the arrays reoresenitng the images and their labels to be used as the learning set

hardy.handling.to_catalogue.learning_set(path=None, split=0.1, target_size=(80, 80), classes=['noisy', 'not_noisy'], batch_size=32, color_mode='rgb', iterator_mode='arrays', image_list=None, k_fold=None, k=None, fold=None, **kwargs)

A funciton that will create an iterator for the files representing the learning sets

Parameters:

path: str

A string containing the path to the files to use for the learning set

split: float

A number between 0 and 1 representing which percentage of the data will compose the validation set

target_size: tuple

A tuple containing the dimentions of the image to be inputted in the model

classes: list

A list containing strings of the classes the data is divided in. The class name represent the folder name the files are contained in.

batch_size: int

The number of files to group up into a batch

color_mode: str

Either grayscale or rgb

iterator_mode: str

string indicating which Keras IamgeDataGenerator mode to use. Options are ‘arrays’ or ‘images’. The first will use the “flow” option, the second will use “flow_from_directory” option

image_list: list

The list of tuples in the following format (filenames, image_array, label)

k_fold: Bool

input to select the k-fold validation for the classification step

k: int

number of total subsets to divide the data in for the k-fold validation

fold: int

the subset number to use for partitioning the data - used as input to avoid inner loop in this function

Returns:

training_set: Keras image iterator

The training set containg labelled images

validation_set: Keras image iterator

The training set containg labelled images

hardy.handling.to_catalogue.regular_plot_list(data_tuples, scale=1.0, storage_location='./')

Returns a list of tuples containing the arrays of images representing x-y plot

Parameters:

data_tuples: list of tuples

The list of tuples in the following format

(filenames, dataframe, label)

scale: float

percentage fo the image to reduce its size to.

Returns

——-

list_of_rgb_tuples: list of tuples

The list of tuples in the following format (filename, image array, label)

hardy.handling.to_catalogue.rgb_list(data_tuples, plot_format='RgBrGb', column_names=None, combine_method='add', scale=1.0, storage_location='./')

Input a path of csv files (with some guidance), plots them RGB-wise into images and returns a list of tuples as to be fed into the pre_processing functions

Parameters:

data_tuples: list of tuples

following the convention (SERIAL, DataFrame, LABEL)

plot_format: string

to pass into rgb_visualize “single”, “else”, or some “RGBrgb”

combine_method: string

string to use as input for rgb_visualize function

column names: list of strings (Optional)

IF given, will drop all columns not in the list given.

scale: float

percentage fo the image to reduce its size to.

Returns:

list_of_rgb_tuples: list

list of tuples following the format: (SERIAL, IMG, LABEL)

SERIAL: str

File name with the extension taken off

IMG: array

ndarray of NxNx3

LABEL: str

Classification label, either from the passed list or from the last part of the serial/filename: “123847_afsukjeh_*LABEL*.csv””

hardy.handling.to_catalogue.rgb_visualize(fdata, plot_format='RGBrgb', combine_method='add', column_names=None, scale=1.0)

Input a list of dataframes (already read and/or processed), Plot them RGB-wise into images return a list of tuples as to be fed into the keras PreProcess f(n)

Parameters:

plot_format: string

EITHER ‘single’ (bodge, depreciate later) OR some combination of “RGBrgb”, which will be the order of columns plotted:

R = red X-axis r = red Y-axis G = green X-axis g = green Y-axis B = blue X-axis b = blue Y-axis * X = do not plot (skip column) ** If RGBrgb letters are missing, simply pass to the plotting function as “None”

combine_method: string

Either “add” or “mlt” - which visualization function to use

scale: float

percentage fo the image to reduce its size to.

Returns:

list_of_tuples: list

list of tuples, following: (SERIAL, IMG, LABEL)

SERIAL: string

File name with the extension taken off

IMG: ndarray

Array of size NxNx3 representing the image of the data

hardy.handling.to_catalogue.save_load_data(filename, data=None, save=None, load=None, file_extension='.npy', location='./')

Function to save and load data

Function that can save or load data depending on given parameters.

Parameters:

filename : str

string indicating the filename for saving or loading dataset.

data : list

dataset that is to be saved or loaded.

save : bool

boolean value if true saves the compressed dataset.

load : bool

boolean value if true loads the compressed dataset.

file_extension : str

String containing the file extension to use

location : str

string containing the path to the folder to save the pickled file in

Returns:

loaded_data : list

dataset that is loaded from the specified location

hardy.handling.to_catalogue.test_set(path=None, target_size=(80, 80), classes=['noisy', 'not_noisy'], batch_size=32, color_mode='rgb', iterator_mode='arrays', image_list=None, training=True, **kwargs)

A funciton that will create an iterator for the files representing the test set

Parameters:

path: str

A string containing the path to the files to use for the test set

target_size: tuple

A tuple containing the dimentions of the image to be inputted in the model

classes: list

A list containing strings of the classes the data is divided in. The class name represent the folder name the files are contained in.

batch_size: int

The number of files to group up into a batch

color_mode: str

Either grayscale or rgb

iterator_mode : str

string indicating which Keras IamgeDataGenerator mode to use. Options are ‘arrays’ or ‘images’. The first will use the “flow” option, the second will use “flow_from_directory” option

image_list : list

The list of tuples in the following format (filenames, image_array, label)

Returns:

test_set : Keras image iterator

The testing set containg labelled images that was not part of the learning dataset

visualization module

hardy.handling.visualization.get_img_from_fig(fig, scale=1.0, dpi=100)

Transforms a matplotlib figure into an array

Parameters:

fig: matplotlib figure

The figure containing the x-y plot of the data

scale: float

percentage fo the image to reduce its size to.

Returns:

img: np.array

A numpy arrays representing the image. Iamge will be in rgb mode

hardy.handling.visualization.normalize(data_array)

Function that returns a normalized data_array

The function takes the maximum value of an array and divides each entry of the array by it. Additionally, if the minimum of the array is negative, it shifts it to zero, so that the resulting normalized array will have a range zero to one.

Parameters:

data_array : array-like

the array to be normalized.

Returns:

normalized_data_array : array-like

the normalized array. All entries in this array should be values in the range zero to one.

hardy.handling.visualization.normalize_image(color_image_array)

Function that normalizes a color image array. The color image array will ahve dimensions (n,n, 3). The ‘n’ value will depends on how big your image is

Parameters:

color_image_array: array-like

a multidimentional array of shape (n,n,3)

Returns:

normalized_image: array-like

the normalized image array. All entries in this array should be values in the range zero to one. The image shape should still be (n,n,3)

hardy.handling.visualization.orthogonal_images_add(image_x, image_y, plot=True, save_image=None, filename=None, save_location=None)

Takes two images and combines them by rotating one of them 90 degrees and adds the two up. The resulting array is then normalized by channel. Takes in two images of shape=(ARBITRARY, Data-axis, 3)

Parameters:

image_x : array-like

A multidimentional array of shape (n,n,3) with entries in range zero to one

image_y : array-like

A multidimentional array of shape (n,n,3) with entries in range zero to one

plot : bool

if True, the color gradient representation of the data will be displayed

filename : str

The filename will be the same as the .csv containing the data used to create this plot.

save_location : str

String containing the path of the forlder to use when saving the data and the image.

save_image : bool

Option to save the output of the simuation as a plot in a .png file format. The filename used for the file will be the same as the raw data file created in this function.

Returns:

combined_image: matplotlib plot

Plot representing the data as a color gradient on the x-axis and on the y-axis in one of the three basic colors: red, blue or green

hardy.handling.visualization.orthogonal_images_mlt(image_x, image_y, plot=True, save_image=None, filename=None, save_location=None)

Takes two images and combines them by rotating one of them 90 degrees and multiplies them. Takes in two images of shape=(ARBITRARY, Data-axis, 3)

NOTE: If one axis of a color (Red X) has data but the other (Red Y)

has nothing, we should Replace the Zero-array with a Ones-Array!

Parameters:

image_x: array-like

A multidimentional array of shape (n,n,3) with entries in range zero to one

image_y: array-like

A multidimentional array of shape (n,n,3) with entries in range zero to one

plot: bool

if True, the color gradient representation of the data will be displayed

filename: str

The filename will be the same as the .csv containing the data used to create this plot.

save_location: str

String containing the path of the forlder to use when saving the data and the image.

save_image: bool

Option to save the output of the simuation as a plot in a .png file format. The filename used for the file will be the same as the raw data file created in this function.

Returns

——-

combined_image: matplotlib plot

Plot representing the data as a color gradient on the x-axis and on the y-axis in one of the three basic colors: red, blue or green

hardy.handling.visualization.regular_plot(tform_df_tuple, scale=1.0)

Function that generates standard x-y plots

Parameters:

tform_df_tuple: list

The list of tuples in the following format

(filenames, dataframe, label)

scale: float

percentage fo the image to reduce its size to.

Returns:

img : np.array

A numpy arrays representing the image. Iamge will be in rgb mode

hardy.handling.visualization.rgb_plot(red_array=None, green_array=None, blue_array=None, plot=True, save_image=None, filename=None, save_location=None, scale=1.0)

Returns a plot which represents the input data as a color gradient of one of the three color channels available: red, blue or green.

This function represents the data as a color gradient in one of the three basic colors: red, blue or green. The color gradient is represented on the x-axis, leaving the y-axis as an arbitrary one. This means that the size or the scale of the y-axis do not have a numerical significance. The input arrays shoudld be of range zero to one. A minimum of one array should be provided. The final representation will be a square plot of the combined arrays.

Parameters:

red_array: array

the data array to be plotted in the red channel.

green_array: array

the data array to be plotted in the green channel.

blue_array: array

the data array to be plotted in the blue channel.

scale: float

percentage fo the image to reduce its size to.

plot: bool

if True, the color gradient representation of the data will be displayed

filename: str

The filename will be the same as the .csv containing the data used to create this plot.

save_location: str

String containing the path of the forlder to use when saving the data and the image.

save_image: bool

Option to save the output of the simuation as a plot in a .png file format. The filename used for the file will be the same as the raw data file created in this function.

Returns:

rbg_plot : matplotlib plot

Plot representing the data as a color gradient on the x-axis in one of the three basic colors: red, blue or green

Module contents