| AIfES 2 2.0.0

Functions
aialgo_sequential_inference.h File Reference

Functions required for inference of models. More...

Go to the source code of this file.

## Functions
uint32_t | aialgo_sizeof_inference_memory (aimodel_t *model)
| Calculate the memory requirements for intermediate results of an inference. More...
uint32_t | aialgo_sizeof_parameter_memory (aimodel_t *model)
| Calculate the memory requirements for the trainable parameters (like weights, bias, ...) of the model. More...
uint8_t | aialgo_schedule_inference_memory (aimodel_t *model, void *memory_ptr, uint32_t memory_size)
| Assign the memory for intermediate results of an inference to the model. More...
void | aialgo_distribute_parameter_memory (aimodel_t *model, void *memory_ptr, uint32_t memory_size)
| Assign the memory for the trainable parameters (like weights, bias, ...) of the model. More...
aitensor_t * | aialgo_forward_model (aimodel_t *model, aitensor_t *input_data)
| Perform a forward pass on the model. More...
uint8_t | aialgo_inference_model (aimodel_t *model, aitensor_t *input_data, aitensor_t *output_data)
| Perform an inference on the model / Run the model. More...
uint8_t | aialgo_compile_model (aimodel_t *model)
| Initialize the model structure. More...
void | aialgo_quantize_model_f32_to_q7 (aimodel_t *model_f32, aimodel_t *model_q7, aitensor_t *representative_dataset)
| Quantize model parameters (weights and bias) More...
void | aialgo_set_model_result_precision_q31 (aimodel_t *model, uint16_t shift)
| Initialize the quantization parameters of the layer results for Q31 data type. More...
void | aialgo_set_model_delta_precision_q31 (aimodel_t *model, uint16_t shift)
| Initialize the quantization parameters of the layer deltas for Q31 data type. More...
void | aialgo_set_model_gradient_precision_q31 (aimodel_t *model, uint16_t shift)
| Initialize the quantization parameters of the gradients for Q31 data type. More...
void | aialgo_print_model_structure (aimodel_t *model)
| Print the layer structure of the model with the configured parameters. More...
void | aialgo_set_layer_settings_model (aimodel_t *model, uint32_t bitmask, uint8_t shift, uint32_t value)
| Apply the specified setting to all layers in the model. More...
void | aialgo_set_training_mode_model (aimodel_t *model, uint8_t value)
| Enables / disables the training mode of the model. More...
void | aialgo_set_batch_mode_model (aimodel_t *model, uint8_t value)
| Enables / disables the batch mode of the model. More...
void | aialgo_set_trainable_model (aimodel_t *model, uint8_t value)
| Freeze / Unfreeze trainable parameters of the model. More...

## Detailed Description

Functions required for inference of models.

Version
  2.2.0

Copyright
  Copyright (C) 2020-2023 Fraunhofer Institute for Microelectronic Circuits and Systems. All rights reserved.
  AIfES is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
  This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
  You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.

The functions target memory allocation/scheduling, the calculation of the forward pass and quantization for model inference

## Function Documentation

## ◆ aialgo_compile_model()

uint8_t aialgo_compile_model | ( | aimodel_t * | model | ) |

Initialize the model structure.

Counts the number of layers and trainable parameters in a model as preparation for inference or training.

Parameters

  *model | The model

Returns
  0 if successful

## ◆ aialgo_distribute_parameter_memory()

void aialgo_distribute_parameter_memory | ( | aimodel_t * | model,
|  | void * | memory_ptr,
|  | uint32_t | memory_size
| ) |  |

Assign the memory for the trainable parameters (like weights, bias, ...) of the model.

Only use this function if the parameters are not pre-trained or manually configured. Afterwards an initialization of the memory (for example by initializing the weights) has to be performed.

The required memory size can be calculated with aialgo_sizeof_parameter_memory()

Parameters

  *model | The model
  *memory_ptr | Pointer to the memory block
  memory_size | Size of the memory block (for error checking)

Returns
  0 if successful

## ◆ aialgo_forward_model()

aitensor_t* aialgo_forward_model | ( | aimodel_t * | model,
|  | aitensor_t * | input_data
| ) |  |

Perform a forward pass on the model.

The result is stored in the result tensor of the output layer and a pointer to this is returned. This output result is stored in the inference memory and is only valid as long as the inference memory is valid. To get the output as a separate tensor, use aialgo_inference_model() instead.

Parameters

  *model | The model
  *input_data | Input data tensor of the same shape as the input_layer shape

Returns
  Pointer to the output data of the forward pass (points to the result tensor of the output layer)

## ◆ aialgo_inference_model()

uint8_t aialgo_inference_model | ( | aimodel_t * | model,
|  | aitensor_t * | input_data,
|  | aitensor_t * | output_data
| ) |  |

Perform an inference on the model / Run the model.

Make shure to initialize the model (aialgo_compile_model()) and schedule the inference memory (for example with aialgo_schedule_inference_memory() or aialgo_schedule_training_memory()) before calling this function.

Example:

float input_data[] = {0.0f, 1.0f, 0.0f};
uint16_t input_shape[] = {1, 3}
aitensor_t input_tensor = AITENSOR_2D_F32(input_shape, input_data);
float output_data[2];
uint16_t output_shape[] = {1, 2}
aitensor_t output_tensor = AITENSOR_2D_F32(output_shape, output_data);
aialgo_inference_model(&model, &input_tensor, &output_tensor);
// The results are now in the output_tensor
aialgo_inference_model
uint8_t aialgo_inference_model(aimodel_t *model, aitensor_t *input_data, aitensor_t *output_data)
Perform an inference on the model / Run the model.
aitensor
A tensor in AIfES.
Definition: aifes_math.h:89

Parameters

  *model | The model
  *input_data | Input data tensor of the same shape as the input_layer shape
  *output_data | Empty tensor for the results of the inference with the size of your outputs

Returns
  0 if successful

## ◆ aialgo_print_model_structure()

void aialgo_print_model_structure | ( | aimodel_t * | model | ) |

Print the layer structure of the model with the configured parameters.

Parameters

  *model | The model

## ◆ aialgo_quantize_model_f32_to_q7()

void aialgo_quantize_model_f32_to_q7 | ( | aimodel_t * | model_f32,
|  | aimodel_t * | model_q7,
|  | aitensor_t * | representative_dataset
| ) |  |

Quantize model parameters (weights and bias)

Parameters

  *model_f32 | Pointer to model with single-precision floating point parameters that should be quantized
  *model_q7 | Pointer to model with quantized, fixed-point parameters in q7 format
  *representative_dataset | Pointer to a dataset that represents real model inputs to determine fixed-point quantization parameters

## ◆ aialgo_schedule_inference_memory()

uint8_t aialgo_schedule_inference_memory | ( | aimodel_t * | model,
|  | void * | memory_ptr,
|  | uint32_t | memory_size
| ) |  |

Assign the memory for intermediate results of an inference to the model.

The required memory size can be calculated with aialgo_sizeof_inference_memory()

Parameters

  *model | The model
  *memory_ptr | Pointer to the memory block
  memory_size | Size of the memory block (for error checking)

Returns
  0 if successful

## ◆ aialgo_set_batch_mode_model()

void aialgo_set_batch_mode_model | ( | aimodel_t * | model,
|  | uint8_t | value
| ) |  |

Enables / disables the batch mode of the model.

When executed in batch mode, a whole batch is processed by one forward pass.
Some layers may behave differently during training than during inference (e.g. the Batch Normalization layer).
The aialgo_train_model() function calls this function internally (Batch mode gets enabled if the batch_size equals the first element of the ailayer_input.input_shape and gets disabled otherwise).

The setting value can be accessed with:

if(AILAYER_SETTINGS_IS(layer->settings, 0b1, AILAYER_SETTINGS_BATCH_MODE)){
...
}

Parameters

  *model | The model
  value | Target value (TRUE or FALSE)

## ◆ aialgo_set_layer_settings_model()

void aialgo_set_layer_settings_model | ( | aimodel_t * | model,
|  | uint32_t | bitmask,
|  | uint8_t | shift,
|  | uint32_t | value
| ) |  |

Apply the specified setting to all layers in the model.

Parameters

  *model | The model
  bitmask | 32-bit mask to address the setting value (usually set to 0b1)
  shift | The amount of bits to shift the bitmask to address the desired setting value (e.g. AILAYER_SETTINGS_TRAINING_MODE)
  value | The target value

## ◆ aialgo_set_model_delta_precision_q31()

void aialgo_set_model_delta_precision_q31 | ( | aimodel_t * | model,
|  | uint16_t | shift
| ) |  |

Initialize the quantization parameters of the layer deltas for Q31 data type.

Initializes the quantization parameters of the layer deltas tensor (ailayer.deltas; output of the backward function) to the given shift and zero_point = 0.
Use this function when you train a model in Q31 data-type.

Parameters

  *model | The model
  shift | Number of decimal places (shift in Q31) of the layer deltas

## ◆ aialgo_set_model_gradient_precision_q31()

void aialgo_set_model_gradient_precision_q31 | ( | aimodel_t * | model,
|  | uint16_t | shift
| ) |  |

Initialize the quantization parameters of the gradients for Q31 data type.

Initializes the quantization parameters of the gradients tensors to the given shift and zero_point = 0.
Use this function when you train a model in Q31 data-type.

Parameters

  *model | The model
  shift | Number of decimal places (shift in Q31) of the gradients

## ◆ aialgo_set_model_result_precision_q31()

void aialgo_set_model_result_precision_q31 | ( | aimodel_t * | model,
|  | uint16_t | shift
| ) |  |

Initialize the quantization parameters of the layer results for Q31 data type.

Initializes the quantization parameters of the layer output (ailayer.result; output of the forward function) to the given shift and zero_point = 0.
Use this function for example when you train a model in Q31 data-type.

Parameters

  *model | The model
  shift | Number of decimal places (shift in Q31) of the layer results

## ◆ aialgo_set_trainable_model()

void aialgo_set_trainable_model | ( | aimodel_t * | model,
|  | uint8_t | value
| ) |  |

Freeze / Unfreeze trainable parameters of the model.

When a layer is set to trainable = FALSE, the corresponding parameters are not modified by aialgo_train_model().

The setting value can be accessed with:

if(AILAYER_SETTINGS_IS(layer->settings, 0b1, AILAYER_SETTINGS_TRAINABLE)){
...
}

Parameters

  *model | The model
  value | Target value (TRUE or FALSE)

## ◆ aialgo_set_training_mode_model()

void aialgo_set_training_mode_model | ( | aimodel_t * | model,
|  | uint8_t | value
| ) |  |

Enables / disables the training mode of the model.

Some layers may behave differently during training than during inference (e.g. the Batch Normalization layer).
The aialgo_train_model() function calls this function internally.

The setting value can be accessed with:

if(AILAYER_SETTINGS_IS(layer->settings, 0b1, AILAYER_SETTINGS_TRAINING_MODE)){
...
}

Parameters

  *model | The model
  value | Target value (TRUE or FALSE)

## ◆ aialgo_sizeof_inference_memory()

uint32_t aialgo_sizeof_inference_memory | ( | aimodel_t * | model | ) |

Calculate the memory requirements for intermediate results of an inference.

This memory is mainly for the result buffers of the layers.

Use aialgo_schedule_inference_memory() to set the memory to the model.

Parameters

  *model | The model

Returns
  Required memory size in bytes

## ◆ aialgo_sizeof_parameter_memory()

uint32_t aialgo_sizeof_parameter_memory | ( | aimodel_t * | model | ) |

Calculate the memory requirements for the trainable parameters (like weights, bias, ...) of the model.

Use aialgo_distribute_parameter_memory() to set the memory to the model.

Parameters

  *model | The model

Returns
  Required memory size in bytes