| AIfES 2 2.0.0

Data Structures | Typedefs | Functions | Variables
ailoss_crossentropy.h File Reference

Base loss implementation of the Cross-Entropy loss. More...

Go to the source code of this file.

## Data Structures
struct | ailoss_crossentropy
| General Cross-Entropy loss struct. More...

## Typedefs
typedef struct ailoss_crossentropy | ailoss_crossentropy_t
| New data type name for code reduction.

## Functions
ailoss_t * | ailoss_crossentropy (ailoss_crossentropy_t *loss, ailayer_t *input_layer)
| Initialize and connect the given Cross-Entropy loss. More...
void | ailoss_crossentropy_calc_delta (ailoss_t *self, const aitensor_t *target_data)
| Calculate the combined derivative of the given Cross-Entropy loss and the output layer for error backpropagation. More...
void | ailoss_crossentropy_calc_loss (ailoss_t *self, const aitensor_t *target_data, void *result)
| Calculate the Cross-Entropy loss on the given target data. More...
void | ailoss_crossentropy_dummy_backward (ailayer_t *self)
| Dummy backward-function for the output layer of the model. More...
void | ailoss_crossentropy_print_specs (const ailoss_t *self)
| Print the loss specification. More...

## Variables
const aicore_losstype_t * | ailoss_crossentropy_type
| Cross-Entropy loss type. More...

## Detailed Description

Base loss implementation of the Cross-Entropy loss.

Version
  2.2.0

Copyright
  Copyright (C) 2020-2023 Fraunhofer Institute for Microelectronic Circuits and Systems. All rights reserved.
  AIfES is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
  This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
  You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.

This is an "abstract" data-type independent implementation. To use the loss, use one of the provided implementations for a specific hardware and data-type (for example from ailoss_crossentropy_default.h) or set the required math functions on your own.

The Cross-Entropy loss ist best suitable for classification tasks and works only with Sigmoid or Softmax output layers.

For binary classification (binary targets 0 or 1) with Sigmoid output layer, the loss / cost is calculated as

\[ L(y, \hat{y}) = - \sum_{i=0}^{N} (y_i \log(\hat{y}_i) + (1-y) \log(1-\hat{y}_i)) \]

with the predicted values \( \hat{y}_i \) and the target values \( y_i \). \( N \) is the number of elements of the \( y \) tensor.

For categorigal classification (one-hot encoded targets) with Softmax output layer, the loss / cost is calculated as

\[ L(y, \hat{y}) = - \sum_{i=0}^{N} y_{i} \log(\hat{y}_{i}) \]

with the predicted values \( \hat{y}_i \) and the target values \( y_i \). \( N \) is the number of elements of the \( y \) tensor.

To get the "mean" normalization, you have to modify the learning rate to \( lr = \frac {1}{o \cdot n} \cdot lr \) with the number of outputs \( o \) and the batch size \( n \).

The loss can be calculated with ailoss_crossentropy_calc_loss(). For training the deltas /errors on the target data are calculated with ailoss_crossentropy_calc_delta() and written to the deltas tensor of the connection layer.

## Function Documentation

## ◆ ailoss_crossentropy()

ailoss_t* ailoss_crossentropy | ( | ailoss_crossentropy_t * | loss,
|  | ailayer_t * | input_layer
| ) |  |

Initialize and connect the given Cross-Entropy loss.

This function represents the "constructor" of the abstract Cross-Entropy loss. It initializes the loss structure and connects it to the output layer of the AIfES model.
This function is not intended to call it directly. Instead use one of the data type specific implementations (like for example ailoss_crossentropy_f32_default()).

Parameters

  *loss | The loss to initialize.
  *input_layer | The output layer of the model that provides the inputs to the loss.

Returns
  Pointer to the (successfully) initialized loss structure.

## ◆ ailoss_crossentropy_calc_delta()

void ailoss_crossentropy_calc_delta | ( | ailoss_t * | self,
|  | const aitensor_t * | target_data
| ) |  |

Calculate the combined derivative of the given Cross-Entropy loss and the output layer for error backpropagation.

Implementation of ailoss.calc_delta.

It uses the result tensor of the output layer and the target data as input and writes the result to the deltas tensor (ailayer.deltas) of the output layer of the model.

By combining the Cross-Entropy loss with a Sigmoid or Softmax output layer, the combined deltas can be calculated very efficiently. The backward function of the Sigmoid / Softmax layer is not used anymore.

Calculation of the deltas:

\[ \delta_{in} \leftarrow p - y \]

\( \delta_{in} \): Result of the delta calculation of this loss (written to ailayer.deltas of the output layer of the model)
\( p \): Result of the forward pass of the output layer of the model (predicted values)
\( y \): Target data / True values / Labels
Used math functions:

- ailoss_crossentropy.tensor_sub

Parameters

  *self | Loss to calculate the deltas for
  *target_data | Target data / True values / Labels

## ◆ ailoss_crossentropy_calc_loss()

void ailoss_crossentropy_calc_loss | ( | ailoss_t * | self,
|  | const aitensor_t * | target_data,
|  | void * | result
| ) |  |

Calculate the Cross-Entropy loss on the given target data.

Implementation of ailoss.calc_loss.

It uses the result tensor of the output layer and the target data as input and writes the result to the given result scalar.

Calculation of the loss with Sigmoid output layer:

\[ result \leftarrow - \sum_i (y_i \log(p_i) + (1 - y_i) \log(1 - p_i)) \]

Calculation of the loss with Softmax output layer:

\[ result \leftarrow - \sum_i y_i \log(p_i) \]

\( result \): Result of the loss calculation
\( p \): Result of the forward pass of the output layer of the model (predicted values)
\( y \): Target data / True values / Labels
Used math functions:

- ailoss_crossentropy.tensor_sub
- ailoss_crossentropy.crossentropy

Parameters

  *self | Loss to calculate the deltas for
  *target_data | Target data / True values / Labels
  *result | Result scalar (the data type is specified by the data type specific implementations)

## ◆ ailoss_crossentropy_dummy_backward()

void ailoss_crossentropy_dummy_backward | ( | ailayer_t * | self | ) |

Dummy backward-function for the output layer of the model.

Implementation of ailayer.backward.

The ailoss_crossentropy_calc_delta() function calculates the combined delta for a Sigmoid or Softmax layer with Cross-Entropy loss. Therefor the backward function of the Sigmoid / Softmax layer is not needed anymore and gets exchanged by this dummy function.

The dummy function does nothing.

Parameters

  *self | The output layer of the model

## ◆ ailoss_crossentropy_print_specs()

void ailoss_crossentropy_print_specs | ( | const ailoss_t * | self | ) |

Print the loss specification.

Parameters

  *self | The loss to print the specification for

## Variable Documentation

## ◆ ailoss_crossentropy_type

const aicore_losstype_t* ailoss_crossentropy_type
| extern

Cross-Entropy loss type.

Defines the type of the loss (for example for type checks and debug prints). See aicore_losstype for more information about the loss type.