17 Deep learning model tuning

Configurations to run this chapter

R
Python

# load packages "torch" and "luz"
library(torch)
library(luz)
torch::install_torch()
# load packages "sits" and "sitsdata"
library(sits)
library(sitsdata)
# set tempdir if it does not exist 
tempdir_r <- "~/sitsbook/tempdir/R/cl_tuning"
dir.create(tempdir_r, showWarnings = FALSE)

# load "pysits" library
from pysits import *
from pathlib import Path
# set tempdir if it does not exist 
tempdir_py = Path.home() / "sitsbook/tempdir/Python/cl_tuning"
tempdir_py.mkdir(parents=True, exist_ok=True)

17.1 Introduction

Model tuning is the process of selecting the best set of hyperparameters for a specific application. When using deep learning models for image classification, it is a highly recommended step to enable a better fit of the algorithm to the training data. Hyperparameters are parameters of the model that are not learned during training but instead are set prior to training and affect the behavior of the model during training. Examples include the learning rate, batch size, number of epochs, number of hidden layers, number of neurons in each layer, activation functions, regularization parameters, and optimization algorithms.

Deep learning model tuning involves selecting the best combination of hyperparameters that results in the optimal performance of the model on a given task. This is done by training and evaluating the model with different sets of hyperparameters to select the set that gives the best performance.

Deep learning algorithms try to find the optimal point representing the best value of the prediction function that, given an input \(X\) of data points, predicts the result \(Y\). In our case, \(X\) is a multidimensional time series, and \(Y\) is a vector of probabilities for the possible output classes. For complex situations, the best prediction function is time-consuming to estimate. For this reason, deep learning methods rely on gradient descent methods to speed up predictions and converge faster than an exhaustive search [1]. All gradient descent methods use an optimization algorithm adjusted with hyperparameters such as the learning and regularization rates [2]. The learning rate controls the numerical step of the gradient descent function, and the regularization rate controls model overfitting. Adjusting these values to an optimal setting requires using model tuning methods.

17.2 How SITS performs model tuning

To reduce the learning curve, sits provides default values for all machine learning and deep learning methods, ensuring a reasonable baseline performance. However, refining model hyperparameters might be necessary. To that end, the package provides the sits_tuning() function.

The most straightforward approach to model tuning is to run a grid search; this involves defining a range for each hyperparameter and then testing all possible combinations. This approach leads to a combinatorial explosion and thus is not recommended. Instead, Bergstra and Bengio propose randomly chosen trials [3]. Their paper shows that randomized trials are more efficient than grid search trials, selecting adequate hyperparameters at a fraction of the computational cost. The sits_tuning() function follows Bergstra and Bengio by using a random search on the chosen hyperparameters.

The sits_tuning() function takes the following parameters:

Parameters for tuning function.
Parameter	Meaning
`samples`	Training dataset to be used by the model.
`samples_validation`	Optional dataset containing time series to be used for validation. If missing, the next parameter will be used.
`validation_split`	If `samples_validation` is not used, this parameter defines the proportion of time series in the training dataset to be used for validation (default is 20%).
`ml_method()`	Deep learning method (either `sits_mlp()`, `sits_tempcnn()`, `sits_lstm_fcn()`, `sits_resnet()` or `sits_lighttae()`).
`params`	Defines the optimizer and its hyperparameters by calling `sits_tuning_hparams()`, as shown in the example below.
`trials`	Number of trials to run the random search.
`multicores`	Number of cores to be used for the procedure.
`progress`	Show a progress bar?

The sits_tuning_hparams() function inside sits_tuning() allows defining optimizers and their hyperparameters, including lr (learning rate), eps (controls numerical stability), and weight_decay (controls overfitting). The default values for eps and weight_decay in all sits deep learning functions are 1e-08 and 1e-06, respectively. The default lr for sits_lighttae() and sits_tempcnn() is 0.005.

Users have different ways to randomize the hyperparameters, including:

choice() (a list of options);
uniform (a uniform distribution);
randint (random integers from a uniform distribution);
normal(mean, sd) (normal distribution);
beta(shape1, shape2) (beta distribution);
loguniform(max, min) (loguniform distribution).

We suggest using the log-uniform distribution to search over a wide range of values that span several orders of magnitude. This is common for hyperparameters like learning rates, which can vary from very small values (e.g., 0.0001) to larger values (e.g., 1.0) in a logarithmic manner. By default, sits_tuning() uses a loguniform distribution between 10^-2 and 10^-4 for the learning rate and the same distribution between 10^-2 and 10^-8 for the weight decay.

17.3 Tuning a LightTAE model

Our first example is tuning a Lightweight Temporal Attention Encoder model (sits_lighttae()) [4] on the MOD13Q1 dataset for the state of Mato Grosso. To recall, this data set contains time series samples from the Brazilian Mato Grosso state obtained from the MODIS MOD13Q1 product. It has 1,837 samples and seven classes (Cerrado, Forest, Pasture, Soy_Corn, Soy_Cotton, Soy_Fallow, Soy_Millet) [5] and is available in the R package sitsdata. In this example, we show how the accuracy is related to hyperparameter lr (learning rate).

We first retrieve the training data from Hugging Face, as follows.

R
Python

# Get the samples from Hugging Face
samples_matogrosso_mod13q1 <- sits_from_hf(
  repo = "e-sensing/samples_book",
  file = "samples_matogrosso_mod13q1.parquet",
  type = "dataset"
)

# Get data point to classify
point_mt_mod13q1 <- sits_from_hf(
  repo = "e-sensing/samples_book",
  file = "point_mt_mod13q1.parquet",
  type = "dataset"
)

# Get the samples from Hugging Face
samples_matogrosso_mod13q1 = sits_from_hf(
  repo = "e-sensing/samples_book",
  file = "samples_matogrosso_mod13q1.parquet",
  type = "dataset"
)

# Get data point to classify
point_mt_mod13q1 = sits_from_hf(
  repo = "e-sensing/samples_book",
  file = "point_mt_mod13q1.parquet",
  type = "dataset"
)

The next step is to apply sits_tuning to the LightTAE model with the training data.

R
Python

# Set seed 
set.seed(29031956)
# Tuning ``sits_lighttae`` model
tuned_mt <- sits_tuning(
     samples = samples_matogrosso_mod13q1,
     ml_method = sits_lighttae(),
     params = sits_tuning_hparams(
         optimizer = torch::optim_adamw,
         opt_hparams = list(
             lr = loguniform(10^-2, 10^-4)
             )
         ),
     trials = 30,
     multicores = 4,
     seed = 29031956,
     gpu_memory = 20,
     batch_size = 2^20,
     progress = FALSE
)
tuned_mt

# A tibble: 30 × 15
   accuracy kappa acc        samples_validation epochs batch_size
      <dbl> <dbl> <list>     <list>              <int>      <int>
 1    0.981 0.977 <sts_ccrc> <NULL>                150         64
 2    0.981 0.977 <sts_ccrc> <NULL>                150         64
 3    0.981 0.977 <sts_ccrc> <NULL>                150         64
 4    0.978 0.974 <sts_ccrc> <NULL>                150         64
 5    0.975 0.970 <sts_ccrc> <NULL>                150         64
 6    0.975 0.970 <sts_ccrc> <NULL>                150         64
 7    0.973 0.967 <sts_ccrc> <NULL>                150         64
 8    0.973 0.967 <sts_ccrc> <NULL>                150         64
 9    0.973 0.967 <sts_ccrc> <NULL>                150         64
10    0.973 0.967 <sts_ccrc> <NULL>                150         64
# ℹ 20 more rows
# ℹ 9 more variables: validation_split <dbl>, optimizer <list>,
#   opt_hparams <list>, lr_decay_epochs <int>, lr_decay_rate <dbl>,
#   patience <int>, min_delta <dbl>, seed <list>, verbose <lgl>

# Set seed 
r_set_seed(29031956)

# Tuning ``sits_lighttae`` model
tuned_mt = sits_tuning(
     samples = samples_matogrosso_mod13q1,
     ml_method = sits_lighttae,
     params = sits_tuning_hparams(
         optimizer = "torch::optim_adamw",
         opt_hparams = dict(
             lr = hparam("loguniform", 10**-2, 10**-4),
             weight_decay = hparam("loguniform", 10**-2, 10**-8)
             )
         ),
     trials = 20,
     multicores = 8,
     seed = 29031956,
     gpu_memory = 16,
     batch_size = 2^16,
     progress = False
)
tuned_mt

# A tibble: 30 × 15
   accuracy kappa acc        samples_validation epochs batch_size
      <dbl> <dbl> <list>     <list>              <int>      <int>
 1    0.981 0.977 <sts_ccrc> <NULL>                150         64
 2    0.981 0.977 <sts_ccrc> <NULL>                150         64
 3    0.981 0.977 <sts_ccrc> <NULL>                150         64
 4    0.978 0.974 <sts_ccrc> <NULL>                150         64
 5    0.975 0.970 <sts_ccrc> <NULL>                150         64
 6    0.975 0.970 <sts_ccrc> <NULL>                150         64
 7    0.973 0.967 <sts_ccrc> <NULL>                150         64
 8    0.973 0.967 <sts_ccrc> <NULL>                150         64
 9    0.973 0.967 <sts_ccrc> <NULL>                150         64
10    0.973 0.967 <sts_ccrc> <NULL>                150         64
# ℹ 20 more rows
# ℹ 9 more variables: validation_split <dbl>, optimizer <list>,
#   opt_hparams <list>, lr_decay_epochs <int>, lr_decay_rate <dbl>,
#   patience <int>, min_delta <dbl>, seed <list>, verbose <lgl>

The result is a tibble with different values of accuracy, kappa, decision matrix, and hyperparameters. The best results obtain accuracy values between 0.978 and 0.973, as shown below. The best result is obtained by a learning rate of 0.0012.

R
Python

# Obtain accuracy, kappa, lr, and weight decay for the 5 best results
# Hyperparameters are organized as a list
hparams_5 <- tuned_mt[1:5,]$opt_hparams

# Extract learning rate and weight decay from the list
lr_5 <- purrr::map_dbl(hparams_5, function(h) h$lr)

# Create a tibble to display the results
best_5 <- tibble::tibble(
    accuracy = tuned_mt[1:5,]$accuracy,
    kappa = tuned_mt[1:5,]$kappa,
    lr    = lr_5)

# Print the best five results
best_5

# A tibble: 5 × 3
  accuracy kappa       lr
     <dbl> <dbl>    <dbl>
1    0.981 0.977 0.000761
2    0.981 0.977 0.00929 
3    0.981 0.977 0.000253
4    0.978 0.974 0.00221 
5    0.975 0.970 0.00880

# Import pandas
import pandas as pd

# Obtain accuracy, kappa, lr, and weight decay for the 5 best results
# Hyperparameters are organized as a list
hparams_5 = tuned_mt.opt_hparams[0:5]

# Extract learning rate from the list
lr_5 = [x for x_ in hparams_5 for x in x_["lr"]]

# Create a tibble to display the results
best_5 = pd.DataFrame(dict(
    accuracy = tuned_mt.accuracy[0:5],
    kappa = tuned_mt.kappa[0:5],
    lr = lr_5
))

# Print the best five accuracies
best_5

# A tibble: 5 × 3
  accuracy kappa       lr
     <dbl> <dbl>    <dbl>
1    0.981 0.977 0.000761
2    0.981 0.977 0.00929 
3    0.981 0.977 0.000253
4    0.978 0.974 0.00221 
5    0.975 0.970 0.00880

17.4 Tuning a TempCNN model

In the example, we use sits_tuning() to find good hyperparameters to train the sits_tempcnn() algorithm for a dataset for measuring deforestation in Rondonia (samples_deforestation_rondonia) available in package sitsdata. This dataset consists of 6,007 samples collected from Sentinel-2 images covering the state of Rondonia. There are nine classes: Clear_Cut_Bare_Soil, Clear_Cut_Burned_Area, Mountainside_Forest, Forest, Riparian_Forest, Clear_Cut_Vegetation, Water, Wetland, and Seasonally_Flooded. Each time series contains values from Sentinel-2/2A bands B02, B03, B04, B05, B06, B07, B8A, B08, B11 and B12, from 2022-01-05 to 2022-12-23 in 16-day intervals. The samples are intended to detect deforestation events and have been collected by remote sensing experts using visual interpretation.

We obtain the training samples from Hugging Face.

R
Python

# Get the samples from Hugging Face
samples_deforestation_rondonia <- sits_from_hf(
  repo = "e-sensing/samples_book",
  file = "samples_deforestation_rondonia.parquet",
  type = "dataset"
)

# Load samples
samples_deforestation_rondonia = sits_from_hf(
  repo = "e-sensing/samples_book",
  file = "samples_deforestation_rondonia.parquet",
  type = "dataset"
)

The hyperparameters for the sits_tempcnn() method include the size of the layers, convolution kernels, dropout rates, learning rate, and weight decay. Please refer to the description of the Temporal CNN algorithm in Chapter Machine learning algorithms for image time series.

R
Python

# Set seed
set.seed(29031956)
# Tuning ``sits_tempcnn`` model
tuned_tempcnn <- sits_tuning(
  samples = samples_deforestation_rondonia,
  ml_method = sits_tempcnn(),
  params = sits_tuning_hparams(
    cnn_layers = choice(c(256, 256, 256), c(128, 128, 128), c(64, 64, 64)),
    cnn_kernels = choice(c(3, 3, 3), c(5, 5, 5), c(7, 7, 7)),
    cnn_dropout_rates = choice(c(0.2, 0.2, 0.2),
                               c(0.3, 0.3, 0.3), 
                               c(0.4, 0.4, 0.4)),
    optimizer = torch::optim_adamw,
    opt_hparams = list(
      lr = loguniform(10^-2, 10^-4),
      weight_decay = loguniform(10^-2, 10^-8)
    )
  ),
  trials = 30,
  multicores = 8,
  seed = 29031956,
  gpu_memory = 16,
  batch_size = 2^16
)

# Set seed
r_set_seed(29031956)

# Tuning ``sits_tempcnn`` model
tuned_tempcnn = sits_tuning(
  samples = samples_deforestation_rondonia,
  ml_method = sits_tempcnn,
  params = sits_tuning_hparams(
    cnn_layers = hparam(
        "choice", (256, 256, 256), (128, 128, 128), (64, 64, 64)
    ),
    cnn_kernels = hparam(
        "choice", (3, 3, 3), (5, 5, 5), (7, 7, 7)
    ),
    cnn_dropout_rates = hparam(
        "choice", (0.2, 0.2, 0.2), (0.3, 0.3, 0.3), (0.4, 0.4, 0.4)
    ),
    optimizer = "torch::optim_adamw",
    opt_hparams = dict(
      lr = hparam("loguniform", 10**-2, 10**-4),
      weight_decay = hparam("loguniform", 10**-2, 10**-8)
    )
  ),
  trials = 20,
  multicores = 8,
  seed = 29031956,
  gpu_memory = 18,
  batch_size = 2^18
)

The result of sits_tuning() is a tibble with different values of accuracy, kappa, decision matrix, and hyperparameters. We list below the five best results.

R
Python

# Obtain accuracy, kappa, cnn_layers, cnn_kernels, and cnn_dropout_rates the best result
cnn_params <- tuned_tempcnn[1:5, c("accuracy", "kappa", "cnn_layers", "cnn_kernels", "cnn_dropout_rates" )]

# Get best learning rates and weight decay
hparams_best <- tuned_tempcnn[1:5,]$opt_hparams
lr_best <- purrr::map_dfr(hparams_best, function(h) 
  tibble::tibble(
    lr = h$lr,
    weight_decay = h$weight_decay)
)

# Print the best parameters 
best_params <- dplyr::bind_cols(cnn_params, lr_best)
best_params

# A tibble: 5 × 7
  accuracy kappa cnn_layers   cnn_kernels cnn_dropout_rates      lr weight_decay
     <dbl> <dbl> <chr>        <chr>       <chr>               <dbl>        <dbl>
1    0.949 0.940 c(64, 64, 6… c(3, 3, 3)  c(0.4, 0.4, 0.4)  5.78e-3 0.0000525   
2    0.947 0.937 c(128, 128,… c(3, 3, 3)  c(0.3, 0.3, 0.3)  7.18e-4 0.000000113 
3    0.946 0.936 c(256, 256,… c(7, 7, 7)  c(0.2, 0.2, 0.2)  6.72e-4 0.0000677   
4    0.945 0.936 c(128, 128,… c(5, 5, 5)  c(0.4, 0.4, 0.4)  1.38e-3 0.00000211  
5    0.945 0.936 c(256, 256,… c(7, 7, 7)  c(0.2, 0.2, 0.2)  1.22e-3 0.0000000176

# Import pandas
import pandas as pd

# Learning rates and weight decay are organized as a list
hparams_best = tuned_tempcnn.opt_hparams[0:5]

# Extract learning rate and weight decay from the list
lr_5 = [x for x_ in hparams_best for x in x_["lr"]]
wd_5 = [x for x_ in hparams_best for x in x_["weight_decay"]]

# Obtain accuracy, kappa, cnn_layers, cnn_kernels, and cnn_dropout_rates the best result
pd.DataFrame(dict(
    accuracy = tuned_tempcnn.accuracy[0:5],
    kappa = tuned_tempcnn.kappa[0:5],
    cnn_layers = tuned_tempcnn.cnn_layers[0:5],
    cnn_kernels = tuned_tempcnn.cnn_kernels[0:5],
    cnn_dropout_rates = tuned_tempcnn.cnn_dropout_rates[0:5],
    lr = lr_5,
    wd_best = wd_5,
))

# A tibble: 5 × 7
  accuracy kappa cnn_layers   cnn_kernels cnn_dropout_rates      lr weight_decay
     <dbl> <dbl> <chr>        <chr>       <chr>               <dbl>        <dbl>
1    0.949 0.940 c(64, 64, 6… c(3, 3, 3)  c(0.4, 0.4, 0.4)  5.78e-3 0.0000525   
2    0.947 0.937 c(128, 128,… c(3, 3, 3)  c(0.3, 0.3, 0.3)  7.18e-4 0.000000113 
3    0.946 0.936 c(256, 256,… c(7, 7, 7)  c(0.2, 0.2, 0.2)  6.72e-4 0.0000677   
4    0.945 0.936 c(128, 128,… c(5, 5, 5)  c(0.4, 0.4, 0.4)  1.38e-3 0.00000211  
5    0.945 0.936 c(256, 256,… c(7, 7, 7)  c(0.2, 0.2, 0.2)  1.22e-3 0.0000000176

17.5 Summary

For large datasets, the tuning process is time-consuming. Despite this cost, it is recommended to achieve the best performance. In general, tuning hyperparameters for models such as sits_tempcnn() and sits_lighttae() will result in a slight performance improvement over the default parameters on overall accuracy. The performance gain will be stronger in the less well-represented classes, where significant gains in producer’s and user’s accuracies are possible. When detecting change in less frequent classes, tuning can make a relevant difference in the results.

References

[1]

Y. Bengio, “Practical recommendations for gradient-based training of deep architectures,” arXiv:1206.5533 [cs], 2012, [Online]. Available: http://arxiv.org/abs/1206.5533.

[2]

R. M. Schmidt, F. Schneider, and P. Hennig, “Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers,” in Proceedings of the 38th International Conference on Machine Learning, 2021, pp. 9367–9376, [Online]. Available: https://proceedings.mlr.press/v139/schmidt21a.html.

[3]

J. Bergstra and Y. Bengio, “Random Search for Hyper-Parameter Optimization,” Journal of Machine Learning Research, vol. 13, no. 10, pp. 281–305, 2012, [Online]. Available: http://jmlr.org/papers/v13/bergstra12a.html.

[4]

V. S. F. Garnot and L. Landrieu, “Lightweight Temporal Self-attention for Classifying Satellite Images Time Series,” in Advanced Analytics and Learning on Temporal Data, 2020, pp. 171–181.

[5]

M. Picoli et al., “Big earth observation time series analysis for monitoring Brazilian agriculture,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 145, pp. 328–339, 2018, doi: 10.1016/j.isprsjprs.2018.08.007.