The `train_concise_ch11` function is the high-level-API counterpart of the from-scratch `train_ch11` harness. Instead of manually managing parameter tensors and calling a custom optimizer function, it constructs a single-layer linear model using `nn.Sequential(nn.Linear(5, 1))`, initializes its weights from a normal distribution with standard deviation $$0.01$$, and instantiates a framework-provided optimizer (e.g., `torch.optim.SGD`) by passing `net.parameters()` along with a hyperparameters dictionary. The loss is computed with `nn.MSELoss(reduction='none')`, which returns per-element squared errors without the $$\frac{1}{2}$$ factor. During each epoch the function iterates over minibatches, zeroes the gradients via `optimizer.zero_grad()`, performs a forward pass, reshapes the labels to match the output, computes and averages the loss, calls `l.mean().backward()` to obtain gradients, and advances the optimizer with `optimizer.step()`. Loss is periodically evaluated on the full dataset and divided by $$2$$ to align with the half-squared-error convention. This concise version produces identical convergence behavior to the from-scratch harness while requiring substantially less boilerplate code.

```python
def train_concise_ch11(trainer_fn, hyperparams, data_iter, num_epochs=4):
    net = nn.Sequential(nn.Linear(5, 1))
    def init_weights(module):
        if type(module) == nn.Linear:
            torch.nn.init.normal_(module.weight, std=0.01)
    net.apply(init_weights)
    optimizer = trainer_fn(net.parameters(), **hyperparams)
    loss = nn.MSELoss(reduction='none')
    animator = d2l.Animator(xlabel='epoch', ylabel='loss',
                            xlim=[0, num_epochs], ylim=[0.22, 0.35])
    n, timer = 0, d2l.Timer()
    for _ in range(num_epochs):
        for X, y in data_iter:
            optimizer.zero_grad()
            out = net(X)
            y = y.reshape(out.shape)
            l = loss(out, y)
            l.mean().backward()
            optimizer.step()
            n += X.shape[0]
            if n % 200 == 0:
                timer.stop()
                animator.add(n/X.shape[0]/len(data_iter),
                             (d2l.evaluate_loss(net, data_iter, loss) / 2,))
                timer.start()
    print(f'loss: {animator.Y[0][-1]:.3f}, '
          f'{timer.sum()/num_epochs:.3f} sec/epoch')
```

Concise Optimization Training Function

The train_ch11 function provides a generic, from-scratch training harness for benchmarking different optimization algorithms on a linear regression model. It accepts five arguments: a trainer function (trainer_fn), optimizer states, a hyperparameters dictionary, a data iterator, and the feature dimensionality. The function initializes weight parameters from a normal distribution with mean $$0$$ and standard deviation $$0.01$$, and a bias of $$0$$. During each epoch, it iterates over minibatches, computes the mean squared loss, performs backpropagation, and calls trainer_fn to update the parameters. It periodically evaluates the full-dataset loss and records the cumulative training time. This reusable design allows any optimizer with a compatible call signature to be plugged in by simply passing a different trainer_fn. A companion function, train_concise_ch11, provides an equivalent harness that delegates to framework-level optimizer APIs instead of custom update functions.

Claude

The main training loop for a deep learning model executes a systematic, iterative process to optimize parameters $$\mathbf{w}$$ and $$b$$. During each epoch, the loop passes through the entire training dataset. For every iteration within an epoch, a minibatch $$\mathcal{B}$$ is processed. The model computes the loss for this minibatch, averages it over the examples in the batch, and calculates the gradients of the averaged loss with respect to each parameter using $$\mathbf{g} \leftarrow \partial_{(\mathbf{w},b)} \frac{1}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} l(\mathbf{x}^{(i)}, y^{(i)}, \mathbf{w}, b)$$. Because the loss is averaged per minibatch, the gradient in the optimization algorithm does not need to be separately divided by the batch size. Finally, the optimization algorithm updates the parameters using the rule $$(\mathbf{w}, b) \leftarrow (\mathbf{w}, b) - \eta \mathbf{g}$$. This cycle repeats until the training is complete.

Deep Learning Minibatch Training Loop

Dive into Deep Learning

When training a model such as linear regression from scratch on a synthetic dataset, the true underlying parameters are perfectly known. This provides a unique opportunity to evaluate the success of the training loop by directly comparing the ground truth weights and bias with the parameters learned by the model. While exactly recovering the true parameters is generally difficult or impossible for deep networks because unique solutions often do not exist, stochastic gradient descent typically discovers parameter configurations that yield highly accurate predictions.

Linear Regression Synthetic Data Training Evaluation

The `fit_epoch` method, typically registered within an object-oriented `Trainer` class, encapsulates the computations required for a single full pass over the training dataset. It begins by setting the model to training mode and iterating through the training dataloader to compute the loss via the `training_step` method. Within this loop, it zeroes the gradients, performs backpropagation, and updates the parameters. If a validation dataloader is provided, the method subsequently sets the model to evaluation mode and processes the validation batches within a no-gradient context to assess the model's performance.

Object-Oriented Fit Epoch Implementation

To execute the training loop for a model from scratch, we instantiate the required components and begin the optimization process. For example, we initialize a linear regression model, `LinearRegressionScratch`, specifying hyperparameters such as the learning rate, `lr=0.03`. We then create the dataset using a class like `SyntheticRegressionData`, passing in ground truth parameter values. Next, an object-oriented `Trainer` is instantiated with settings such as `max_epochs=3`. Finally, training is executed by calling `trainer.fit(model, data)`, which orchestrates the iterations over the dataset to optimize the model's parameters.

Linear Regression Training Execution Example

Generic Optimization Training Function

To train the image classification model and select optimal hyperparameters based on validation set performance, a comprehensive training function is defined. This function orchestrates the training loop over a specified number of epochs using data parallelism across multiple devices. It initializes an optimization algorithm, such as stochastic gradient descent (SGD) with momentum and weight decay, and incorporates a step learning rate scheduler to periodically decay the learning rate by a specified factor. Within each epoch, the function iterates through training mini-batches to update parameters and accumulate training loss and accuracy. If a validation iterator is provided, it evaluates the model's accuracy on the hold-out validation set, plotting these metrics dynamically to monitor the model's generalization progress.

Learn Before

Related

Learn After