Similar to the loss function used in linear regression, the content loss in a CNN-based style transfer model measures the difference in content features between the synthesized image and the content image using a squared loss function. Both inputs to this squared loss function are the raw feature map outputs of the specified content layer, which are computed via a feature extraction function. Because the target content features extracted from the original content image are static, they do not require gradient computation and are treated as stated values during the optimization of the synthesized image.

```python
# PyTorch
def content_loss(Y_hat, Y):
    # We detach the target content from the tree used to dynamically compute
    # the gradient: this is a stated value, not a variable. Otherwise the loss
    # will throw an error.
    return torch.square(Y_hat - Y.detach()).mean()
```

Content Loss in CNN-Based Style Transfer

Similar to content loss, the style loss in a neural style transfer model uses the squared loss function to measure the difference in style between the synthesized image and the style image. However, instead of comparing raw feature maps directly, the two inputs of the squared loss function for style loss are the Gram matrices derived from the style layer outputs of the synthesized image and the style image. It is typically assumed that the Gram matrix based on the original style image has been precomputed, allowing it to act as a static target detached from gradient computation during training.

```python
# PyTorch
def style_loss(Y_hat, gram_Y):
    return torch.square(gram(Y_hat) - gram_Y.detach()).mean()
```

Style Loss in CNN-Based Style Transfer

In neural style transfer, the learned synthesized image often accumulates high-frequency noise, manifesting as particularly bright or dark pixels. To mitigate this, a noise reduction method known as *total variation denoising* is applied. Denoting $$x_{i, j}$$ as the pixel value at coordinate $$(i, j)$$, the total variation loss is defined as:

$$\sum_{i, j} \left|x_{i, j} - x_{i+1, j}\right| + \left|x_{i, j} - x_{i, j+1}\right|$$

Reducing this total variation loss encourages the values of neighboring pixels on the synthesized image to become closer, thereby promoting a smoother and visually cleaner final output.

```python
# PyTorch
def tv_loss(Y_hat):
    return 0.5 * (torch.abs(Y_hat[:, :, 1:, :] - Y_hat[:, :, :-1, :]).mean() +
                  torch.abs(Y_hat[:, :, :, 1:] - Y_hat[:, :, :, :-1]).mean())
```

Total Variation Loss in CNN-Based Style Transfer

The overall loss function for neural style transfer is calculated as a weighted sum of the content loss, style loss, and total variation loss. Specific hyperparameters—the content weight, style weight, and total variation weight—are applied to each respective loss component. Adjusting these weights allows for fine-tuning the balance between preserving the original content, applying the target style, and reducing high-frequency noise in the synthesized image.

```python
# PyTorch
content_weight, style_weight, tv_weight = 1, 1e4, 10

def compute_loss(X, contents_Y_hat, styles_Y_hat, contents_Y, styles_Y_gram):
    # Calculate the content, style, and total variance losses respectively
    contents_l = [content_loss(Y_hat, Y) * content_weight for Y_hat, Y in zip(
        contents_Y_hat, contents_Y)]
    styles_l = [style_loss(Y_hat, Y) * style_weight for Y_hat, Y in zip(
        styles_Y_hat, styles_Y_gram)]
    tv_l = tv_loss(X) * tv_weight
    # Add up all the losses
    l = sum(styles_l + contents_l + [tv_l])
    return contents_l, styles_l, tv_l, l
```

Total Loss Computation and Hyperparameters in Neural Style Transfer

The overall loss function in neural style transfer is defined as the weighted sum of three distinct components: the content loss, the style loss, and the total variation loss. By tuning the respective weight hyperparameters for each component, one can control the trade-off between retaining the original content, applying the desired artistic style, and reducing high-frequency noise in the synthesized image.

```python
content_weight, style_weight, tv_weight = 1, 1e4, 10

def compute_loss(X, contents_Y_hat, styles_Y_hat, contents_Y, styles_Y_gram):
    # Calculate the content, style, and total variance losses respectively
    contents_l = [content_loss(Y_hat, Y) * content_weight for Y_hat, Y in zip(
        contents_Y_hat, contents_Y)]
    styles_l = [style_loss(Y_hat, Y) * style_weight for Y_hat, Y in zip(
        styles_Y_hat, styles_Y_gram)]
    tv_l = tv_loss(X) * tv_weight
    # Add up all the losses
    l = sum(styles_l + contents_l + [tv_l])
    return contents_l, styles_l, tv_l, l
```

Claude

In CNN-based style transfer, a synthesized image is initially created, often by copying the content image, and serves as the only variable updated during training. A pretrained deep Convolutional Neural Network (CNN) is employed to extract hierarchical image features, while its own model parameters remain frozen. Specific layers within this deep CNN are selected such that some output the content features and others output the style features, which collectively guide the transformation of the synthesized image.

CNN-Based Style Transfer Method

Dive into Deep Learning

Loss Function in CNN-Based Style Transfer

In neural style transfer, the synthesized image acts as the sole optimizable variable during the training process. Instead of updating the weights of a neural network, the synthesized image itself is treated as the model parameter. We can define a simple model, such as a `SynthesizedImage` class, where forward propagation simply returns these optimizable image parameters.

Before training begins, this synthesized image model must be initialized. It is typically initialized as a copy of the content image to preserve the original structural elements from the start. Additionally, static features required for the loss computation, such as the Gram matrices of the style image at various layers, are precomputed prior to the training loop to reduce computational overhead.

```python
# PyTorch
import torch
from torch import nn

class SynthesizedImage(nn.Module):
    def __init__(self, img_shape, **kwargs):
        super(SynthesizedImage, self).__init__(**kwargs)
        self.weight = nn.Parameter(torch.rand(*img_shape))

    def forward(self):
        return self.weight

def get_inits(X, device, lr, styles_Y):
    gen_img = SynthesizedImage(X.shape).to(device)
    gen_img.weight.data.copy_(X.data)
    trainer = torch.optim.Adam(gen_img.parameters(), lr=lr)
    styles_Y_gram = [gram(Y) for Y in styles_Y]
    return gen_img(), styles_Y_gram, trainer
```

Synthesized Image as Model Parameter in Style Transfer

In a neural style transfer task, the content image provides the macroscopic structure and semantic objects for the synthesized output. For example, a photograph of a landscape (such as Mount Rainier) can be loaded and visualized as the content input. By examining its printed coordinate axes, one can observe its specific spatial dimensions, which frequently differ from the size of the accompanying style image.

Content Image Example in Neural Style Transfer

In a neural style transfer task, the style image provides the artistic textures and color palette for the synthesized output. For example, a painting of an autumn oak can be loaded and visualized as the style input. Its printed coordinate axes demonstrate its specific spatial dimensions, explicitly showing that the style image and the content image can have completely different sizes.

Learn Before

Related

Learn After