Style Transfer using VGG19

Before we get into how this style transfer happens we first need to understand how CNNs work. In general, CNNs can look at images and learn to identify spatial patterns such as prominent colour and shapes.

The shapes and colours that define an image or any object in an image are often called features.

To preserve these features our CNN uses a convolutional layer. A convolutional layer applies different image filters also known as convolutional kernels to an input image.

Convolutional filters being applied to our input image

We take the image of a cat (content image)and take the image of The Great Wave off Kanagawa painting (style image). As a result, we get an image of a cat with the look of our painting.

Within the CNN, the maxpooling layers discard detailed spatial information, that is irrelevant to the task of classification. When we go deeper into a CNN the input image is transformed goes into the convolutional layers that care about the content of an image rather than detail about the texture and color of pixels.

We take the image of a cat (content image)and take the image of The Great Wave off Kanagawa painting (style image). As a result, we get an image of a cat with the look of our painting.

Within the CNN, the maxpooling layers discard detailed spatial information, that is irrelevant to the task of classification. When we go deeper into a CNN the input image is transformed goes into the convolutional layers that care about the content of an image rather than detail about the texture and color of pixels.

Later layers in the CNN as sometimes referred to as a content representation of an image. In this situation the CNN learned how to represent a content image, but what about style?

The style of an image can be thought of as traits that might be found in the brush strokes of a painting: the texture, colors, curvature, etc. To perform style transfers we need to combine the content of one image with the style of another.

To represent the style of an input image, the convolutional layers are designed to capture texture, and colour information is used. These layers look at spatial correlations within a layer of a network. A correlation is a measure of the relationship between two or more variables.

For example, you could look at the first convolutional layer which has some depth (64), the depth corresponds to the number of feature maps in that layer.

For each feature map, we can measure how strongly its detected features relate to the other feature maps in that layer.

Let’s say, we detect that mini-feature maps in the first convolutional layer have similar pink edge features. if there are common colours and shapes among the feature maps, then this can be thought of as part of that image’s style.

So the similarities and differences between features in a layer should give us some information about the texture and color information found in an image. At the same time, it should leave out anything about the identity or placement of objects in an image.

So when we do our style transfer we take 2 images: a content image and a style image. From the content image, we will take the object and shape arrangement and take the colors and textures from the style image.

For this project, I’m recreating a style transfer method outlined in this paper which uses a CNN called VGG19 and creates the style transfer effect using the pre-trained network features.

This network (VGG19) takes a colored image as input and passes it through a series of convolution and pooling layers. Followed by 3 fully connected layers that classify the input image.

In between the 5 pooling layers (the orange squares), there are stacks of 2 or 4 convolution layers. In this image above we have the name of the stack and its order in the stack. For example, conv2_1 is in the second stack and we are looking at layer number 1.

In order to create the target image (output of style transfer). We will first pass our content image through our VGG19 network. The content image will go through the feed-forward process until it reaches a deep convolutional layer in the network. The output of this layer will be the content representation of the input image.

Next, when it sees the style image, it will extract different features from multiple layers that represent the style of that image. Finally, it will use both the content and style representations to inform the creation of the target image.

When we get the content representation from conv4_2 as outlined in the paper we will have a copy of our content image and we need to manipulate it with our style so that its content is close to that of our content image and its style is close to that of our style image.

As we form our new target image, we’ll compare its content representation with that of our content image. These 2 representations should be close to the same even as our target image changes its style.

For this, we’ll define a content loss, a loss that calculates the difference between the content and target image representations which for the sake of this example will be called Cc (content representation of the content image)and Tc (content representation of target image).

In this case, we calculate the mean squared difference between the two representations. This is our content loss and it measures how far away these two representations are from one another.

As we try to create the best target image we will aim to minimize this loss. Our goal is not to minimize the classification error, rather our goal is to change only the target image updating its appearance until its content representation matches that of our content image.

In this situation, we are using the VGG19 as a feature extractor and using backpropagation to minimize a defined loss function between our target and content images.

Next, we do the same thing but for our style representations of our target image and style image. The style representation of an image relies on looking at correlations between the features in individual layers of the VVG19 network.

Similarities will include the general colors and textures found in that layer. We typically find the similarities between features in multiple layers in the network.

By including the correlations between multiple layers of different sizes, we can obtain a multiscale style representation of the input image, one that captures large and small style features. The style representation is calculated as an image passes through the network in the first convolutional layer in all 5 stacks. Conv1_1, Conv2_1, Conv3_1, Conv4_1, and Conv5_1.

The correlations in each layer are given by a gram matrix. The matrix is a result of a couple of operations.

Say we start with a 4x4 image and we convolve it with 8 different image filters to create a convolutional layer. This convolutional layer will be 4x4 in height and width (to represent the 4x4 image) and 8 in-depth (to represent the number of filters that the image will go through).

4x4 image turning it into a 3D convolutional layer

Thinking about the style representation for this layer, we can say that this layer has 8 feature maps that we want to find the relationships between.

The first step in calculating the Gram matrix will be to vectorize the values in this layer. The first values in the feature map will become the first four values in a vector with length 16.

By flattening the x and y dimensions of the feature maps, we are converting a 3D convolutional layer into a 2D matrix of values. The next step is to multiply this matrix by its transpose.

A transpose is a matrix that has the x and y values switching spots. For example our 2D matrix, 16x8 has a transpose of 8x16. The reason we do this is that when we do matrix multiplication the y value of the first matrix and the x of the second matrix needs to match up otherwise we can’t multiply the 2 matrices.

To reason we need to multiply by the transpose is that it helps us get our Gram matrix. This operation treats each value in the feature map as an individual sample unrelated in space to other values. So the resultant Gram matrix contains non-localized information about the layer.

Getting the gram matrix as a result of the matrix multiplication

For example, even if the content of a filtered image is not identifiable, our network will still be able to see prominent colors and textures, the style.
Finally, we are left with the square 8x8 Gram matrix whose values indicate the similarities between the layers.

If we look at row 4, column 2 will hold a value that indicates the similarity between the fourth and second feature maps in a layer (highlighted in yellow in the image above).

Importantly the dimensions of this matrix are related only to the number of feature maps in the convolution layers. It doesn’t depend on the dimensions of the input image.

To calculate the style loss between a target and style image we find the mean squared distance between the style and target image gram matrices. All five pairs that computed at each layer in our predefined list (conv1_1 up to conv5_1). These lists will be called Ss (style representation of style image) and Ts(style representation of target image. With acting as a constant that accounts for the number of values in each layer.

We will multiply these calculated distances by some style weights w that we specify and then add them up. The style weights are values that will give more or less weight to the calculated style at each of the five layers, thereby changing how much effect each layers’ style representation will have on our final target image.

Again we will only be changing that target image’s style representations as we minimize this loss over some number of iterations. Now we have the content loss, which tells us how close the content of our target image is to that of our content image, and our style loss which tells us how close our target is in style to our style image.

We can now add both of these losses together to get the total loss. Then we use the typical backpropagation and optimization to reduce this loss by iteratively changing the target image to match our desired content and style images.

Since both of these images are calculated differently, these values will be different and we want our target image to take both into account fairly and equally. So, it’s necessary to apply constant weights, alpha and beta to the content and style losses such that the total loss reflects an equal balance.

In practice, this means multiplying the style loss by a much larger weight value than the content loss. This is often expressed as the ratio of content and style weights alpha over beta.

In the paper, we see the effects of a bigger or smaller ratio. here is an example of a content and style image. We can imagine the content weight, alpha is one and the style weight beta is 10. You can see that this target image is mostly content without much style.

As beta increases to 100, then 1,000, and alpha stays at 1. We can see more and more styles in the generated image. Finally, we can see that this image can go too far, at a ratio of 10 to the negative 4 we see that most of the content is gone and the only style remains. In general the smaller the alpha-beta ratio the more stylistic effect you will see.

This makes intuitive sense because a smaller ratio corresponds to a larger value for beta, the style weight. You may find that certain ratios work well for one image, but not another, and depending on the image that you are making you can make the changes the create the exact kind of stylized effect that you want.