Page History

The GAN concept solves the problem of training a generative network (

In the original setup introduced by Goodfellow et al. ⁽¹⁾, fully connected layers were used for both

The setup can be interpreted as a zero-sum game within game theory, which follows a min-max character. On the one hand,

The outcome of the game is fed back to both networks using back-propagation. This allows both networks to iteratively improve their policy on how to decide and act within their respective tasks.

In terms of game theory, training a GAN requires finding Nash equilibria in high-dimensional, continuous, non-convex games. This is an incredibly hard task with no clear solution strategy to it.

Mathematically, the dataset of real images is a number of examples drawn from a true data distribution, which is shown in blue in figure 3.

Figure 4 gives us a good impression how

Figure 1: GAN concept:

Figure 2: Two scenarios: On the left, training examples

Figure 3: Representation of distribution space.  Source: (8)

Figure 4: Evolution of

A big breakthrough in the research about GANs has been achieved through the Deep Convolutional GAN (DCGAN) by Radford et al. ⁽²⁾ who applied a list of empirically validated tricks as the substitution of pooling and fully connected layers with convolutional layers. For

By changing the code incrementally, we observe meaningful changes in the generated images, while remaining real-looking. This is a clear sign that the network has learned actual representation or features of how the world looks like and it has encoded this information in the network.

The procedure of incrementally changing parts of the input code is described as walking in the latent space. As an example, we could perform a stepwise interpolation between two codes and see how the result changes. One experiment in the DCGAN publication ⁽²⁾ trained a network on images of bedrooms. An interpolation between the code of two different bedrooms showed how the rooms are slowly transforming while constantly keeping the look of a bedroom. In figure 5 we can observe lamps turning into windows (first row) or a TV transforming into a large window (last row).

Another evidence that the network has learned representations can be found in visualizing the filters of the network in figure 6. We can clearly see that the untrained, random filters look almost like a single sample picture. However, the trained filters have clearly discovered features that are significant for a bedroom.

Another remarkable observation is that the input codes support vector arithmetic. This means if we take, e.g., the averaged code of three smiling women, subtract neutral women, add neutral men and then feed the resulting code through the network, we get a smiling man (as shown in figure 7).

Figure 4: DCGAN generator: A series of fractionally-strided convolutions convert the input vector (code) into an image (

The power of the features encoded in the latent variables was further explored by Chen at al. ⁽⁴⁾. They made use of the fact that the latent space of a regular GAN is underspecified to add additional input parameters (referred to as extend code) and thereby functionality. They decomposed the code in the latent code seen before and an additional latent component, which targets the semantic features of the data distribution. The goal is to learn disentangled and interpretable representations.

This is achieved by maximizing the mutual information between small subsets of the representation variables and observations. Therefore, Chen at al. introduced a regularization term that gives weight to an entropy based indicator of how much information is shared between the latent code and the generated distribution.

The publication showed remarkable results. In the MNIST ⁽⁹¹⁰⁾. dataset the digit type, rotation, and width of the digits was captured in separate designated parts of the additional latent component. On a 3D faces data set the azimuth, elevation, lighting, and the width of the face were captured and identified.

GANs have also been used in the field of semi-supervised learning, where a small amount of labeled data is combined with a large amount of unlabeled data. In this setting, 

This allows the classifier to be additionally trained with unlabeled data which is known to be real, and with samples from the generator, which are known to be fake.

Through this approach, named feature matching GANs, good performance on the MNIST data set can be achieved with as little as 20 labeled training examples. ⁽³⁾

Figure 8: Manipulating latent codes on MNIST. Source: (4)