@huangGANDeadLong2025

The GAN is Dead; Long Live the GAN! A Modern GAN Baseline (2025)

Open in webOpen in zoteroOpen pdf

1 Abstract

There is a widely-spread claim that GANs are difficult to train, and GAN architectures in the literature are littered with empirical tricks. We provide evidence against this claim and build a modern GAN baseline in a more principled manner. First, we derive a well-behaved regularized relativistic GAN loss that addresses issues of mode dropping and non-convergence that were previously tackled via a bag of ad-hoc tricks. We analyze our loss mathematically and prove that it admits local convergence guarantees, unlike most existing relativistic losses. Second, our new loss allows us to discard all ad-hoc tricks and replace outdated backbones used in common GANs with modern architectures. Using StyleGAN2 as an example, we present a road map of simplification and modernization that results in a new minimalist baseline – R3GAN. Despite being simple, our approach surpasses StyleGAN2 on FFHQ, ImageNet, CIFAR, and Stacked MNIST datasets, and compares favorably against state-of-the-art GANs and diffusion models.

2 NOTES

This is an awesome paper on how to improve a complete GAN architecture, taking an already strong one (StyleGAN2), stripping it from a lot of its “techniques”, then building new ones upon it, focusing on simplification.

There are a couple of things that are relevant to speak here:

1. The losses RpGAN + $R_1$ + $R_2$: the first one is named relativistic pairing GAN (RpGAN) and is meant to address mode dropping $\mathcal{L}(\theta ,\psi )={\mathbb{E}}_{z\sim p_{z}x\sim p_D}[f(D_{\psi }(G_{\theta }(z))-D_{\psi }(x))]$ and as can be seen, it has a clear difference from the usual minimax loss, where $f(\cdot )$ is applied separately to each term. The idea is that in RpGAN a fake sample is critiqued by its realness relative toa real sample, which effectively maintains a decision boundary in the neighborhood of each real sample and hence forbids mode dropping.
2. However, RpGAN does not always converge, therefore a gradient penalty, with a similar idea to the GP penalty in WGAN-GP is applied to it. However, in here they are zero-centered, where the two most commonly-used 0-GP are $R_1$ and $R_2$: $R_1(\psi )=\frac{\gamma }{2}{\mathbb{R}}_{x\sim p_D}[\Vert {\nabla }_x{D}_{\psi }{\Vert }^2]$ $R_2(\theta ,\psi )=\frac{\gamma }{2}{\mathbb{R}}_{x\sim p_{\theta }}[\Vert {\nabla }_x{D}_{\psi }{\Vert }^2]$ $R_1$ penalizes the gradient norm of $D$ on real data, and $R_2$ penalizes the gradient of $D$ on fake data.

The table below shows all the configurations they tried for their network, after stripping StyleGAN2 from a ton of techniques (some that were only clear when looking at the code).

But the ones that stood out the most for me were:

• dimension of $z$ being only 64
• extremely small learning rate of $5\times 10^{-5}$
• Adam ${\beta }_1=0$ due to the small learning rate
• removal of batch normalization, as they say it is incompatible with $R_1$, $R_2$, or RpGAN.
• much simpler architecture (as can be seem below)
  

Overall, their results show that they achieved better or almost equal FID as StyleGAN and other models. And in the cases were StyleGAN2 was better, it is argued that they had feature leakage, so it can’t be properly compared. A great paper, the loss is great if it really works so well as they said. Will have to give it a test and will come back here with feedback.