@sunDeepCORALCorrelation2016

Deep CORAL: Correlation Alignment for Deep Domain Adaptation (2016)

Open in webOpen in zoteroOpen pdf

1 Abstract

Deep neural networks are able to learn powerful representations from large quantities of labeled input data, however they cannot always generalize well across changes in input distributions. Domain adaptation algorithms have been proposed to compensate for the degradation in performance due to domain shift. In this paper, we address the case when the target domain is unlabeled, requiring unsupervised adaptation. CORAL is a “frustratingly easy” unsupervised domain adaptation method that aligns the second-order statistics of the source and target distributions with a linear transformation. Here, we extend CORAL to learn a nonlinear transformation that aligns correlations of layer activations in deep neural networks (Deep CORAL). Experiments on standard benchmark datasets show state-of-the-art performance.

2 NOTES

In this paper the authors apply CORAL, which is a unsupervised domain adaptation method, into a deep neural network and evaluate its efficiency. Coral, is a method based on the approximation of the second-order statistics of source and target datasets. It is expressed as the following loss:
$L_{CORAL}=\frac{1}{4d^2}\Vert C_S-C_T{\Vert }_F^2$
where $C_S$ and $C_T$ are the covariance matrices of some input (which can be the data itself or features from some layer on the network). The authors express that this could be calculated from any layer, but as shown in figure bellow, they simply applied this to the last layer of the network alongside a usual classification loss. Therefore, their real loss is defined as:
$l=l_{CLASS.}+{\sum }_{i=1}^t{\lambda }_i{l}_{CORAL}$
where ${\lambda }_i$ is a parameter that implicates on the relation of the loss. They start with it small and then progressively increase it until it gets to the same proportion as the classification loss. They also started all layer, expect the last one, with pre-trained parameters, but did not froze them. What is interesting here is that it could relate to the covariance used in SPDNet-based neural networks, and not necessarily on the last layer alone, since the other also are covariance matrices.