DreamNet

1 DreamNet

DreamNet is a deep SPD classification network that appends a shortcut-connected stacked Riemannian autoencoder to an SPDNet backbone to mitigate statistical-information degradation as depth increases.

Date of publication: 26/02/2023

Ref: @wangDreamNetDeepRiemannian2023

Code available at https://github.com/GitWR/DreamNet

Overview

The paper’s actual title is “DreamNet: A Deep Riemannian Manifold Network for SPD Matrix Learning.”

DreamNet uses a four-layer SPDNet backbone followed by a stacked Riemannian autoencoder (SRAE). Reconstruction supervision encourages the SRAE and its constituent Riemannian autoencoders (RAEs) to approximate identity mappings, while shortcut connections allow later RAEs to recover residual structural information from earlier stages. Each RAE has its own classifier; predictions are combined by maximum voting. Evaluated on AFEW (emotion), FPHA (hand action), and UAV-Human (skeleton action).

In the authors’ words, a “stacked Riemannian auto-encoder (SRAE) on the tail of the backbone network, i.e., SPDNet. […] we implant several residual-like blocks using shortcut connections to augment the representational capacity of SRAE, and to simplify the training of a deeper network.”

Architecture

Input representation: a sequence $S_i=[s_1,\dots ,s_{n_i}]$ is represented by the covariance

$$X_i=\frac{1}{n_i-1}\sum _{t=1}^{n_i}(s_t-u_i)(s_t-u_i{)}^{\top },u_i=\frac{1}{n_i}\sum _{t=1}^{n_i}{s}_t,$$

regularized as $X_i\leftarrow X_i+\rho I$ with $\rho =\mathrm{tr}(X_i)\times 10^{-3}$.

The four-layer SPDNet backbone is

$$X_i\to \mathrm{BiMap}\to \mathrm{ReEig}\to \mathrm{BiMap}\to \mathrm{ReEig},$$

giving $Z_i={\varphi }_{{\theta }_1}(X_i)$. The SRAE contains $E$ cascaded RAEs, each with five layers:

$${\hat{H}}_{e-1}\to {\mathrm{BiMap}}_{\text{input}}\to \mathrm{ReEig}\to {\mathrm{BiMap}}_{\text{hidden}}\to \mathrm{ReEig}\to {\mathrm{BiMap}}_{\text{reconstruction}},$$

with encoder/decoder bilinear maps

$$H_e=W_{e1}^{\top }{M}_e{W}_{e1},{\hat{H}}_e=W_{e2}{H}_e{W}_{e2}^{\top },$$

where $M_1=Z_i$. Every hidden representation also feeds a classifier ($H_e$ or ${\tilde{H}}_e$ → LogEig → FC → cross-entropy).

Shortcut connections use ambient element-wise matrix addition:

$${\tilde{H}}_e=H_e+{\tilde{H}}_{e-1},$$

followed by ReEig. Addition introduces no parameters, preserves positive definiteness when both operands are SPD, and is cheaper than the paper’s alternative Abelian group operation, which requires at least $O(d^3)$ computation. This is a residual-like SPD operation, but it is not the log-domain LogEuclideanResidual operation $\exp (\log X+\log Y)$ - DreamNet explicitly uses element-wise addition $X+Y$ for efficiency.

The compound loss is

$$\mathcal{L}({\theta }_2,\varphi ;\mathcal{X})=\sum _{e=1}^E\sum _{i=1}^N{\mathcal{L}}_e(X_i,l_i)+\lambda \sum _{i=1}^N{\mathcal{L}}_2(Z_i,{\hat{H}}_E),$$

where ${\mathcal{L}}_e$ is the cross-entropy of the $e$-th classifier and

$${\mathcal{L}}_2(Z_i,{\hat{H}}_E)={\Vert Z_i-{\hat{H}}_E\Vert }_F^2.$$

The authors justify the Euclidean/Frobenius reconstruction distance (instead of the Log-Euclidean metric) because it avoids matrix inversion during backpropagation and directly measures “statistical-level” similarity; by smoothness of $\log$ and $\exp$, infinitesimal Euclidean distance implies infinitesimal Log-Euclidean distance and conversely. Empirically, on AFEW, Euclidean reconstruction gives 36.59% at 31.32 s/epoch vs Log-Euclidean 36.71% at 88.16 s/epoch.

Model Parameters

Dataset	Input SPD	Backbone dimensions	Per-RAE dimensions
AFEW	$400\times 400$	$400\times 200$, $200\times 100$	$100\times 50$, $50\times 100$
FPHA	$63\times 63$	$63\times 53$, $53\times 43$	$43\times 33$, $33\times 43$
UAV-Human	$51\times 51$	$51\times 43$, $43\times 37$	$37\times 31$, $31\times 37$

Depth variants: DreamNet-27 ($E=3$ RAEs), DreamNet-47 ($E=5$), DreamNet-92 ($E=10$), DreamNet-182 ($E=20$). Parameter counts for 27/47/92: AFEW 0.36M / 0.53M / 0.95M; FPHA 0.11M / 0.18M / 0.36M; UAV-Human 0.10M / 0.16M / 0.31M.

Training Parameters

• Learning rate: 0.01.
• Batch size: 30.
• BiMap initialization: random semi-orthogonal matrices; FC: random matrices.
• ReEig threshold: $10^{-4}$ (AFEW, FPHA); $10^{-5}$ (UAV-Human).
• Reconstruction-loss trade-off $\lambda$: described only as a “small value” (exact value not reported).
• Optimizer, epochs, weight decay, momentum: not reported.
• Hardware: Intel i7-9700 3.4 GHz, 16 GB RAM; a GTX 2080 Ti did not accelerate training (eigenvalue operations are the bottleneck).
• FCMNet ablation: initial lr 0.02 / 0.05 (FCMNet-47 / FCMNet-92), attenuated ×0.9 every 100 epochs.

Results

Depth ablation (accuracy % and s/epoch):

Dataset	DreamNet-27	DreamNet-47	DreamNet-92
AFEW	36.59 (31.32 s)	36.98 (46.98 s)	37.47 (80.62 s)
FPHA	87.78 (2.60 s)	88.64 (3.66 s)	88.12 (6.70 s)
UAV-Human	44.88 (49.04 s)	45.57 (71.33 s)	46.28 (129.29 s)

DreamNet-182 obtains 46.03% on UAV-Human. Shortcut ablation (wSCMNet without shortcuts) on UAV-Human: 43.86 / 44.04 / 44.40 for 27/47/92 - all below the corresponding DreamNets. Classifier ablation on FPHA (DreamNet-27): all three classifiers combined 87.78%; single best classifier 87.39%; FCMNet (final classifier only) 87.18 / 87.60 / 87.30 for 27/47/92.

Comparison with SPDNet (best DreamNet):

Dataset	SPDNet	SPDNetBN	Best DreamNet
AFEW	34.23	36.12	37.47
FPHA	86.26	86.83	88.64
UAV-Human	42.31	43.28	46.28