U-SPDNet

1 U-SPDNet

U-SPDNet extends SPDNet with a symmetric manifold-valued decoder, reconstruction supervision, and Log-Euclidean skip connections to mitigate structural-information degradation.

Date of publication: 14/12/2022

Ref: @wangUSPDNetSPDManifold2023

Source code: https://github.com/GitWR/U-SPDNet

Overview

U-SPDNet treats successive low-dimensional BiMap operations as a source of structural-information loss. Its contracting path is an SPDNet encoder, while a symmetric expanding path reconstructs the original SPD representation from the encoder output. A reconstruction error term encourages the complete manifold-to-manifold embedding to approximate an identity map, while cross-entropy supervises the encoder representation used for classification. Skip connections fuse corresponding encoder and decoder features through a Log-Fusion-Exp (LFE) implementation of the Log-Euclidean Riemannian barycenter. Evaluated on MDSD, Virus, FPHA, and UAV-Human.

Quote

The fundamental reasons for the effectiveness of Riemannian neural networking technique lie in two aspects: (1) the Riemannian geometry of the input data manifold can be preserved through all the layers during training; (2) the deep and nonlinear feature embedding mechanism.

Architecture

The full pipeline is

$$M_0\to \underset{\text{encoder}}{\underbrace{\mathrm{BiMap}\to \mathrm{ReEig}\to \mathrm{BiMap}\to \mathrm{ReEig}\to \mathrm{BiMap}}}\to \{\begin{array}{l}\mathrm{LogEig}\to \mathrm{FC}\to \text{cross-entropy},\\ \text{symmetric SPD decoder with LFE skip connections}.\end{array}$$

Encoder

The BiMap layer is $M_k=W_k^{\top }{M}_{k-1}{W}_k$ with $W_k\in {\mathbb{R}}^{d_{k-1}\times d_k}$, $d_k<d_{k-1}$; full column rank preserves positive definiteness and semi-orthogonality places $W_k$ on a compact Stiefel manifold. The ReEig layer computes $M_k=U_{k-1}\max (ϵI,{\Sigma }_{k-1})U_{k-1}^{\top }$. The classification branch applies LogEig ($\bar{M}=\log (M)=U\log (\Sigma )U^{\top }$), symmetric vectorization, a fully connected projection $Z$, softmax, and cross-entropy.

Decoder

The decoder mirrors the encoder and upsamples SPD matrices with $M_{\alpha }=W_{\alpha }{M}_{\alpha -1}{W}_{\alpha }^{\top }$ where $W_{\alpha }\in {\mathbb{R}}^{d_{\alpha }\times d_{\alpha -1}}$ and $d_{\alpha }>d_{\alpha -1}$. ReEig is applied in the decoder so that the upsampled outputs remain SPD. The symmetric structure avoids extra dimensional-alignment parameters at skip connections.

LFE Skip Connections

For corresponding encoded and decoded SPD features $\{M_h{\}}_{h=1}^H$, the Log-Euclidean Fréchet mean has closed form

$$P^{\ast }=\exp \left[\frac{1}{H}\sum _{h=1}^H\log (M_h)\right],$$

implemented as the LFE module:

\{M_h\} \xrightarrow{\ [[Deep Riemannian Network (DRN)#LogEig|LogEig]]\ } \{\log M_h\} \xrightarrow{\ [[Deep Riemannian Network (DRN)#AriM|AriM]]\ } \widetilde A = \frac{1}{H}\sum_h\log M_h \xrightarrow{\ [[Deep Riemannian Network (DRN)#ExpEig|ExpEig]]\ } P^*.

The AriM (arithmetic mean) layer computes the Euclidean barycenter $\tilde{\mathbf{A}}$ in the log domain. The paper calls this Riemannian optimization, but the skip connection is realized through the closed-form Log-Euclidean barycenter rather than explicit parallel transport.

Reconstruction and Total Losses

The reconstruction error term uses the Euclidean/Frobenius metric:

$$L_1({\theta }_1,{\theta }_2,\varphi ;\mathcal{M})=\sum _{i=1}^N\Vert M_i-{\hat{M}}_i{\Vert }_F^2,$$

where ${\hat{M}}_i$ is the reconstructed SPD matrix. The classification term is cross-entropy, and the complete objective is

$$L({\theta }_3,\varphi ;\mathcal{M})=\lambda \sum _{i=1}^N{L}_1(M_i,{\hat{M}}_i)+\sum _{i=1}^N{L}_2(M_i,c_i),$$

where ${\theta }_3=\{{\theta }_1,{\theta }_2,Z\}$ and $\lambda$ balances reconstruction and classification.

The paper also derives the matrix backpropagation for ExpEig,

$$\frac{\partial L^{(k)}}{\partial M_{k-1}}=2U\left(\Lambda \circ (U^{\top }{\Omega }_1{)}_{\mathrm{sym}}\right)U^{\top }+U{\Omega }_2{U}^{\top },{\Lambda }_{ij}=\{\begin{array}{ll}\frac{1}{{\pi }_i^2-{\pi }_j^2},& i\ne j,\\ 0,& i=j.\end{array}$$

Model Parameters

Dataset	Encoder dimensions	Decoder dimensions	Parameters
MDSD	$400\to 200\to 100\to 50$	$50\to 100\to 200\to 400$	0.24 M
Virus	$400\to 200\to 100\to 40$	$40\to 100\to 200\to 400$	0.23 M
FPHA	$63\to 53\to 43\to 33$	$33\to 43\to 53\to 63$	0.06 M
UAV-Human	$51\to 43\to 37\to 33$	$33\to 37\to 43\to 51$	not reported

Dataset	Batch size	ReEig $ϵ$	Loss weight $\lambda$
MDSD	20	$10^{-5}$	$10^{-6}$
Virus	10	$10^{-4}$	$10^{-6}$
FPHA	30	$10^{-5}$	$10^{-1}$
UAV-Human	30	$10^{-5}$	$10^{-2}$

Input covariance matrices are regularized as $M_i\leftarrow M_i+\beta I$ with $\beta =\mathrm{tr}(M_i)\times 10^{-3}$.

Training Parameters

Setting	MDSD	Virus	FPHA	UAV-Human
Optimizer	SGD on Stiefel manifolds (Riemannian matrix backprop)	Same	Same	Same
Initial learning rate	0.01	0.01	0.01	0.01
Schedule	×0.8 every 50 ep	×0.8 every 50 ep	not reported	×0.9 every 50 ep
Maximum epochs	500	300	1,400	1,400
Time/epoch	2.84 s	2.35 s	3.62 s	not reported

Hardware: i7-9700 3.4 GHz CPU, 8 cores, 16 GB RAM. A GTX 2080 Ti did not accelerate training (the sequence of eigenvalue operations is the bottleneck). BiMap initialization: random semi-orthogonal.

Data protocols: MDSD 7 train / 3 test videos per class, $20\times 20$ frames → $400\times 400$ covariances; Virus 3 train / 2 test sets of $400\times 400$; FPHA 63-D joints → $63\times 63$, 600 train / 575 test; UAV-Human 51-D (PCA 99% energy) → $51\times 51$, 16,723 sequences split 70:30.

Results

Reconstruction metric comparison on MDSD:

Reconstruction metric	Accuracy (%)	Time (s/epoch)
RET-EuM	38.97	2.84
RET-LEM	39.49	6.97

LEM improves accuracy by 0.52 points but more than doubles epoch time, confirming the choice of EuM.

MDSD and Virus (accuracy %):

Method	MDSD	Virus
SPDNet	$32.05\pm 1.81$	$65.00\pm 3.60$
SymNet	$35.58\pm 8.16$	$73.00\pm 8.67$
U-SPDNet	$38.97\pm 2.15$	$73.67\pm 2.46$

FPHA (accuracy %): SPDNet 86.26, SymNet 82.96, U-SPDNet 87.83. UAV-Human: SPDNet 42.31, SymNet 35.89, U-SPDNet 43.39.

Ablations:

Architecture	MDSD	Virus	FPHA
SPDNet baseline	$32.05\pm 1.81$	$65.00\pm 2.36$	86.26
U-SPDNet without skip connections	$37.43\pm 2.93$	$72.67\pm 2.63$	87.13
U-SPDNet	$38.97\pm 2.15$	$73.67\pm 2.46$	87.83

Fusion method	MDSD	Virus	FPHA
Direct AriM (arithmetic)	$37.18\pm 2.56$	$73.00\pm 2.46$	87.42
LFE Riemannian barycenter	$38.97\pm 2.15$	$73.67\pm 2.46$	87.83

Trade-off parameter $\lambda$ searched in $\{10^{-3},10^{-4},10^{-5},10^{-6},10^{-7},0\}$; best MDSD/Virus at $\lambda =10^{-6}$. $\lambda =0$ removes reconstruction supervision and reduces to the SPDNet baseline.

Notes and Caveats

• The decoder reconstructs SPD representations, the embedding is not bijective, so U-SPDNet cannot segment like a Euclidean U-Net.
• The skip connection uses the Log-Euclidean barycenter (LogEig → AriM → ExpEig); no explicit parallel transport is given.
• No momentum or weight decay is reported. U-SPDNet was not evaluated on EEG.