1 MSNet

MSNet augments SPDNet with a multi-scale submanifold block that extracts and fuses local principal-submatrix geometry alongside the global SPD representation.

Date of publication: 26/06/2023

Ref: @chenRiemannianLocalMechanism2023

Overview

MSNet argues that existing SPD networks treat each SPD matrix as a global representation and neglect local geometric information. It identifies local patterns on an SPD manifold with regular SPD submanifolds formed by principal submatrices, then uses several scales of such submatrices for visual and skeleton-based action recognition. Evaluated on Cambridge-Gesture (CG), First-Person Hand Action Benchmark (FPHA), and a 50-class subset of UCF-101 (UCF-sub).

Architecture

An SPDNet backbone first produces lower-dimensional discriminative SPD features through repeated BiMap and ReEig layers:

S_{k - 1}^{i} = (W_{k - 1}^{i})^{⊤} S_{k - 2} W_{k - 1}^{i}, S_{k}^{i} = U_{k - 1}^{i} max (ϵ I, Σ_{k - 1}^{i}) (U_{k - 1}^{i})^{⊤} .

Each branch uses a BiMap-ReEig block with a different target scale. A submatrix-selection operation, SubSec, extracts principal submatrices $P_{k}^{i, 1}, \dots, P_{k}^{i, n_{i}}$ from branch $i$ . Principal submatrices are used because they remain SPD and therefore form regular SPD submanifolds; arbitrary square submatrices need not be SPD.

For a learned SPD feature interpreted as the covariance of a latent $d \times d$ random matrix, a $k \times k$ sliding receptive field corresponds to a $k^{2} \times k^{2}$ principal covariance submatrix. With step $s = 1$ , the selection produces $(\frac{d - k}{s} + 1)^{2}$ principal submatrices.

Each selected submatrix is mapped with LogEig: $M_{k}^{i, j} = lo g (P_{k}^{i, j})$ . TrilCon extracts and vectorizes the lower-triangular part, then concatenates all submanifolds within a branch. A final Concat layer combines all branch vectors, followed by a fully connected layer, softmax, and cross-entropy. One branch retains the entire SPD matrix as a trivial submanifold, providing global information.

The full MSNet-MS sequence is

S_{0} \to f_{b}^{(1)} \to f_{r}^{(2)} \to f_{b}^{(3)} \to f_{r}^{(4)} \to f_{m}^{(5)} \to f_{f}^{(6)} \to f_{s}^{(7)},

where $f_{b}$ , $f_{r}$ , $f_{m}$ , $f_{f}$ , and $f_{s}$ denote BiMap, ReEig, multi-scale submanifold, fully connected, and softmax layers.

Despite the paper’s convolutional motivation, MSNet does not define an SPD convolution or attention layer. Its “local mechanism” is principal-submatrix selection plus log-domain fusion.

Model Parameters

Five configurations are studied:

MSNet-H: holistic/global SPD information only.
MSNet-PS: all proper submanifold types, excluding the whole-matrix trivial submanifold.
MSNet-AS: all proper and trivial submanifolds.
MSNet-S: selected proper submanifolds used by MSNet-MS, without the global branch.
MSNet-MS: selected multi-scale proper submanifolds plus the global branch.

Dataset	BiMap settings	Submanifold scales
CG	$100 \to 80 \to 50 \to 25$	$2^{2}, 3^{2}, 4^{2}, 5^{2}$
FPHA	$63 \to 56 \to 46 \to 36$	$5^{2}, 6^{2}$
UCF-sub	$100 \to 80 \to 49$	$2^{2}, 6^{2}, 7^{2}$

The optimal ReEig threshold and BiMap dimensions are first searched with SPDNet, then reused unchanged in MSNet (exact thresholds not reported). Parameter counts and FLOPs are not reported.

Training Parameters

Initial manifold learning rate: $1 0^{- 2}$ ; decay ×0.8 every 50 epochs; minimum $1 0^{- 3}$ .
Batch size: 30.
BiMap initialization: random semi-orthogonal matrices.
Submatrix-selection step: 1.
Loss: softmax cross-entropy.
Hardware: Intel i5-9400 2.90 GHz CPU, 8 GB RAM.
Table-reported per-dataset learning rates/epochs: CG $1 0^{- 5}$ / 500; FPHA $1 0^{- 4}$ / 3,500; UCF-sub $1 0^{- 5}$ / 500. (The relation between these and the $1 0^{- 2}$ manifold schedule is not explained in the paper.)

Data: CG — 900 videos, 9 classes; 20 clips/class train, 80 test; frames $20 \times 20$ grayscale, PCA to 100 dims, $100 \times 100$ covariances. FPHA — 1,175 videos, 45 classes; 600 train / 575 test; 63-D joint vectors → $63 \times 63$ . UCF-sub — 50 classes × 100 clips; PCA to 100 dims; 70/30 gallery/probe split.

Results

FPHA recognition accuracy (%):

Method	Accuracy
Lie Group	82.69
Gram Matrix	85.39
SPDNet	85.57
SymNet	82.96
MSNet-H	85.74
MSNet-S	86.61
MSNet-MS	87.13

CG and UCF-sub accuracy (%):

Method	CG	UCF-sub
SPDNet	89.03	59.93
SymNet	89.81	56.73
MSNet-H	89.03	58.27
MSNet-S	90.14	59.40
MSNet-MS	91.25	60.87

UCF-sub backbone-depth ablation: MSNet-MS improves on SPDNet under every tested backbone, with the largest gain in the most underfitting configuration (e.g., ${100, 80, 50, 25, 9}$ : 28.73 → 36.07). Training time per epoch: CG 0.53 s (SPDNet) vs 1.29 s (MSNet-MS); FPHA 1.53 vs 2.67; UCF-sub 11.33 vs 34.50.

Code

Official source and supplement: https://github.com/GitZH-Chen/MSNet.git

Lucas Heck's Notes

Explorer

MSNet