@kangSMLNetSPDManifold2025

SMLNet: A SPD manifold learning network for infrared and visible image fusion (2025)

Open in webOpen in zoteroOpen pdf

1 Abstract

Euclidean representation learning methods have achieved promising results in image fusion tasks, which can be attributed to their clear advantages in handling with linear space. However, data collected from a realistic scene usually has a non-Euclidean structure, evaluating the consistency of latent representations from paired views using Euclidean distance raises challenges. To address this issue, a novel SPD (symmetric positive definite) manifold learning is proposed for multi-modal image fusion, named SMLNet, which extends the image fusion approach from the Euclidean space to the SPD manifolds. Specifically, we encode images according to the Riemannian geometry to exploit their intrinsic statistical correlations, thereby aligning with human visual perception. The SPD matrix fundamentally underpins our network’s learning process. Building upon this mathematical foundation, we employ a cross-modal fusion strategy to exploit modality-specific dependencies and augment complementary information. To capture semantic similarity in images’ intrinsic space, we further develop an attention module that meticulously processes the cross-modal semantic affinity matrix. Based on this, we design an end-to-end fusion network based on cross-modal manifold learning. Extensive experiments on public datasets demonstrate that our framework exhibits superior performance compared to the current state-of-the-art methods. Our code will be publicly available at https://github.com/Shaoyun2023.

2 NOTES

The paper TRIES to use SPDNet into a auto-encoder, however, looking at their code nothing make sense. So, not a good paper. The best idea is shown in the second figure bellow: their construction of the matrix of interest. First, they split both infrared and visible light image into multiple blocks. These are then flattened into and combined into two matrices $X_{ir}$ and $X_{vi}$, and stacked $X_{stacked}=\left[\begin{array}{c}{X}_{ir}\\ X_{vi}\end{array}\right]$. This whole structure is then converted into a covariance matrix, and the most interesting aspect is that the resulting matrix is as follow:
$C_X=\left[\begin{array}{cc}{C}_{X_{ir}}& C_{X_{ir}{X}_{vi}}\\ C_{X_{vi}{X}_{ir}}& C_{X_{vi}}\end{array}\right]$
where $C_{X_{vi}{X}_{ir}}=C_{X_{ir}{X}_{vi}}$ and the main diagonals have the covariance matrices of the separated images.

The problem is that the SPDAM (Symmetric Positive Definite Attention Module), is very badly explained!!