Multimodal Neuroimaging — Brain Age Estimation
Predicting biological brain age from paired T1-weighted and FLAIR MRI on 712 ADNI subjects, using SFCN, a 3D Vision Transformer, and a hybrid CNN-transformer architecture — and a systematic comparison of how the two modalities should be combined.
The finding
Three ways to fuse T1w and FLAIR were trained and evaluated under identical conditions. Fusing the two modalities earlier in the network made predictions worse; fusing latest — combining two independently trained models' final opinions rather than their internal features — produced the best model in the whole project.
A linear model combining two independently trained T1w-only and FLAIR-only networks outperforms every jointly-trained fusion architecture tested — including the unimodal T1w baseline (R² 0.557).
Model comparison — held-out test set
Predicted vs. true age
Every point below is a real held-out test subject. Switch models to see how each architecture's error distribution actually looks — not just its summary statistic.
Three ways to combine two scans
How this came together
Baseline
SFCN (a lightweight 3D CNN), transfer-learned from a UK Biobank checkpoint and fine-tuned with a soft-label / KL-divergence age-regression objective (following Peng et al., 2021), was the strongest single-modality model: MAE 3.82 years, R² 0.557 on T1w alone. A Vision Transformer with a 3D-CNN tokenizer stem, and a hybrid architecture that feeds SFCN's pretrained features into a cross-attention transformer, were also built and tuned to close most of the gap to that CNN baseline.
Why naive fusion failed
Every multimodal model in the original project concatenated T1w and FLAIR features inside a jointly-trained network — technically intermediate fusion, regardless of what it was called. That approach consistently underperformed the T1w-only baseline. A controlled diagnostic (holding architecture fixed, varying only loss function and pooling) traced this to MSE regression collapsing toward the mean under weak signal, and mean-pooling discarding spatially localized information — both fixable, and fixing them roughly doubled the multimodal transformer branch's R².
The fusion-tier audit
That fix improved intermediate fusion, but didn't test whether intermediate fusion was the right idea in the first place. Building a genuine early-fusion baseline (channel-stacked input) and a genuine late/decision-fusion baseline (two independently trained models combined only by their final predictions) — neither of which existed in the original project — produced a clean, monotonic result: the later the fusion happens, the better the model performs. Decision-level fusion of two already-good, independently trained specialists beat every architecture that tried to learn from both modalities jointly.