CortexLab
Multimodal fMRI brain encoding toolkit with streaming inference, interpretability, and brain-alignment benchmarking
CortexLab extends Meta's TRIBE v2 foundation model for in-silico neuroscience. TRIBE v2 predicts fMRI brain activation from video, audio, and text inputs using a LLaMA 3.2-3B backbone. CortexLab adds the tooling researchers need to actually use these predictions: real-time streaming inference, interpretability visualization, cross-subject adaptation, and quantitative benchmarking.
The toolkit includes a brain-alignment benchmark that scores how "brain-like" any AI model's internal representations are using RSA, CKA, and Procrustes analysis. It also provides a cognitive load scorer that predicts mental demand from predicted activation patterns, and modality attribution that reveals which input modality (text, audio, or video) drives each brain region's response.
Key Highlights
Architecture Details
- Streaming Inference: Sliding-window predictor processes live feature streams with configurable window and step sizes for real-time brain activation prediction.
- ROI Attention Maps: Extracts and visualizes attention patterns from the transformer backbone, showing which brain regions attend to which temporal moments.
- Modality Attribution: Computes per-vertex importance scores for each input modality using gradient-based attribution, revealing what drives each brain region.
- Cross-Subject Adaptation: Ridge regression and nearest-neighbour methods adapt the pretrained model to new subjects with minimal calibration data.
- Brain-Alignment Benchmark: Quantitative framework to score how closely any AI model's representations match actual brain activation patterns.
- Cognitive Load Scorer: Predicts visual complexity, language processing demand, and overall cognitive load from predicted brain activations.