Research

← Home

My research has focused on answering the following question:

How do we meaningfully evaluate and diagnose the capabilities and failures of modern vision and multimodal systems?

My work spans three connected directions:

  • Geometry and information in latent representations, probing how representation structure and content affect downstream model performance.
  • Spurious correlations and robustness, investigating what kinds of (mis)associations may exist in both models and datasets.
  • Multimodal evaluation, proposing new, complementary approaches to evaluate frontier model performance.

Geometry and information in latent representations

The usefulness of latent representations depends on both their geometry and the training objectives used. In distance-based few-shot classification, for instance, high-dimensional representations can exhibit hubness, where certain exemplar points appear in the nearest-neighbor lists of many other points, often hindering classification performance. In our work using hyperspherical embeddings 4, we showed that few-shot learning performance can benefit significantly if we learn a projection onto the hypersphere that balances two competing objectives: uniformity, where representations are spread further apart, and local similarity preservation, which encourages representations to preserve local neighborhoods.

Training objectives, alongside representation structure, are also important in diagnosing models. We showed that the masked image modeling (MIM) objective, for instance, leads to models that exhibit significant non-semantic positional biases in their representations 5. As a result, we proposed an orthogonal denoising strategy that suppresses these non-semantic directions. This work also demonstrates how useful diagnosis and intervention can occur in a post hoc fashion, without requiring the model to be retrained.

In prototypical self-supervised learning (e.g., DINO), we investigated the partial collapse problem, where multiple redundant prototypes converge to nearly identical representations. In 6, we traced this phenomenon to the joint training of the encoder and the prototypes and demonstrated that simply decoupling these two components during training leads to wider coverage and more stable prototypes in the learned model.

Spurious correlations and robustness

Modern vision and multimodal systems are replete with instances of shortcut learning. Part of my work therefore addresses two related questions: how can we identify shortcuts and improve group robustness in vision models, and how can we remove biases from their training datasets? In 3, we addressed the former challenge by merging clustering and an established explainability technique to demonstrate unsupervised group robustness (no access to group annotations). In 2, we addressed the latter question by investigating whether we could (a) diagnose vision datasets for object co-occurrence biases and (b) directly generate debiased data.

Multimodal evaluation

With the rise of multimodal large language models (MLLMs), the community has highlighted the so-called "language prior": the observation that much of a model's "reasoning" derives from the strength of its language decoder. For a system to be truly multimodal, it must synthesize information from multiple, complementary modalities.

Our work 1 on large video models began with the observation that modern video model benchmarks were overwhelmingly text-based. We investigated what would happen if we changed the evaluation modality from text to vision. We found that frontier video models demonstrated a significant drop in performance under this change.

This work addressed a long-standing concern of mine: what do frontier model benchmarks actually measure? Do they measure reasoning capabilities, or do they measure how well the model uses a particular modality (text) to reach the correct answer (the shortcut)?

An interdisciplinary perspective and future directions

Based on my interactions with scientists over the past few months, fine-grained abilities and representing data beyond the vision-language domain (e.g., diffusing wave spectra, neural recordings) remain key challenges for modern models.

As a result, over the next few years, I want to investigate the following directions in the context of scientific discovery:

  1. Representation structure: What representation geometries are useful for scientific prediction and inference tasks? In multimodal settings where embedding spaces are aligned, I am particularly interested in how information is preserved for downstream scientific tasks.

  2. Spurious correlations and robustness: When do multimodal models favor evidence from one modality that conflicts with task-relevant evidence from another? My work on video-language is one instantiation of this concern.

  3. Fine-grained abilities: How do we evaluate fine-grained capabilities of multimodal models in science? Illustrative examples include distinguishing between similar but subtly different structures in microscopy images or separating brain regions with overlapping selectivity.

These questions are connected to each other. For example, representation analysis could inform targeted evaluation methods for fine-grained abilities, while identifying shortcut reliance could motivate improved training and/or model/data design.

References

[1] Rwiddhi Chakraborty, Yinong Oliver Wang, Cheng Zhang, Zhuoran Yu, Fan Bai, Michael Kampffmeyer, Yong Jae Lee, Fernando de la Torre, Robert Jenssen. Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Models. 2026 Conference on Neural Information Processing Systems (To appear).

[2] Rwiddhi Chakraborty, Yinong Oliver Wang, Jialu Gao, Runkai Zheng, Cheng Zhang, Fernando de la Torre. Visual Data Diagnosis and Debiasing with Concept Graphs. 2024 Conference on Neural Information Processing Systems.

[3] Rwiddhi Chakraborty, Adrian Sletten, Michael C. Kampffmeyer. ExMap: Leveraging Explainability Heatmaps for Unsupervised Group Robustness to Spurious Correlations. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[4] Daniel J. Trosten*, Rwiddhi Chakraborty*, Sigurd Løkse, Kristoffer Knutsen Wickstrøm, Robert Jenssen, Michael C. Kampffmeyer. Hubs and Hyperspheres: Reducing Hubness and Improving Transductive Few-Shot Learning with Hyperspherical Embeddings. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. *Equal contribution.

[5] Martine Hjelkrem-Tan, Marius Aasan, Rwiddhi Chakraborty, Gabriel Y. Arteaga, Changkyu Choi, Adín Ramírez Rivera. Suppressing Non-Semantic Noise in Masked Image Modeling Representations. 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[6] Gabriel Y. Arteaga, Marius Aasan, Rwiddhi Chakraborty, Martine Hjelkrem-Tan, Thalles Silva, Michael Kampffmeyer, Adín Ramírez Rivera. Why Prototypes Collapse: Diagnosing and Preventing Partial Collapse in Prototypical Self-Supervised Learning. 2026 International Conference on Learning Representations.