research
An overview of my research on LLM and VLM post-training, unsupervised segmentation and self-supervised representation learning.
LLM & VLM post-training
My current focus is post-training large language models (LLMs) and vision-language models (VLMs) with minimal human supervision, in particular synthetic data generation, model-based data filtering and quality estimation, and self-improvement loops. OPST, the result of my Student Researcher time at Google, lets models improve themselves via on-policy self-distillation without any human-curated questions or answers.
OPST: On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
On-Policy Self-Training (OPST) is a data-free post-training method that lets an LLM or VLM improve itself without any human-curated questions or answers. Seeded only by coarse topic descriptors such as "Abstract Algebra", the model generates its own problems together with privileged information (a rationale and a candidate answer), keeps only well-posed problems it cannot yet solve reliably via pseudo-verification and a consensus gate, and trains on them with on-policy self-distillation from an EMA teacher that sees the privileged information. On Gemma 4 E2B, OPST improves MMLU by +10.2 points and outperforms SFT and GRPO on the same self-generated data; Visual OPST improves VQAv2 by +6.9 points using unlabeled images alone.
Unsupervised segmentation
During my Ph.D., I developed methods that segment images and videos without any human annotations: DepthG for unsupervised semantic segmentation, CutS3D for unsupervised instance segmentation, and S2D for unsupervised video instance segmentation. A common thread is to use geometric and temporal cues, such as depth, 3D structure and motion, as free supervision signals.
S2D: Sparse-To-Dense Keymask Distillation for Unsupervised Video Instance Segmentation
S2D is an unsupervised video instance segmentation method trained exclusively on real, unlabeled videos instead of the synthetic video sequences used by prior state-of-the-art methods. Keymask Discovery uses deep motion priors (point tracks) to select sparse, temporally coherent masks from noisy single-frame predictions, and Sparse-To-Dense Distillation with a Temporal DropLoss turns them into dense video labels. S2D outperforms existing unsupervised baselines on YouTube-VIS and achieves strong zero-shot performance when scaled to 13,000 real-world training videos.
Project page · arXiv · Code · Hugging Face
CutS3D: Cutting Semantics in 3D for 2D Unsupervised Instance Segmentation
CutS3D is an unsupervised instance segmentation method for 2D images that separates instances along their actual 3D boundaries. It cuts semantic masks in a point cloud of the scene obtained from monocular depth (LocalCut), sharpens semantics with a depth-based Spatial Importance function, and trains a class-agnostic detector with three Spatial Confidence components. CutS3D outperforms competing methods, including CutLER, on standard benchmarks for unsupervised instance segmentation and object detection such as COCO val2017, COCO20K and LVIS.
DepthG: Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation and Sampling
DepthG is an unsupervised semantic segmentation method that uses depth maps to align the learned feature space with the 3D structure of the scene. It learns depth-feature correlation, samples feature locations evenly in 3D with farthest-point sampling, and propagates the learning signal to neighboring patches (3D local hidden positives). Building on STEGO, DepthG outperforms relevant baselines on COCO-Stuff, Cityscapes and Potsdam.
Project page · arXiv · Code
Self-supervised representation learning
Self-supervised pre-training of image and 3D scene representations.
AttG: Attention-Guided Masked Autoencoders for Learning Image Representations
Attention-guided masked autoencoders inform MAE pre-training with attention maps from unsupervised object discovery (TokenCut), putting more weight on reconstructing relevant objects. The resulting representations improve over the vanilla MAE in linear probing and k-NN evaluations for classification, retrieval, semantic segmentation and Taskonomy tasks, and make ViTs more robust to background and texture changes.
Project page · arXiv · Code
MSM: Masked Scene Modeling: Narrowing the Gap Between Supervised and Self-Supervised Learning in 3D Scene Understanding
Masked Scene Modeling is a 3D-native self-supervised pre-training framework that narrows the gap between supervised and self-supervised learning in 3D scene understanding.
Further collaborations
Further work with my collaborators.
Weakly Supervised Virus Capsid Detection with Image-Level Annotations in Electron Microscopy Images
Detects virus capsids in electron microscopy images using only image-level annotations.
TTT-KD: Test-Time Training for 3D Semantic Segmentation through Knowledge Distillation from Foundation Models
Test-time training for 3D semantic segmentation via knowledge distillation from 2D foundation models.
Leveraging Self-Supervised Vision Transformers for Neural Transfer Function Design
Uses self-supervised vision transformer features to design transfer functions for volume rendering.
Evaluating Text-to-Image Synthesis: Survey and Taxonomy of Image Quality Metrics
A survey and taxonomy of image quality metrics for evaluating text-to-image synthesis.