Image-only training for video instance segmentation

QueenVIS: Rethinking Image-Only Training for Video Instance Segmentation via Query Enrichment

Arian Kheirandish1 Fardin Ayar1 Ehsan Javanmardi2 Manabu Tsukada2 Mahdi Javanmardi1,*
1 Amirkabir University of Technology, Tehran, Iran 2 The University of Tokyo, Tokyo, Japan * Corresponding author
Code

Can temporally stable object identities be learned without video supervision? QueenVIS shows that stronger image-trained queries can close much of the gap.

Qualitative demos

Video results

Example QueenVIS outputs on challenging video sequences.

Example 01
Example 02
Example 03
Example 04
Example 05
Motivation

Abstract

Video instance segmentation (VIS) requires models to detect, segment, and track object identities across frames, and most methods enforce this temporal consistency through video-level supervision. Image-only training approaches, with MinVIS as one prominent example, have challenged this assumption, reaching competitive VIS without video training by treating frames as independent images and associating instances only at inference. The field has nonetheless moved toward ever more elaborate video-trained trackers, which depend on costly temporally linked, identity-consistent annotations, leaving the image-only direction under-explored.

We revisit this direction and ask how far single-frame training can be pushed. A diagnostic analysis identifies object query quality as the under-explored bottleneck: queries trained only to localize objects within a frame drift apart across frames and destabilize association. QueenVIS introduces a query-centric framework for strengthening image-trained VIS. During single-frame training, we enrich Mask2Former queries with two auxiliary heads: a feature-prediction loss that aligns each query with the pooled backbone descriptor of its instance, and a center-prediction loss that injects spatial structure. Both heads are discarded at inference, adding zero parameters, and temporal identity is maintained by a training-free query-propagation and memory-bank scheme.

On YouTube-VIS and OVIS with a ResNet-50 backbone, QueenVIS consistently improves over MinVIS, with gains of up to +6.7 AP on YouTube-VIS, +4.8 AP on OVIS, and +10.3 AP on the long-sequence YouTube-VIS split. Beyond image-trained baselines, QueenVIS achieves 50.9 AP on YouTube-VIS and remains competitive with recent video-supervised state of the art, without processing a single video clip during training.

Approach

Enrich the query, not the training data

QueenVIS separates single-frame representation learning from training-free temporal association.

QueenVIS image-only training architecture with feature and center prediction branches
Image-only training. Appearance and center prediction supervise object queries using individual frames. Both auxiliary heads are used only during training.
QueenVIS temporal inference architecture with query propagation and a memory bank
Temporal inference. Confidence-guided query propagation and a lightweight memory bank maintain identity without video-level training.
01

Appearance-aware queries

Queries predict pooled backbone features for their matched instances, learning denser identity cues from single frames.

02

Spatially grounded queries

A center-prediction objective injects explicit spatial structure, helping distinguish similar nearby instances.

03

Training-free association

Query propagation and a small memory bank maintain identity through appearance changes and occlusion.

Results

Strong gains without video supervision

All comparisons use a ResNet-50 backbone.

+4.4 AP

YouTube-VIS 2019

Improvement over the image-only MinVIS baseline.

+6.7 AP

YouTube-VIS 2021

Query enrichment closes much of the video-supervision gap.

+4.8 AP

OVIS

A substantial improvement in the heavily occluded setting.

+10.3 AP

YouTube-VIS 2022 Long

The largest gain appears where long-horizon association matters most.

Main AP comparison with the image-only MinVIS baseline
Method Training OVIS YouTube-VIS 2019 YouTube-VIS 2021 YouTube-VIS 2022 Long
MinVIS Image only 25.0 47.4 44.2 23.3
QueenVIS Image only 29.8 51.8 50.9 33.6
Gain - +4.8 +4.4 +6.7 +10.3
Zero added parameters at inference

The auxiliary prediction heads are discarded after training, while QueenVIS maintains the same GFLOPs as MinVIS.

Three OVIS sequences comparing input frames, MinVIS predictions, and QueenVIS predictions during fragmentation, occlusion, and target disappearance
Qualitative comparison on OVIS. From top to bottom: input frames, MinVIS, and QueenVIS. Enriched queries reduce mask fragmentation, preserve identity through heavy occlusion, and suppress false assignments when a target disappears.
t-SNE plots comparing compact QueenVIS object-query clusters with more scattered MinVIS clusters
Queries that stay together across time. On OVIS, QueenVIS produces tight clusters for the same ground-truth identity across frames, while MinVIS queries drift and become harder to associate.
Takeaway

Video training is not the only path to temporal consistency.

Strong spatial and appearance modeling can make image-trained object queries far more stable across time. QueenVIS treats query quality as a complementary research axis to increasingly complex video supervision.

Reference

Citation

@misc{kheirandish2026queenvis,
  title  = {QueenVIS: Rethinking Image-Only Training for Video Instance Segmentation via Query Enrichment},
  author = {Kheirandish, Arian and Ayar, Fardin and Javanmardi, Ehsan and Tsukada, Manabu and Javanmardi, Mahdi},
  year   = {2026},
  note   = {Manuscript}
}