Continuous Perception Benchmark (CP-Bench)

CP-Bench is built on the CLEVR dataset, using Blender to render synthetic scenes. Each scene consists of a static planar surface populated by a set number of cubes. All cubes share identical appearance, i.e are identical in shape, color, size, material, and texture. This is a critical design choice made to eliminate texture-based shortcuts and prevent recognition based on superficial visual cues. The virtual camera performs a single, smooth horizontal panning motion across the scene over a duration of 10 seconds. The camera's field of view is constrained, ensuring that only a subset of the total cubes is visible at any given moment. All rendering parameters other than cube positions are kept constant across scenes. The task is to answer the question: "How many cubes are present in the scene?". The benchmark includes five object-count configurations: 3, 4, 5, 6, and 7 cubes. We generated 20 unique instances for each count, resulting in a balanced diagnostic set of 100 total video instances.

To further eliminate potential confounding factors and verify that model failures on CP-Bench are not attributable to low-level perceptual limitations, we design a control test set that preserves the exact visual appearance of the main benchmark while removing the need for continuous perception. In this control condition, the camera remains static, and all cubes are simultaneously visible in every frame. The control set contains the same five ground-truth object counts (3–7), with 20 instances per count, for a total of 100 test videos.

Experiment Results

Main Results

We evaluate several state-of-the-art models on our benchmark, including the Qwen-VL and InternVL series for open-source models, as well as GPT and Gemini for proprietary models. Test videos are sampled at 1 fps. A summary of the results is provided in the table below.

None of the models are able to reliably infer the total number of cubes when continuous temporal integration is required. These results demonstrate that state-of-the-art multimodal systems, despite their impressive abilities on existing video benchmarks, lack the capacity for continuous perception. The dramatic accuracy drop compared to the control condition suggests that failure is not due to object recognition ability or difficulty in counting identical cubes. Rather, the models struggle to maintain object permanence, form temporally stable correspondences, and integrate sequential observations into a coherent global representation of the scene.

Does higher sampling FPS help?

To examine whether denser temporal sampling improves performance, potentially enabling models to approximate continuous perception by observing more frames, we conduct an additional experiment varying the input frame rate from 1 to 5 fps.

Results show that increasing the frame rate does not meaningfully improve performance. This supports our central hypothesis: contemporary video models do not perform continuous perception, and increasing input density alone cannot compensate for this architectural and algorithmic limitation.

Finetuning on Synthetic Training Data

To further probe whether current models can learn continuous perception when provided with supervised data, we generate a synthetic training set that mirrors the structure of CP-Bench. Using the same rendering pipeline, we create 1,000 training videos: 200 instances for each ground-truth count from 3 to 7. All scenes strictly avoid overlap with test-set configurations, ensuring a clean separation between training and evaluation. We then finetune Qwen2.5-VL-7B for 5 epochs.

The finetuned model achieves 89% accuracy on the CP-Bench test set, which is significantly higher than the 40% accuracy of the original model. At first glance, this might suggest that the model has successfully acquired continuous perception capabilities. However, the extreme simplicity and uniformity of our setting raise the possibility that the model may have discovered shortcut strategies instead of learning to integrate information across time.

To distinguish true continuous perception from shortcut-based learning, we modify the training set by shortening video duration from 10 seconds to 5 seconds, effectively halving the camera’s traversal distance. The test set remains unchanged (10 seconds). If the model had learned a temporally generalizable perception strategy, then reducing training video length should not substantially impact test performance. However, under this setting, the finetuned model achieves only 20% accuracy on the 10-second test set, a drastic drop from the earlier 89%. We also perform the reverse experiment: training on 10-second videos and testing on 5-second videos. Again, the model achieves only 20% accuracy while training on 5-second achieves perfect accuracy.

This result reinforces our core thesis that current architectures and training paradigms fundamentally lack mechanisms for robust continuous perception. Notably, this train-test-generalization setup provides a simple testbed for distinguishing genuine continuous perception from superficial overfitting, facilitating development of models explicitly designed to maintain temporally cohesive representations. Ultimately, models that achieve true spatio-temporal reasoning should be capable of generalizing to arbitrarily long videos without performance degradation.

Conclusion

In this work, we argue that the prevailing paradigm of processing video as a collection of discrete, sampled frames is a fundamental barrier to achieving human-like visual intelligence. This approach leads to a fragmented and superficial understanding, in stark contrast to the holistic, continuous perception employed by humans. To empirically demonstrate this gap, we introduced the Continuous Perception Benchmark (CP-Bench), a minimalist diagnostic task where models must count visually identical objects in a panning video. This simple setup effectively neutralizes the texture-based shortcut solutions that many contemporary benchmarks permit, isolating the core capability of spatio-temporal correspondence. Despite the benchmark’s extreme simplicity, our experiments reveal that modern state-of-the-art multimodal models consistently fail on this trivially simple task.

We hope that this work opens a new research direction toward models capable of genuinely continuous perception, i.e., systems that track persistent structure in the world, accumulate information over time, and reason about scenes the way humans do. Achieving this capability will likely require innovations in architecture, memory, temporal modeling, and training methodology. CP-Bench offers a stepping stone toward that goal, highlighting both the limitations of current approaches and the opportunities for progress in building physically grounded, temporally coherent multimodal intelligence.

@article{cpbench2025,
  title={Continuous Perception Matters: Diagnosing Temporal Integration Failures in Multimodal Models},
  author={Zeyu Wang and Zhenzhen Weng and Serena Yeung-Levy},
  journal={arXiv preprint arXiv:2408.07867},
  year={2025}
}