A scene holds more detail than a task may need. Foveated vision keeps the focus sharp and the periphery coarse, asking a different question: where should a model look?
Click Search to watch both models find the food in the same scene.
16 / 2,000
MONITOR FEED / SYNCHRONIZED
MODEL 01
VLM
SLIDING WINDOW
Total pixels used
0 pixels
RGB bandwidth proxy
0 KB
OV / CAM 01FRAME 0001
food
Awaiting HD frameReady for frame
Food found in 6.45s
MODEL 02
OpenVector VLM
FOVEATED ENCODING
Total pixels used
0 pixels
RGB bandwidth proxy
0 KB
BL / CAM 02FRAME 0001
1
2
3
food
Original frameShared input acquired
Food found in 4.88s
The bottleneck
More pixels. More to process.
High-resolution cameras capture rich scenes. Giving every part of every frame the same attention can be expensive.
01
Frames are dense
More resolution means more visual information to handle, even in regions that contribute little to the task.
02
Patches add work
Dividing a frame into smaller views preserves local detail, but processing and combining those views adds work.
03
Small details can disappear
Reducing the whole frame treats important and unimportant regions alike. A small object may be exactly what the question depends on.
The insight
Keep the detail where it matters.
Human vision combines a detailed center of gaze with a coarser periphery. Foveated sampling brings that idea to a model’s visual input.
Instead of shrinking everything equally, a sub-sampler preserves fine detail near a fixation. A sequence of fixations can bring different regions into focus while retaining the surrounding context.
Camera→Sub-sampler→Vision model
↳ Attention guides the next fixation ↲
A conceptual processing loop. The comparison above uses a preset path to make the idea visible.
Reported in the paper
A smaller input. A useful representation.
Finding 01
Models achieved up to 80% of their full capability using just 3% of the pixels in the paper’s evaluations.
