Video language models (VLMs) can describe a sunset, identify a golden retriever, and explain the plot of a movie trailer. They can tell you what's in a frame with impressive accuracy. But ask one how many times a person opened a door in a 30-second clip, and it will likely get the answer wrong.
This isn't a minor glitch. It's a fundamental failure in how these models understand—or fail to understand—events unfolding over time.
The promise of VLMs is enormous: systems that can watch surveillance footage and flag anomalies, analyze sports plays in real time, or guide you through a recipe by tracking each step. Yet when tested on simple event bookkeeping tasks—counting occurrences, ordering actions, tracking sequences—these models stumble. They perform near chance level on benchmarks designed to test exactly these skills.
The culprit? A phenomenon we call the low frequency trap.
Here's the core problem: VLMs are trained on massive datasets where static visual content dominates. A door might be visible in 90% of frames, but the act of opening it happens in just a few. The model learns to associate the question "How many times did the person open the door?" with the most frequent visual cue—the door itself—rather than the rare, dynamic event of it actually opening. So it guesses "once" when the answer is three.
This article breaks down 7 reasons why VLMs fail at event bookkeeping, from data imbalances to architectural blind spots. We'll examine benchmark evidence, real-world consequences, and what it would take to fix this. If you're building with video AI or simply paying attention to where the field is headed, this is the gap you need to understand.
Key Takeaway: Video language models excel at recognizing what's in a frame but struggle with tracking what happens across frames. This "low frequency trap" makes them unreliable for tasks that require counting, ordering, or sequencing events.
Event bookkeeping is the ability to track the occurrence, order, and count of events across a video. It's the cognitive skill that lets you say, "The chef added salt, then pepper, then tasted the soup three times."
For a VLM, this means answering questions like:
These seem trivial. A human can do them without conscious effort. For VLMs, however, they're surprisingly hard.
Why it matters:
Event bookkeeping is foundational for any real-world application where timing and sequence matter:
If a model can't reliably count how many times something happened or determine the order of events, it can't be trusted for these tasks. Event bookkeeping is a fundamental test of true video understanding—not just recognizing objects, but comprehending dynamic processes.
Key Takeaway: Event bookkeeping is the ability to track what happens, when, and how often. It's essential for surveillance, sports, instruction, and any domain where sequence and count matter.
The low frequency trap describes a failure mode where models rely on frequent, static visual cues rather than rare, dynamic event cues.
Here's how it works. Imagine a 30-second video of someone entering and leaving a room through a door. The door is visible in almost every frame. The act of opening it—the actual event—takes maybe 2 seconds each time. If the person opens the door three times, that's 6 seconds of "opening" out of 30.
A VLM trained on this data learns that the door is the most salient, frequently occurring visual element. When asked "How many times did the person open the door?", the model latches onto the door's presence and guesses "once"—because that's the most common association in its training data.
This is analogous to frequency bias in NLP. In language models, common words like "the" and "is" dominate training data, while rare but important tokens (like "not" in "do not") can be overlooked. The same principle applies to video: frequent static elements overshadow rare dynamic events.
How the trap manifests:
This isn't a bug in a specific model. It's a systemic issue across architectures and training regimes. The low frequency trap is baked into how these models learn from data.
Key Takeaway: The low frequency trap causes VLMs to focus on static, frequently visible objects rather than rare, dynamic events. This leads to errors in counting and sequencing because the model confuses what's present with what actually happened.
VLMs are trained on enormous video-text datasets. The problem? These datasets are dominated by static frames with weak event labels.
Consider HowTo100M, one of the largest video-text datasets. It contains over 136 million video clips scraped from YouTube, paired with narration transcripts. But those narrations are weak labels—they describe what's happening in general terms, not when or how many times specific events occur.
What's missing:
Without this supervision, models learn spurious correlations. They associate questions with the most salient visual features in the training data, not with the actual events being asked about.
Example: A model trained on cooking videos might see salt in most frames and learn that "salt" is associated with "adding salt." When asked "Did the chef add salt before or after pepper?", it answers "before" because salt appears more frequently—not because it tracked the sequence.
The consequence is clear: models fail to generalize to event bookkeeping tasks because they never learned to attend to the rare, dynamic moments that define those events.
Key Takeaway: Training datasets like HowTo100M provide weak, narration-based labels that lack dense event annotations. This imbalance teaches models to rely on static visual cues instead of learning to track dynamic events.
Even with better data, many VLM architectures are fundamentally ill-suited for event bookkeeping.
The core issue: Most models use uniform frame pooling—they sample frames at regular intervals, extract features, and average them. This discards temporal order.
Imagine watching a movie by shuffling the frames and averaging them. You'd lose the plot entirely. That's what frame pooling does to temporal information.
What the research shows:
Analyses of attention maps in VLMs reveal low entropy across time. In plain terms, the model focuses on a single keyframe rather than distributing attention across the video. It's not tracking events; it's finding the most representative frame and answering based on that.
Even temporal attention and memory banks struggle. Some models attempt to address this with mechanisms like temporal attention (attending to multiple frames) or memory networks (storing information across time). But performance remains limited on simple counting tasks.
Why? Because these mechanisms are often bolted on rather than integrated into the core architecture. The model still learns to rely on static features because that's what the training data rewards.
What's needed: Explicit temporal modeling with event boundaries—architectures that understand when one event ends and another begins, and that can maintain a running count or sequence.
Key Takeaway: Most VLMs use frame pooling that averages features and discards temporal order. Attention maps show models focus on single keyframes rather than tracking events over time. Fixing this requires architectural changes, not just fine-tuning.
The failures aren't anecdotal. Benchmarks designed to test temporal reasoning show consistent, dramatic gaps.
NExT-QA (Xiao et al., CVPR 2021): - State-of-the-art VLMs achieve 60-70% accuracy on temporal questions - But 80-90% on descriptive questions - The gap shows models are much better at "what" than "when" or "how many"
AGQA (Grunde-McLaughlin et al., CVPR 2021): - Models like VideoLLaMA and Video-ChatGPT score below 30% exact match on counting tasks - This is near chance level for many questions
STAR (Wu et al., NeurIPS 2021): - Top models achieve ~50% accuracy on sequence questions - Humans perform over 90% - The benchmark tests situated reasoning—understanding actions in context
TVQA (Lei et al., EMNLP 2018): - Contains 152k question-answer pairs, many requiring temporal localization - Models often fail on "before/after" questions
GPT-4V custom evaluation: - In informal tests, GPT-4V correctly counted events in only 45% of cases - Humans achieved 95%
These aren't edge cases. They're systematic failures on tasks that should be straightforward.
Key Takeaway: Across benchmarks like NExT-QA, AGQA, STAR, and TVQA, VLMs consistently underperform on counting, sequencing, and temporal ordering. The gap between model and human performance is stark—often 40-50 percentage points.
These failures aren't just academic. They have practical consequences.
Surveillance: A VLM monitoring a restricted area fails to count how many people entered because it focuses on the static background rather than the brief entry events. An intrusion goes unnoticed.
Sports analytics: A model tracking passes in a soccer match miscounts because it relies on the most frequent player positions rather than tracking the ball. Coaches and analysts get bad data.
Instructional videos: A VLM describing cooking steps gets the order wrong because it uses frame-level features without temporal ordering. A viewer following along adds ingredients in the wrong sequence.
Medical monitoring: A system tracking medication administration or procedural steps fails to count doses or verify sequence. Patient safety is compromised.
The common thread: In any domain where accuracy matters—where missing a rare event or miscounting has consequences—these failures reduce reliability. A model that can't reliably count events can't be trusted for critical tasks.
Key Takeaway: Event bookkeeping failures have real-world impact in surveillance, sports, instruction, and healthcare. When models can't count or sequence events, they become unreliable for any task where accuracy is essential.
The low frequency trap isn't insurmountable, but escaping it requires more than incremental improvements.
Current approaches:
What's working: Some models show improvement on specific tasks with these techniques. But gains are often narrow—fine-tuning on one benchmark doesn't generalize to others.
What's not working: Fine-tuning alone doesn't solve the problem. The low frequency trap is rooted in data imbalances and architectural limitations. You can't fine-tune your way out of a model that fundamentally doesn't track events.
The path forward:
This is a multi-year effort. But it's necessary if VLMs are going to move from demo-ready to deployment-ready.
Key Takeaway: Escaping the low frequency trap requires joint advances in data (dense event annotations), architecture (explicit temporal modeling), and evaluation (benchmarks that test bookkeeping). Fine-tuning alone won't fix it.
The low frequency trap is a critical barrier for video language models. It explains why these systems—impressive as they are—fail at tasks that seem trivial: counting how many times something happened, ordering events correctly, tracking sequences over time.
The trap is rooted in:
Overcoming it requires coordinated advances across all three. Better data with dense annotations. Architectures that explicitly model time and events. Benchmarks that reward true temporal reasoning.
The stakes are high. Reliable event bookkeeping is essential for deploying VLMs in surveillance, sports, healthcare, and any domain where accuracy matters. A model that can't count events can't be trusted.
The field is making progress. New benchmarks like TempCompass and Video-MME are pushing models to do better. But there's a long way to go.
Stay informed. Demand better. As video AI moves into real-world applications, the ability to track events accurately won't be optional—it'll be the baseline.
What is the 'low frequency trap' in video language models?
The low frequency trap is a failure mode where VLMs rely on frequent, static visual cues (like objects present in most frames) rather than rare, dynamic event cues (like a brief action). This causes errors in counting and sequencing because the model confuses what's visible with what actually happened.
Why do VLMs struggle with simple event counting?
VLMs are trained on datasets dominated by static frames with weak event labels. They learn to associate questions with the most salient visual features rather than tracking events over time. Architecturally, many models use frame pooling that discards temporal order, making it impossible to count or sequence events accurately.
What are some benchmarks for testing event bookkeeping in VLMs?
Key benchmarks include NExT-QA (temporal vs. descriptive questions), AGQA (compositional spatio-temporal reasoning), STAR (situated reasoning), and TVQA (localized video QA). Newer benchmarks like TempCompass and Video-MME are also designed to test temporal understanding.
How does the low frequency trap affect real-world applications?
In surveillance, models miss rare events like intrusions. In sports analytics, they miscount passes or actions. In instructional videos, they get step order wrong. Any application where counting or sequencing matters is compromised.
Can VLMs be improved to overcome the low frequency trap?
Yes, but it requires more than fine-tuning. Advances are needed in data (dense event annotations), architecture (explicit temporal modeling), and evaluation (benchmarks that test bookkeeping). Some models show improvement with temporal attention and memory networks, but gains are limited.
Is the low frequency trap similar to frequency bias in NLP?
Yes. In NLP, models overfit to common tokens and underweight rare but important ones (like "not" in negations). In video, models overfit to frequent static visual elements and ignore rare dynamic events. The principle is the same.
What is the difference between event bookkeeping and action recognition?
Action recognition identifies what action is happening in a clip (e.g., "opening a door"). Event bookkeeping tracks occurrence, order, and count across the video (e.g., "the door was opened three times, first at 00:03, then 00:12, then 00:27"). Bookkeeping requires temporal reasoning; recognition does not.
Do all VLMs suffer from the low frequency trap?
Most do, to varying degrees. The trap is rooted in common training data and architectures. Some models with explicit temporal mechanisms perform better on specific tasks, but no model has fully solved the problem.
How can we evaluate event bookkeeping performance?
Use metrics like exact match accuracy for counts (e.g., "3" vs. "1") and temporal ordering accuracy (e.g., "before" vs. "after"). Benchmarks like AGQA and STAR provide standardized tests.
What is the role of training data in the low frequency trap?
Training data with weak event labels and imbalanced static vs. dynamic content teaches models to rely on frequent visual cues. Without dense event annotations, models never learn to track rare, dynamic events—which is exactly what bookkeeping requires.
Want to dive deeper into the future of video AI? Subscribe to our newsletter for the latest research breakdowns and expert insights on multimodal learning.