Show a video language model (VLM) a clip of a person in a kitchen, and it will produce a fluent, confident paragraph: "A person is cooking in a bright kitchen. They crack eggs into a bowl, add ingredients, and stir the mixture." Impressive. Then ask a follow-up question that a five-year-old could answer: "How many times did the person stir?" The model answers "once." The actual count is seven.
This is not an edge case. It is the default behavior of most video language models deployed today.
VLMs are multimodal systems that ingest video frames alongside text and produce answers, captions, or descriptions. They have become a fixture of multimodal AI research, powered by the same transformer architectures that transformed image and language modeling. On benchmarks that reward descriptive fluency, they look close to solved. On benchmarks that require tracking what happened, in what order, and how many times, they fall apart.
The failure is easiest to see with counting. Ask a VLM to count jumps, passes, handshakes, or door openings in a video, and the answer often bears little relation to reality. In one representative case, a model shown a video of a person jumping twice answered "once." The reason is not that the model cannot see a jump. It is that jumping is rare in the training data relative to standing, walking, and sitting, so the model's output distribution is pulled toward the common answer.
This is the low frequency trap: a failure mode where VLMs perform poorly on events that occur infrequently in training data, defaulting instead to common, high-frequency events. The trap is not a bug in a single model. It is a structural property of how these systems are trained and evaluated.
Event bookkeeping is the ability to track and reason about the occurrence, count, order, and duration of events in a video. It answers questions like:
This is distinct from recognizing what is in a single frame. A model can identify "a person is jumping" in a still image and still fail to count how many jumps occurred across a 30-second clip. Bookkeeping requires temporal integration — maintaining state across frames and updating it as new evidence arrives.
The gap between descriptive accuracy and temporal understanding has consequences. Surveillance systems miss rare security events. Sports analytics miscount passes. Robots misread the sequence of human actions. Cooking assistants suggest the wrong ingredient because they follow common associations rather than the actual order of events. In safety-critical settings, the cost of a missed rare event is not a lower benchmark score — it is an accident.
Key Takeaway: VLMs can describe video content fluently while failing at basic event bookkeeping. The low frequency trap explains why: models are biased toward common events and struggle when the answer depends on rare ones.
This deep-dive moves from definition to diagnosis to mitigation. We will define event bookkeeping and distinguish it from action recognition. We will examine the low frequency trap in detail, including its roots in the long-tail problem. We will survey the benchmarks that expose the failure — NExT-QA, AGQA, STAR, TempCompass, and VITATECS. We will explain why VLMs fail, covering language priors and single-frame bias. We will trace real-world consequences, review mitigation techniques, and close with the open challenges that remain.
Event bookkeeping is the set of temporal reasoning tasks that require integrating information across multiple frames:
Each of these requires the model to maintain and update an internal state as the video unfolds. Counting is the purest test: it cannot be solved by looking at any single frame. You must remember what you have already seen and increment.
Action recognition classifies a short clip into a category: "jumping," "running," "cooking." It is a single-label problem, often solved with a few frames or even one. Event bookkeeping is a sequence problem. The same action appearing twice versus five times produces the same recognition label but different bookkeeping answers.
This distinction is where many evaluation pipelines go wrong. A model that scores 95% on action recognition can score near chance on counting the same action. The two tasks measure different capabilities, and conflating them hides the failure.
Event bookkeeping is not an academic exercise. It underpins:
In each case, the value comes from accurate temporal tracking, not from describing a single frame.
Humans are near-perfect at event bookkeeping for short videos. We count repetitions, remember order, and notice rare events without effort. This makes human performance a natural upper bound: on the NExT-QA benchmark, humans exceed 90% accuracy on temporal questions, while state-of-the-art VLMs sit around 60–70%. The gap is not a matter of scale. It is a matter of architecture and training.
Key Takeaway: Event bookkeeping — occurrence, count, order, duration — is a temporal reasoning problem that action recognition does not capture. Humans solve it near-perfectly; VLMs do not.
The low frequency trap describes a pattern: when the correct answer depends on an event that is rare in training data, model accuracy collapses. The model does not output "I don't know." It outputs the most statistically likely answer, which is usually the common event.
The trap is not limited to counting. It affects ordering (the model picks the common sequence), occurrence detection (the model misses the rare event), and duration (the model defaults to typical durations).
The low frequency trap is a specific instance of the long-tail problem in machine learning. In any natural distribution, a few classes dominate and a long tail of rare classes trails off. Video datasets are no exception: walking, standing, and sitting appear far more often than jumping, climbing, or falling.
When models are trained with standard objectives, they optimize average loss. Rare events contribute little to the average, so the model learns to predict common events well and rare events poorly. This is rational behavior under the training objective — and catastrophic under deployment conditions where the rare event is the one that matters.
The bias operates at multiple levels:
These priors compound. The result is a model that answers from expectation, not observation.
Consider a dataset where jumping appears in fewer than 5% of videos. A model trained on this data learns that "jumping" is unusual. When asked "How many times does the person jump?" in a video with two jumps, the model has weak evidence for the concept and strong prior toward "zero" or "one." It answers "once." The correct answer — "twice" — requires overriding the prior with visual evidence the model has not learned to trust.
Research on long-tail video understanding shows that when an event appears in less than 5% of training videos, model accuracy drops by up to 40%. This is not a small degradation. It is the difference between a usable system and an unreliable one. And because rare events are precisely the ones that matter in surveillance, safety, and anomaly detection, the drop hits hardest where the stakes are highest.
Key Takeaway: The low frequency trap is the long-tail problem applied to temporal reasoning. When the correct answer depends on a rare event, models default to common events, and accuracy can drop by up to 40%.
The failure was not obvious until benchmarks were designed to measure it. Descriptive captioning benchmarks reward fluency; temporal reasoning benchmarks reward tracking. The gap between the two is the trap.
NExT-QA (Xiao et al., CVPR 2021) was built to move video question answering beyond simple recognition toward explaining temporal actions. It includes questions about causal and temporal relationships. On its temporal subset, state-of-the-art VLMs achieve around 60–70% accuracy, compared to over 90% for humans. The gap is largest on questions that require counting or ordering.
AGQA (Grunde-McLaughlin et al., CVPR 2021) focuses on compositional spatio-temporal reasoning. Its analysis shows that models perform 20–30% worse on questions requiring counting compared to descriptive questions. The benchmark's controlled generation lets researchers isolate the failure: it is not that models cannot see the objects, it is that they cannot track them over time.
STAR (Wu et al., NeurIPS 2021) tests situated reasoning in real-world videos. Accuracy on tasks involving event sequencing is often below 50%. The benchmark's design — combining interaction, sequence, prediction, and feasibility questions — makes clear that the failure is systematic, not a quirk of one question type.
TempCompass (Li et al., arXiv 2024) was built specifically to diagnose temporal reasoning in video LLMs. Models score an average of 45% on temporal ordering tasks. The benchmark separates different temporal skills — action order, direction, speed — and finds that models fail unevenly, with ordering among the weakest.
VITATECS provides fine-grained diagnosis of temporal failures, distinguishing between problems with temporal concepts (like "before" and "after") and problems with temporal perception (actually seeing the order). This distinction matters: some failures are linguistic, others are perceptual, and the fixes differ.
A survey of video-language models (arXiv:2304.09355) found that only 15% of evaluated models could reliably count repeated actions. That is roughly one in seven. The rest either fail outright or produce inconsistent counts across runs.
Key Takeaway: Benchmarks built to test temporal reasoning — NExT-QA, AGQA, STAR, TempCompass, VITATECS — consistently show large gaps between VLM performance and human performance, with counting and ordering among the weakest skills.
Language priors are the associations a model learns from text. They are useful for fluency and dangerous for grounding. When a VLM is asked a question, it can often answer from the question alone, without looking at the video, because the question's wording strongly suggests an answer.
This is not hypothetical. In a cooking video where flour is added after eggs, a VLM asked "What ingredient is added after the eggs?" answers "salt." Salt is the common association with eggs in text. The model retrieves the association rather than reading the sequence.
Many VLMs process video as a bag of frames: they encode each frame independently and pool the representations. This architecture has no mechanism for tracking state across time. Counting requires memory — you must remember previous occurrences. Pooling discards that information.
The result is a model that can describe any frame accurately but cannot answer questions that depend on the relationship between frames. It sees the forest as a collection of trees.
Some architectures add temporal attention or 3D convolutions, but many deployed VLMs do not. Even when temporal layers exist, they are often shallow relative to the spatial and language components. The model's capacity is allocated to recognizing content, not to tracking it.
Counting is a memory task. The model must maintain a running count and update it as events occur. Transformer attention has limited effective memory over long sequences, and the relevant evidence for a count may be spread across hundreds of frames. Without an explicit memory mechanism, the model loses track.
The most misleading aspect of the failure is that descriptive accuracy stays high. A model that cannot count jumps can still produce a beautiful caption of the video. Evaluation pipelines that rely on captioning metrics will report strong performance while the model fails at the tasks users actually need.
Key Takeaway: VLMs fail at event bookkeeping because they rely on language priors and single-frame cues, lack explicit temporal modeling, and have limited memory for tracking state across frames. Descriptive fluency masks the failure.
A surveillance system using a VLM is trained on footage where walking and standing dominate. A person climbing a fence is rare. The model, biased toward common events, fails to flag the climb. The event that matters most is the one the model is least equipped to see.
In soccer analytics, a VLM miscounts passes because it cannot track the ball consistently across frames, especially for infrequent pass types. The count is wrong, the derived statistics are wrong, and the analysis is unreliable.
In human-robot interaction, a robot must understand the order of human actions to respond correctly. If the model misreads the sequence — confusing "pick up then place" with "place then pick up" — the robot acts on the wrong step. The failure is not cosmetic; it is a safety issue.
A VLM shown a magic trick where a ball disappears fails to notice the disappearance. It focuses on the common action — the magician's hand waving — and misses the rare event that defines the trick. This is a clean demonstration of the trap: the model sees what it expects, not what happens.
The pattern across domains is the same. The events that matter most are often the rarest. A system that is accurate on common events and unreliable on rare ones is not a system you can trust in safety-critical settings. The benchmark gap is a proxy for real-world risk.
Key Takeaway: The low frequency trap hits hardest where the stakes are highest. Surveillance, sports analytics, robotics, and safety-critical systems all depend on detecting rare events — the exact events VLMs are most likely to miss.
Temporal attention lets the model attend to relationships across time rather than treating frames independently. Instead of pooling frame features, the model learns which frames are relevant to the current question and how they relate. This is a necessary but not sufficient fix: attention helps, but without memory, counting remains hard.
Memory networks maintain explicit state across time. For counting, the model can store a running count and update it as events occur. For ordering, it can store the sequence of events and query it later. Memory is the missing ingredient in most current architectures.
Contrastive learning trains the model to distinguish rare events from common ones. By constructing positive and negative pairs that differ only in the rare event, the model learns to attend to the distinguishing features rather than the common background.
If rare events are underrepresented, sample them more. Balanced sampling and augmentation can reduce the skew that causes the trap. The risk is overcorrection: too much rare-event data can hurt common-event performance. The goal is calibration, not inversion.
3D convolutions and temporal transformers model time explicitly rather than treating video as a sequence of independent images. These architectures are more expensive but better suited to bookkeeping tasks. The trend in recent research is toward hybrid designs that combine efficient frame encoding with explicit temporal reasoning.
Captioning metrics do not measure bookkeeping. Evaluation must include counting, ordering, and duration questions, with controlled event frequencies. Benchmarks like TempCompass and VITATECS are steps in this direction. The field needs more of them, and it needs them to be standard practice.
Key Takeaway: Mitigation requires architectural change (temporal attention, memory networks), training change (contrastive learning, balanced sampling), and evaluation change (metrics that test temporal understanding). No single fix is sufficient.
From 2021 benchmarks (NExT-QA, AGQA, STAR) to 2024 diagnostics (TempCompass, VITATECS), the field has moved from observing the failure to measuring it precisely. The progress is real but partial: we now know what is broken, not yet how to fix it.
Models like VideoLLaMA and Video-ChatGPT show strong captioning and descriptive performance but weak temporal reasoning. They fail on simple counting questions and misorder events. They are representative of the current state: fluent, useful for some tasks, unreliable for bookkeeping.
No current VLM reliably performs event bookkeeping across event frequencies. The best models narrow the gap on common events and still fail on rare ones. The trap persists.
Progress requires datasets with controlled event frequencies — datasets where researchers can vary how often an event appears and measure the effect. Without this control, the trap is hard to isolate and harder to fix.
The field needs architectures with explicit temporal state, not just temporal attention. Memory, recurrence, and structured representations of events are candidates. The current default — encode frames, pool, attend — is not enough.
Evaluation must go beyond single-frame accuracy and captioning metrics. It must test counting, ordering, and duration under controlled conditions. The benchmarks exist; they need to become standard.
Key Takeaway: The field has diagnosed the low frequency trap but not solved it. Progress requires controlled datasets, temporal architectures, and evaluation that measures bookkeeping, not fluency.
The low frequency trap is not a minor flaw. It is a fundamental limitation of current video language models: they describe what they expect and miss what is rare. The failure shows up in benchmarks, in surveillance, in sports, in robotics, and in any setting where the rare event is the one that matters.
Escaping the trap requires three things working together:
The gap between human and machine event bookkeeping is large, but it is not mysterious. We know why models fail. The work ahead is to build systems that do not.
What is the low frequency trap in video language models? It is a failure mode where VLMs perform poorly on events that occur infrequently in training data, defaulting to common events. When the correct answer depends on a rare event, accuracy drops sharply.
Why do VLMs struggle with counting events in videos? Counting requires maintaining state across frames. Many VLMs process video as independent frames and pool the representations, which discards the temporal information needed to track a running count.
What are some benchmarks for evaluating temporal reasoning in VLMs? NExT-QA, AGQA, STAR, TempCompass, and VITATECS. Each tests different aspects of temporal and causal reasoning, and all show significant gaps between VLM and human performance.
How does the low frequency trap affect real-world applications? In surveillance, it causes missed rare security events. In sports analytics, it miscounts passes. In robotics, it misreads action sequences. In safety-critical systems, it misses anomalies.
Can VLMs be improved to overcome the low frequency trap? Yes, with temporal attention, memory networks, contrastive learning, balanced sampling, and architectures that model time explicitly. No single technique is sufficient; progress requires combining them.
What is the difference between event bookkeeping and action recognition? Action recognition classifies a clip into a category. Event bookkeeping tracks occurrence, count, order, and duration across time. A model can be excellent at the first and poor at the second.
Are humans better than VLMs at event bookkeeping? Yes, by a wide margin. Humans exceed 90% on temporal questions in NExT-QA; state-of-the-art VLMs sit around 60–70%, and lower on counting and ordering.
What causes VLMs to rely on language priors? Training on large text corpora teaches strong word associations. When a question's wording suggests an answer, the model retrieves the association instead of reading the video, especially when the visual evidence is weak or rare.
How can we evaluate if a VLM suffers from the low frequency trap? Test it on counting, ordering, and occurrence questions where the correct answer depends on events with varying training frequency. If accuracy drops as event frequency drops, the trap is present.
Is the low frequency trap unique to video language models? No. It is a specific instance of the long-tail problem, which affects many machine learning systems. VLMs are especially vulnerable because temporal reasoning compounds the difficulty.
Dive deeper into the research: explore the benchmarks, test your own models, and join the effort to build VLMs that truly understand time. The low frequency trap won't escape itself.