Why we need AI to understand what the eyes can’t see.
Every day, drivers navigate traffic with their eyes open – but are they truly aware of what’s around them? Cognitive distraction while driving is one of the most underestimated risks in road safety. This is why EuroNCAP is calling industry and researchers to develop effective sensing technology to introduce cognitive distraction detection in all series production vehicles by 2029. While physical distractions like texting get a lot of attention, the invisible pull of our thoughts can be just as dangerous. A driver may look straight ahead, see a pedestrian, and still not react. Why? Because their mind is elsewhere.
What is cognitive distraction?
Cognitive distraction inside the car occurs when a driver’s attention is diverted from the driving task to internal thoughts. Unlike manual or visual distractions, cognitive distraction doesn’t involve hands leaving the wheel or eyes leaving the road – it involves the mind drifting away.
Psychologists like David Strayer (University of Utah) and Natasha Merat (University of Leeds) have studied cognitive distraction extensively. Strayer’s research shows that even hands-free phone conversations can impair reaction times and hazard perception significantly. Merat’s simulator studies highlight that cognitively distracted drivers often fail to respond to critical events, even if they are looking directly at them. In fact, such “looked-but-failed-to-see” errors are among the most common causes of accidents at intersections. Dr. Diederichs, also psychologist from the department Human-AI-Interaction at Fraunhofer IOSB, developed a scale to rate cognitive distraction based on the content of conversations on the with other passengers, on the phone, or with chatbot. The more a conversion requires System 2 thinking, the higher the cognitive distraction from the driving task. System 2 thinking is the slower but conscious thinking in a cognitive model introduced by David Kahnemann in his famous book “Thinking, fast and slow”. The so called BABS-scale was originally used in a PhD thesis sponsored by Porsche, to rate drivers attentive involvement in a sportive driving task.
Current state of research
Over the last two decades, cognitive distraction has been measured in lab and simulator studies using a combination of:
- Self-rating scales such as NASA-TLX questionnaire.
- Secondary task performance: Drivers are asked to perform mental tasks (e.g., mental arithmetics, n-back-tasks or phone conversations) while driving.
- Reaction time tests: Drivers’ response times to sudden events have been measured.
- Eye tracking: Patterns like prolonged fixations, reduced saccades or less special variance are interpreted as signs of reduced situational awareness.
- Physiological measurements: Missing P300 in EEG measures, heart rate variability indicators and pupil dilation to measure cognitive workload.
These methods are effective in controlled environments – but implementing them in everyday driving, especially in series production vehicles, remains a challenge. Some are invasive and some are too slow for realtime detection. There is currently no robust, non-intrusive way to assess cognitive distraction in real-world driving.
Enter SensAI: A digital twin for your cognitive state
This is where the Fraunhofer approach SensAI enters the scene. SensAI aims to develop a multimodal AI model that can estimate the driver’s cognitive state using:
- Computer vision: Detecting gaze behavior and body pose, as well as activities and intentions (a Fraunhofer IOSB technology).
- Audio analysis: Interpreting speech content, voice pitch, tempo, and hesitation patterns in speech (a Fraunhofer IDMT technology).
- Physiological data: Processing camera image signals to derive breathing patterns and pulse (a Fraunhofer IMS technology).
By combining these signals, SensAI creates a cognitive digital twin of the driver – a real-time model of their mental engagement. This twin can detect when the driver is mentally overloaded, distracted, or disengaged – even if they seem alert from the outside.
From science to safety: Preventing accidents before they happen
Imagine a situation at a busy intersection. A cyclist approaches from the side. The driver looks in that direction – but doesn’t see the cyclist. A second later, a collision.
If the vehicle had access to the driver’s cognitive digital twin, it could have recognized the inattentive state – a blank stare, reduced eye movement, slow speech patterns – and issued a prompt: a gentle auditory alert or even a braking intervention.
This is not science fiction, but rather AI rocket science: A combination of psychology and technology from the Fraunhofer Labs.
Cognitive distraction is invisible – but not undetectable. With multimodal AI, we can close the gap between lab science and series production sensing. Projects like SensAI point the way to a future where vehicles support drivers not just with sensors and algorithms, but with real empathy for the human mind.