We have been working a lot with 3D information for activity recognition and also for assistance of blind persons.
Question: What kinds of applications for in-cabin cameras have you been working on?
Prof. Stiefelhagen: I have been working together with PhD students on different use cases for in-cabin monitoring. A main focus was activity recognition of all passengers, including the driver. Knowing the historical and current activities allows human machine interactions that respect the context of the user. Automated cars also promise important mobility improvements for blind passengers. Blind persons can be guided to find objects, doors, seatbelts and can be informed about the general situation by camera-based assistance systems – also in cars.
Question: Which cameras are used in these applications?
Prof. Stiefelhagen: We have been working a lot with 3D information for activity recognition and also for assistance of blind persons. A 3D representation of the scene provides independence from changing camera positions. It also measures distance and hence provides a more accurate positioning of body parts and objects. 2D information can only guess distances which means extra efforts. However, it is very good to classify surface types and objects.
We used Time-of-Flight cameras and multi-view cameras in a setup of 3 or more mono cameras. Time-of-Flight cameras require less installation and calibration efforts, while multi-camera setups cover larger areas and suffer less from occlusions. I think the detection area inside the jagged car interior is of very high relevance to get a complete understanding of the situation. E.g. children or objects in the footwell cannot be seen from a camera in the windshield. Of course, it depends on the use case – but my guess is that 3D information and a large coverage is a main feature for in-cabin monitoring. This can be achieved by multi-view systems. Furthermore, they provide additional RGB or FIR signals.
Question: What trends do you see from your research for in-cabin monitoring systems?
Prof. Stiefelhagen: Current developments of generative artificial intelligence with powerful foundation models open a large space for new approaches. Models should be able to handle different but modular sensor input for a holistic representation. Sensors should learn from each other. For this reason, we started a new PhD to investigate multimodal representation learning for in-cabin semantic situation analysis.
Thank you, Prof. Stiefelhagen for your insights!
About Prof. Stiefelhagen
Prof. Dr. Rainer Stiefelhagen holds the professorship “Computer Science Systems for Visually Impaired Students“ at the Faculty of Computer Science at the Karlsruhe Institute of Technology (KIT). He heads the Computer Vision for Human-Computer Interaction Lab at the KIT’s Institute of Anthropomatics and Robotics, as well as the KIT’s Center for Digital Accessibility and Assistive Technology – ACCESS@KIT. He received his doctorate in computer science from the University of Karlsruhe (TH) in 2002 and habilitated in 2009. His research interests include image and video understanding, multimodal interfaces, applications in medical image analysis, assistive technology for seeing impaired users, driver assistance, robotics and surveillance.
More Information
Fraunhofer IOSB is a key player in advancing research for Occupant Monitoring Systems. We focus on recognizing all passengers within the car and classify up to 35 different activities in real time. Read more about IOSB’s Advanced Occupant Monitoring System here, if you want to find out, how you can benefit from it today!