To summarize, the overall goal of this research is the development and construction of complete technical solutions to make the mentioned aspects measurable.
2. Means and Methods
To record the hazard perception and responsiveness of drivers, as many parameters as possible of the recognition and reaction process according to ÖNORM V 5050 must be recorded. It must be known what the drivers see or perceive and how they react to it. What drivers see or recognize can be recorded from the driver’s perspective using eye-tracking videos or photos. The response can be detected by a change in vehicle operation. For example, a change in the position of the accelerator, brake pedal, or clutch pedal, or a change in the steering wheel angle. In modern vehicles, all pedal positions and driving data are transmitted unencrypted on the CAN bus. Rather, vehicle manufacturers are pursuing the “security through obscurity” approach by using proprietary data formats. When reading out the communication taking place on a specific CAN bus, it remains completely unclear which byte in which identifier transmits the searched signal or which unit it has. Even if there is some research work that deals with machine-learning-based (partially) automated reverse engineering, for example, there is often no other option for manual reverse engineering, even if this process turns out to be laborious and time-consuming.
3.1. CAN Bus Reverse Engineering
The CAN bus is also relevant for vehicle safety and improper handling of the vehicle electronics would lead to a loss of warranty. Therefore, safety was of great importance in this work. To rule out any negative interference with the CAN bus, an inductive CAN bus coupler was used to pick up the signals without contact and therefore with no ability to send messages to the bus. Furthermore, the CAN controllers used are operated exclusively in listen-only mode. This would also ensure that no signals are sent even if there is a galvanic connection between the system and the CAN bus.
3.2. Developing an Eye-Tracking-Based System
3.2.1. Eye-Tracking System (System 1)
3.2.2. Data Logging
The DL1 Pro data logger from Race Technology was chosen to record and store driving and vehicle data. This consists of an aluminum housing and can be supplied via the 12-volt vehicle on-board voltage. It also has a connection for a GPS antenna that records GPS data at 20 Hz, an integrated acceleration sensor, two CAN bus ports with 105 channels each, eight analog inputs for a detection range of 0–25 V with a 12 bits analog-to-digital converter, and a sampling rate of 1000 Hz, as well as several other functions. The recorded data are stored in the form of a RUN file on a maximum 32 GB SD card, which can be inserted into the integrated SD card slot on the front. The recorded data can either be read from the SD card using a card reader or directly from the DL1 using a USB cable. The RUN file is then evaluated in the supplied “RTAnalysis” software (version 8.5.369). The data logger and the data to be recorded or CAN IDs are configured from the PC using a USB cable and the supplied software. Appropriate cables with open cable ends are supplied for the rear connections. A separate input–output board was developed to connect all the required signals from and to the data logger. To be able to operate the data logger from the driver’s or front passenger’s seat, a wired remote control was also developed. Finally, the fully assembled measuring device was positioned behind the passenger seat in the Audi A3 and all components were connected.
3.2.3. Overlay with the Eye-Tracking Video
3.2.4. Use in Practical Driver Training
In the next step, the described system installed in the Audi A3 was practically tested in an Austrian driving school. In Austria, it is mandatory for learner drivers to carry out a so-called multi-phase drive with a specially trained driving instructor about 2–4 months or 6–12 months after passing the practical driving test, depending on the driving license class they have completed. In these units, among other things, the topic of gaze technique is a focus. Since all learner drivers in the multi-phase training already have driving experience and are very familiar with the handling of vehicles, this group was selected as the subject of the study. In total, lessons with 25 drivers were recorded and 50 situations were analyzed in detail. Due to the well-known problems of eye-tracking glasses with made-up eyes, only male drivers were analyzed. The average age was 18 years. The test subjects had held a driver’s license for an average of 10.2 months and had driven an average of 15,100 km to date.
Before the drive was carried out, each candidate was asked to assess their driving skills themselves with regard to their hazard perception skill on a scale of 1–10. The average self-assessment of the 25 candidates was 8 out of 10 points. After watching the recorded videos following the driving lesson, every tested driver admitted that there was at least one situation where they were surprised or shocked by their own gaze behavior or by the fact that a prospective hazardous situation was not recognized properly. Faced with these videos, the tested drivers downscaled the self-assessment of their hazard perception abilities from 8/10 to 6/10 points on average, which provides a deterioration of 25%.
The four driving instructors involved in the evaluation period all agreed that the time-consuming process of calibration and evaluation represents a serious obstacle to the accompanying use in driver training. For this reason, the use of System 1 was limited to the second phase of the multi-phase drive.
3.2.5. Conclusion—System 1
A disadvantage of the overall developed solution is the high unit costs of approx. EUR 12,000 net per vehicle without installation. Furthermore, the generation and evaluation of the video material is complex, time-consuming, and requires good computer skills. Furthermore, the adjustment or calibration of the eye-tracking glasses requires a certain lead time and does not always work reliably for some people (e.g., with made-up eyes). All in all, it must be assumed that a person with very good computer skills and a high-performance PC would need around three to four times the time of the measurement run just for the evaluation and generation of the finished material.
3.3. Developing an App-Based System (System 2)
According to the experience gained so far, an ideal technical solution for the cost-effective and comprehensive training and testing of hazard perception skills and road and infrastructure safety must be as cheap as possible to purchase. Additionally, a quick and uncomplicated operation in everyday life or real traffic, even by people with little IT knowledge, must be guaranteed. The preparation and follow-up time must be negligible.
To meet the requirements of real traffic, as with the previous approach, only a combination of image material and vehicle data (pedal positions and speed) can be considered. Without image material, it would not be known what the driver sees or which traffic situation they are in, and without driving data it would not be known how or at what speed the driver approaches a situation or traffic scene. The measurement of response times on the subject of hazard perception serves to compare the abilities with those of other test subjects. However, since every situation is unique in real time traffic anyway and therefore there is no comparability of students or situations, this functionality can be omitted in favor of the goals defined here.
For the basic evaluation of road and infrastructure safety or hazard perception, however, it ultimately plays a subordinate role where the exact error that happened inside the black box was a driving error or a non-adapted behavior can ultimately still be pointed out.
The current generation of tablets and smartphones already have a powerful processor, a GPS receiver, a high-resolution camera, and a three-axis acceleration sensor. This means that these functions can already be used for the system setup without having to purchase a GPS receiver or camera. Furthermore, these devices usually also have enough storage space, which makes an additional data logger superfluous. As a result, and by omitting eye tracking, the costs can already be drastically reduced.
3.3.1. Hardware Setup
Due to the optimization carried out during the development phase in the direction of sustainability, the power consumption of the finished prototype with full BLE visibility is only 135 microamperes. In this state, the circuit could theoretically remain connected to a car battery with 75 ampere hours used in small cars for approx. 63 full years until it was completely discharged. This energy-saving mode makes it possible to operate the circuit on the vehicle’s permanent plus. A cumbersome switching on and off before and after each ride or an operation limited to the ignition plus of vehicles is no longer necessary. In addition, a microSD card reader was integrated into the circuit to write the received data onto an SD card if desired.
3.3.2. Software Setup
In addition, software is required that at least displays and evaluates the CAN bus data transmitted via BLE and can link this with image material. For this purpose, an Android app was developed using the free development environment Android Studio, which includes the desired range of functions as well as the additional functions described below. The programming language Kotlin was used for programming. Kotlin has been Google’s official preferred programming language for Android apps since May 2019. [HEI21].
To take pictures, the supervisor can either hold the tablet in their hands and choose any perspective or attach the tablet to a magnetic holder on the dashboard, for example. The trigger can be pressed on the tablet or an extra trigger the size of a button cell connected to the tablet via Bluetooth can be used to be able to react more quickly to interesting situations and not to distract the driver.
3.3.3. Picture Overlay
3.3.4. Creation of Road and Infrastructure Safety Maps
3.3.5. Use in Practical Driver Training
System 2 was evaluated for 3 months in 4 driving school cars in Austria in selected driving lessons. A total of 114 driving hours were accompanied and recorded. All learner drivers who completed at least one driving lesson with the system were asked about the added value of accompanied driving lessons in an anonymous questionnaire after completing their training. In total, 97% of the students stated that the images created represent added value in their driver training compared to driving lessons that are not accompanied by the system. A total of 84% stated that the lessons learned from the pictures taken had a more lasting impact on driving behavior than purely verbal feedback. Furthermore, they agreed that the quality of the debriefing was significantly better through the use of the system without costing the student valuable driving time. The 4 driving instructors involved in the evaluation period all agreed that the demonstrative character of the pictures taken during the debriefing led to significantly fewer discussions with the students, since the students no longer had the opportunity to dispute their (wrong) behavior.
By using the map view, it was also possible to identify some road areas where learner drivers had problems particularly often or where the traffic signs installed obstruct the view of the approaching traffic. This information is also of great importance with regard to sustainable road design. Since then, the map view and the recordings made have also been used in the theory course in order to learn from the mistakes of others at an early stage.
3.3.6. Conclusion—System 2
The developed System 1 has the great advantage that the complete process of visual information acquisition can be analyzed and recorded down to the last detail. This makes it possible to record whether a driver focuses on a possible danger at all or not, and in what exact period they react to it. This makes it possible to analyze chains of behavior in complex traffic situations with an accuracy of fractions of a second and to give the supervisor or driver undreamt-of insights into traffic perception and their own traffic characteristics. However, the price of this powerful tool is very high in terms of acquisition and time required for evaluation and presentation. The temporal component could be countered by integrating the driving data read from the CAN bus directly into the eye-tracking video via an interface at runtime. This would make the laborious synchronization and overlaying of the two data streams, which were previously only recorded separately, superfluous. However, the high price of an eye-tracking system remains a stumbling block for widespread use.
In summary, the development of System 2 reverses the advantages and disadvantages of System 1. The omission of eye-tracking lowers the costs, but also the possibility to dive deep into the process of visual information acquisition. The use of images instead of videos, the automatic time-synchronous superimposition, and the elimination of calibration reduces the time required to accompany the process many times over. The use of an Android app also simplifies operation considerably and eliminates the need for evaluation on a desktop PC. As a result, far less IT knowledge is required for operation, which makes System 2 ideal for use in the fields of infrastructure and road safety or driver training across the board and at low cost. In addition, the generated image material can also be displayed automatically online in a map view used for further investigation of road safety aspects such as identifying accident blackspots or confusing road design. The possibility of combining the advantages of both systems without merging the disadvantages would only be possible through the development of an eye-tracking system which is able to record the external data stream of the vehicle data as an input signal and to process it synchronously without any further action.
If it is of great importance for an application to depict the complete perception process instead of a snapshot, and the dynamic creation of this moment is also of great importance, then the disadvantages of System 1 must be accepted. In summary, based on the experience gained so far, it can be recommended that System 2 be used frequently due to the simpler operation and lower costs to identify frequent sources of error or dangerous road sections based on the frequency of recordings made in those locations. With the experience gained in this way, these errors or situations can then be subjected to a more in-depth analysis with System 1.
As part of this research, both systems were successfully tested in the following vehicles: Audi A3 8V, Audi A3 8Y, Audi A6 4G, BMW 3 Series G21, Mercedes A W177, Mercedes GLA X156, VW Golf 2021, VW T6 2018, VW T-Cross 2019, and VW T Roc 2021. In principle, however, both systems can be used in all vehicles in which the relevant driving data is available on the CAN bus.
For further research work, the development of algorithms has been planned that, based on the recorded driving data, are able to automatically assess the driver with regard to certain characteristics such as anticipatory driving and economical driving style, but also with regard to their ability of hazard perception. By integrating the map display, the possibility of automatic detection of hazard accumulation points and problem areas in road design should also be examined. The cost-effective and comprehensive use of System 2 could make a valuable contribution to sustainable driver training and traffic management.