A CNN2D-LSTM Framework for Rule-Based Pedestrian-Vehicle Risk Scenario Detection

Abstract

In this work, we developed an approach for rule-based pedestrian-vehicle risk scenario detection from video sequences. The contributions lie in the classification of "risky" and "non-risky" situations automatically derived from behavioral and physical cues, which could improve accident prevention and intelligent driver-assistance systems. A two-dimensional convolutional neural network (CNN2D) is employed over the frames of the videos for visual features extraction, while the LSTM recurrent network models the temporal dynamics of the sequences. The data used in these experiments are sequences of video frames extracted from the JAAD dataset. Behavioral and physical variables include pedestrian crossing, gaze direction, vehicle action, proximity, which are used only for generating the risk labels. They are not provided as explicit input to the model. While these labels are heuristically generated and may not capture all possible risky scenarios, they provide a practical framework for model training and evaluation. The CNN2D extracts the spatial visual features from the frames, while the LSTM captures temporal dependencies, which permits the model to learn both in the spatial and temporal axes for the prediction of risk. Tests on the JAAD dataset composed of varied traffic conditions and images of pedestrian crossings report overall accuracy of 97% and class-wise precision 99.5% for the "No Risk" class and 92.2\% for the "Risk" class. These results confirm the effectiveness of the suggested model and demonstrate the usefulness of fusing visual and temporal information collectively for automatic risk detection in difficult traffic environments.