KiD3-Knowledge-Infused-Distracted-Driving-Detection

This repository implements KiD3 introduced in the paper Towards Infusing Auxiliary Knowledge for Distracted Driver Detection which was accepted by KiL'24: Workshop on Knowledge-infused Learning co-located with 30th ACM KDD Conference, August 26, 2024, Barcelona, Spain

Introduction

This repository presents the implementation of KiD3, a novel method for distracted driver detection (DDD) by infusing auxiliary knowledge about semantic relations between entities in a scene and the structural configuration of the driver's pose. Specifically, a unified framework that integrates the scene graphs, and driver's pose information with the visual cues in video frames to create a holistic representation of the driver's actions.

Getting Started

Prerequisites

Install the Python dependencies (Python 3.9+ recommended):

pip install -r requirements.txt

This project also builds on two external repositories:

1. Clone the RelTR repository into the RelTR folder of this repository and replace the inference.py file with the one provided in the RelTR folder of this repository. Make sure to follow the installation instructions provided in the RelTR repository (including downloading the pre-trained models).

2. Similarly, clone the OpenPose repository into the Openpose folder of this repository and add the openpose.py file provided in the Openpose folder of this repository. Make sure to follow the installation instructions provided in the OpenPose repository (including downloading the pre-trained models).

Data Preparation

1. Download the SynDD1 dataset by filling up the form here and annotate every frame with the corresponding driver's action label and save the frames in '{user_id}_{minute}_{second}_{label}.jpg' format. Do this for all videos in the dataset and save them in the Datasets/annotatedvideosv1/AnnotatedVideos/{video_name} folder.

2. Extract the scene graphs for each frame by running Preprocessing/scene_graph.py. Running this will create a sceneGraphs.json file in each of the Datasets/annotatedvideosv1/AnnotatedVideos/{video_name} folders corresponding to each video. Make sure you have a GPU on your device, else this won't run.

3. Steps to extract pose information:

   1. Download the body_pose_model.pth and hand_pose_model.pth files from this Google Drive Link as mentioned in the readme of the OpenPose repository and put it in Openpose/model directory.
   2. Extract the driver's pose information for each frame by running Openpose/openpose.py. This creates the pose directory with {video_name}.json files for each video.

4. Extract the object features for each frame by running Preprocessing/yolo_objects.py. This creates the objects directory with {video_name}.json files for each video.

5. Construct the peripheral information for each frame by running Preprocessing/distances.py

6. Get the image embeddings for each frame by running Preprocessing/image_embeddings.py. This script loads a VGG-16 network whose classifier head is replaced with an 18-way linear layer and fine-tuned on the driver-action classes, expected as model.pth in the repository root. Place this fine-tuned checkpoint at the repo root before running. This creates an imageEmbeddings.json file in Datasets/annotatedvideosv1/AnnotatedVideos/{video_name} directories for each video.

7. Run Data Preparation/handle_peripheral_inputs.py. This will store the combined peripheral inputs in results/ALL.json.

8. Run Data Preparation/copy_frames_and_combine_jsons.py. This will store all the combined scene graph and image embeddings in Datasets/annotatedvideosv1/AnnotatedVideos/ALL/frames.

9. Run Data Preparation/split_data.py to create the train, eval and test split.

Training and Evaluation

Run experiments by running experiment.py. The results will be displayed in the console.

Citation

If you find this repository useful in your research, please consider citing the following paper:

@inproceedings{kiD3,
  title={Towards Infusing Auxiliary Knowledge for Distracted Driver Detection},
  author={Ishwar B Balappanawar and Ashmit Chamoli and Ruwan Wickramarachchi and Aditya Mishra and Ponnurangam Kumaraguru and Amit P. Sheth},
  booktitle={KiL'24: Workshop on Knowledge-infused Learning co-located with 30th ACM KDD Conference},
  year={2024},
  organization={ACM},
  url={https://openreview.net/forum?id=WQdNGavZu3}
}