Multimodal-Few-Shot-Learning-for-Gait-Recognition

This repository is the official implementation of Multimodal-Few-Shot-Learning-for-Gait-Recognition paper, which proposes a system that learns a mapping from a multimodal time series collected using insole to a latent space to address the open set gait recognition problem. The system maps unit steps to embedding vectors using an ensemble consisting of a convolutional neural network and a recurrent neural network. To recognize each individual, the system learns a decision function using a one-class support vector machine from a few embedding vectors (few-shot) of the person in the latent space, then the system determines whether an unknown unit step is recognized as belonging to a known individual or not.

Requirements

Some of the main packages used for this project are Tensorflow-gpu 1.14, Keras 2.2.4, and scikit-learn 0.23.2. It is recommended to create a new environment and install the packages listed in requirements.txt:

pip install -r requirements.txt

Experiments

This repository contains the code to perform experiments with the CNN, RNN, and the Ensemble individually. Each experiment is divided in two steps:

• 1. Train the encoder: this script trains the encoder (CNN, RNN, or Ensemble). It saves the trained encoder and the predicted embeddings for the next step.
• 2. Train and test the classifier: it trains the OSVM classifier with the few-shot learning method and test it with known-unknown-test and unknown-unknown-test sets. It requires the predicted embeddings obtained in the previous step. The results are saved in a CSV file for later plot and analysis.

Datasets

As is shown in the following image, the data was collected from 30 subjects and it was split into three sets:

• Training set: used to train the CNN, RNN, and ensemble models independently. It consists of all the unit steps of 16 individuals selected randomly.
• Unknown-Known test set: it contains the unit steps of 7 individuals selected randomly from the 14 remaining people after selecting the training set. This dataset is divided in two subsets. The first subset consists of 10 unit steps for each individual and it is used for training the OSVM classifier. The second subset is the remaining steps of the same 7 individuals and it is used to test the classifier as known data in the open set gait recognition problem.
• Unknown-unknown test set: it contains all the unit steps of the remaining 7 subjects which were not used in any training process, therefore they are unknown subjects. It is used for testing the classifier as unknown data in the open set gait recognition problem.

Evaluation

The system is evaluated in terms of Accuracy (ACC), True Positive Rate (TPR), and True Negative Rate (TNR) defined as follows:

• 

• 

• 

Where,

• TP stands for True Positive and it is the total unit steps in the known test set that were classified correctly.
• FN stands for False Negative and it is the total unit steps in the known test set that were classified incorrectly.
• TN stands for True Negative and it is the total unit steps in the unknown test set that were classified correctly as an unknown participant.
• FP stands for False Negative and it is the total unit steps in the unknown test set that were classified incorrectly as a known participant.

Results

The following countour plots show the obtained distributions of ACC as a function of γ and ν for the CNN, RNN, and ensemble models respectivelly. A comparison of the area in which the rates are greater than 90% (light green to yellow areas) indicates that the region of the ensemble model is broader than that of the regions of the CNN or RNN model. This means that the ensemble model has a weak dependency when selecting γ and ν, which affects the robustness of the recognition result.

The distribution of the TPR is shown in the following plot. A comparison of the area in which the rates are greater than 93% (yellow), the region of the RNN model is slightly broader than that of the CNN model. The overall distribution of the ensemble model is similar to that of the RNN model.

The distribution of the TNRs is shown below. Contrary to the distributions of the TPR, the overall distribution of the ensemble model is almost identical to the distribution of the CNN model. In particular, a comparison of the area in which the rates are greater than 93% (yellow) reveals that the region of the CNN model is significantly broader than that of the RNN model.

To determine the effect of τ, we specified separate values of γ and ν for the different models in the experiment. We used γ = 1.9 and ν = 0.06 for the ensemble model, γ = 1.8 and ν = 0.06 for the CNN model, and γ = 2.2 and ν = 0.08 for the RNN model. In the figure, we see that choosing a τ value smaller than 0 significantly improves the TPR and ACC.

Contributors

Nelson Minaya [email protected]
Nhat Le [email protected]