Similar to LLIO repository, the following installations are required to run the scripts and reproduce the results presented here and in the paper. Given scripts for creating the pyshoe virtual environment have worked successfully on various Windows systems, although we cannot guarantee compatibility with all system configurations.
Note: The Anaconda virtual environment is named pyshoe because initially the research was completely based on Brandon Wagstaff's PyShoe study and public dataset. Despite the transition to using exclusively self-collected data and building own inertial odometry public dataset, the original name was retained.
After installing Anaconda, launch Anaconda PowerShell and then type
conda create --name pyshoe python=3.7
to create pyshoe virtual environment (venv). Subsequently, type
conda activate pyshoe
to activate pyshoe venv.
Type and enter the following commands in Anaconda PS terminal to install the required packages and libraries to run PyShoe codes and reproduce the results in the page and the paper. We thank Dr. Ramazan Özgür Doğan for the assistance in setting up the environment.
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
Continue and complete the installation by entering the following commands in Anaconda PS in pyshoe virtual environment.
conda install scikit-learn==0.19.1
conda install matplotlib
pip install pandas==1.1.5
After cloning this repository to your local computer, you must install liegroups package if you would like to reproduce the results shown here in this repo or the paper.
If one fails or has troublesome experience in setting up the virtual environment pyshoe, the video in pyshoe venv setting will soon be available.
We aim to form a gait-driven system (in other words a stride & heading system - SHS) from VICON room experiments of PyShoe dataset where VICON room experiments are sample-wise annotated pedestrian trajectories. Stride indexes are also required in training a stride-wise INS.
Here, some troublesome experiments are shown to understand Zero Velocity (ZV) interval and stride detection problems. The optimal ZV detectors are selected (e.g., SHOE for experiment 4, ARED for experiment 6) with the corresponding optimal threshold values (optimal values are supplied by Wagstaff et. al. in the structure of the mat files for VICON room experiments of PyShoe dataset) for all VICON room experiments.
The process depicted below is the ZV interval & stride index annotation (or correction) of some troublesome trajectories produced in VICON room experiments (PyShoe dataset). Eventually, ZV interval and stride index detection errors in VICON data are fixed. Extracted dataset is going to be used in training Loose Learned Inertial Odometry (LLIO), a gait & data driven INS. Some researchers call a data-driven (or modern) INS as learned inertial odometry.
Please use detect_missed_strides.m located at data/vicon/processed if you like to reproduce the figures related to ZV interval annotation & correction.
When carefully tracked starting from the initial stride, one can see that the 10th stride is not detected in the trajectory plot shown below. To detect the missed ZV interval(s), supplementary ZUPT detectors such as VICON, ARED, MBGTD or AMVD can be utilized. In general, VICON detector was able to generate ZV labels correctly; therefore, in many cases, only VICON ZUPT detector is used as the supplementary detector.
Integration of filtered optimal ZUPT detector SHOE with the filtered supplementary ZUPT detector (i.e., VICON) enabled successful detection of the ZV interval as shown in the combined ZUPT detector plot above (located at the bottom). The corrected ground-truth data (as a sample-wise and a stride & heading system trajectory) and ZV signals can be seen below. Note that the annotation is only going to be used in extracting x-y axes displacement (or displacement and heading change) values for LLIO training dataset generation; therefore, corrected ZV labels are not used in any trajectory generation.
We see that the 9th stride is not detected in the plots below.
Just like we did to compensate for the errors ZUPT phase and stride detection in experiment 4, here VICON ZUPT detector is selected again as the supplementary detector to correctly detect the missed stride.
Integration of filtered optimal ZUPT detector SHOE with the supplementary ZUPT detector (i.e., filtered VICON) enabled successfull detection of the missed stride as shown in the combined ZUPT detector plot above (located at the bottom). The corrected stride & heading system trajectory and ZV labels can be seen below for the experiment 6.
To see the correction by the supplementary ZUPT detector, check the gif file inserted below.
We see that the 7th stride is not detected in the plots below.
Just like we did to compensate for the errors in ZUPT phase and stride detection in experiments 4 and 6, here VICON ZUPT detector is selected again as the supplementary detector to correctly detect the missed stride.
Integration of filtered optimal ZUPT detector SHOE with the supplementary ZUPT detector (i.e., filtered VICON) enabled successfull detection of the missed stride as shown in the combined ZUPT detector plot above (located at the bottom). The corrected stride & heading system trajectory and ZV labels can be seen below.
To see the correction by the supplementary ZUPT detector, check the gif file inserted below.
We see that the 7th stride is not detected in the plots below. Notice that this is the example experiment that is shown in the paper.
Just like we did to compensate for the errors in ZV interval and stride index detection in experiments 4, 6, and 11, here VICON ZUPT detector is selected again as the supplementary detector to correctly include the missed stride to the stride & heading system trajectory.
Integration of filtered optimal ZUPT detector SHOE with the supplementary ZUPT detector (i.e., filtered VICON) enabled successfull detection of the missed stride as shown in the combined ZUPT detector plot above (located at the bottom). The corrected stride & heading system trajectory and ZV labels can be seen below.
To see the correction by the supplementary ZUPT detector, check the gif file inserted below.
We see that the strides {2, 10} are not detected in the plots below.
Unlike experiments {4, 6, 11, 18, 27}, here SHOE ZUPT detector is selected as the supplementary detector to correctly detect the missed strides.
Integration of filtered optimal ZUPT detector VICON with the supplementary ZUPT detector (i.e., filtered SHOE) enabled successfull detection of the missed stride as shown in the combined ZUPT detector plot above (located at the bottom). The corrected stride & heading system trajectory and ZV labels can be seen below for the experiment 30.
To see the correction by the supplementary ZUPT detector, check the gif file inserted below.
We see that the strides {9, 11, 20} are not detected in the plots below.
Unlike experiments {4, 6, 11, 18, 27, 30}, here supplementary detectors were not able to detect all missed strides. While first two was recovered by VICON ZV detector, the last stride needed to be introduced via manual annotation as can be seen below.
Integration of filtered optimal ZUPT detector SHOE with the supplementary ZUPT detector (i.e., filtered VICON) and the MANUAL ANNOTATION enabled successfull detection of all missed strides as shown in the combined ZUPT detector plot above (located at the bottom). The corrected stride & heading system trajectory and ZV labels can be seen below for the experiment 32.
To see the correction by the supplementary ZUPT detector, check the gif file inserted below.
We see that the 7th stride is not detected in the plots below.
Just like the 4th experiment, here the supplementary detector is selected as VICON, which was able to recover the missed stride.
Integration of filtered optimal ZUPT detector SHOE with the supplementary ZUPT detector (i.e., filtered VICON) enabled successfull detection of the missed stride as shown in the combined ZUPT detector plot above (located at the bottom). The corrected stride & heading system trajectory and ZV labels can be seen below for the experiment 36.
To see the correction by the supplementary ZUPT detector, check the gif file inserted below.
We see that the strides {3, 27, 33} are not detected in the plots below.
The supplementary detector is selected as VICON, which was able to recover the missed strides all.
Integration of filtered optimal ZUPT detector SHOE with the supplementary ZUPT detector (i.e., filtered VICON) enabled successfull detection of the missed strides as shown in the combined ZUPT detector plot above (located at the bottom). The corrected stride & heading system trajectory and ZV labels can be seen below for the experiment 38.
To see the correction by the supplementary ZUPT detector, check the gif file inserted below.
We see that the strides {3, 14, 16} are not detected in the plots below.
The supplementary detector is selected as VICON, which was able to recover the missed strides all.
Integration of filtered optimal ZUPT detector SHOE with the supplementary ZUPT detector (i.e., filtered VICON) enabled successfull detection of the missed strides as shown in the combined ZUPT detector plot above (located at the bottom). The corrected stride & heading system trajectory and ZV labels can be seen below for the experiment 43.
To see the correction by the supplementary ZUPT detector, check the gif file inserted below.
Thus far, due to some undetected steps in VICON room experiments data (recall that Wagstaff et. al. conducted crawling motion experiments in PyShoe) and self-collected data, we examined 56 experiments coarsely in the training dataset (i) to correct for undetected steps (they are classified as 0 in ZV signal plot despite them actually being 1, i.e., it is false-negative) and (ii) to exclude motions like crawling, which are not of type bipedal locomotion. As can be seen above, experiments {4, 6, 11, 18, 30, 32, 36, 38, 43} are corrected with manual ZV interval and stride index annotations by utilizing supplementary detectors. In plot_vicon_data.py file, right before processing experimental data in a loop, annotated experiments are tagged as -1 as follows:
training_data_tag = [1, 1, 1, -1, 1, -1, 1, 1, 1, 1, -1, 1, 0, 1, 1, 1, 1, -1, 1, 1,
1, 1, 1, 1, 1, 1, 0, 1, 1, -1, 1, -1, 1, 1, 1, -1, 1, -1, 1, 1,
1, 1, -1, 1, 1, 1, 0, 0, -1, 0, 1, 1, 1, 1, 0, 1]
where
len(training_data_tag) = 56
In the signal processing main loop (where the loop visits each experiment one by one), experiments are accepted as training data for LLIO according to the absolute values of given tags. In other words, experiments labeled as 1's directly satisfy conditions while the ones labeled as -1s needed corrections on ZV intervals and stride indexes to be included in LLIO training dataset. On the other hand, 0s stand for eliminated experiments due to being nonbipedal locomotion data or unrecoverable errors in ZV intervals and/or stride indexes (we actually emailed Brandon Wagstaff about motion types of experiments yet no documentation was made regarding the motion types (e.g., walking, running, crawling) in the experiments at the time of PyShoe dataset generation). In the decision of labeling an experiment as 0, trajectory (spatial) plots along with IMU data (time series) plots with stride indexes are coarsely examined by the second author of the paper by using MATLAB and Python environments. Eventually, the elimination of VICON room data yielded shrink in the traveled distance by the pedestrian, which negatively affcets LLIO training to produce a data-driven INS. Therefore, additional training data was required in LLIO training. The next section describes dataset expansion/enlargement process.
Here robust (pre-trained LSTM based ZV detector) pedestrian INS is applied on our own-collected data where our sensor is 3DM-GX5-25 and sensor data capture software is SensorConnect.
| 3DM-GX5-25 Sensor | SensorConnect software screen |
|---|---|
 |
 |
Experiments conducted here are manually annotated by using a ruler (see the paper for more details). As the VICON room walks/experiments of PyShoe dataset is very different than hallway traversals, proposed LLIO system required a bigger dataset that accounts for straight walk gait characteristics (at various walking paces). Additionally, faster motion experiments are conducted to make the dataset diverse.
Similar to the notation used for VICON room experiments, the start point is called as stride #0, i.e., initial stride. If plot_vicon_data.py is checked, one can see that ZV labels are filtered for accurate stride index detection. However the filtered ZV values are not used in the trajectory generation. In other words, the pedestrian trajectories are obtained with the raw (not filtered) LSTM based PyShoe generated ZV labels while the strides that are visualized on the trajectories with x correspond to the last index of the ZV intervals of the filtered ZV signals (i.e., stride index).
As all strides produced a foot-print on the ground thar are later measured to form ground-truth data for the experiments, each stride location is called a Ground Control Point (GCP).
Note: One can run plot_own_sensor_data.py to obtain the same results shown below. To learn more about experiment info and results, one can view output.txt log file (located at results/figs/own) recorded while the code was running.
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
Here, the motion speed varies during the experiment as can be seen in the video. PyShoe (LSTM based ZUPT aided ESKF) is able to detect only 22 of 24 strides as can be seen below.
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
Here, the motion speed varies during the experiment as can be seen in the video. PyShoe (LSTM based ZUPT aided ESKF) is able to detect only 23 of 28 strides as can be seen below.
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
Here, the motion speed is slow and constant during the experiment as can be seen in the video. PyShoe (LSTM based ZUPT aided ESKF) is able to detect all 58 strides successfully. Stride#17 annotation is slightly corrected after examination of detected stride indexes on IMU data, i.e., the magnitudes of acceleration and angular velocity vectors.
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
Here, the pedestrian speed is normal (walking style) during the experiment. The video of this experiment was recorded but due to a computer SSD hard disk failure, unfortunately it could not be retrieved. Stride #{33, 34, 41, 43, 60} annotations are slightly corrected after coarse examination of detected stride indexes on IMU data, i.e., the magnitudes of acceleration and angular velocity vectors.
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
Here, the pedestrian motion becomes extreme in some moments. Therefore, PyShoe LSTM was able to detect 43/60 ZV intervals (and strides) in the trajectory. Missing 17 strides are manually annotated after careful examination of IMU data. Also two stride indexes (i.e., stride 45 and 59) are manually corrected to form the training data for LLIO.
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
| Stride Indexes | Trajectory (INS) |
|---|---|
 |
 |
| Zero Velocity | Trajectory (SHS) |
|---|---|
 |
 |
[1] X. Liu, N. Li and Y. Zhang, "A Novel Adaptive Zero Velocity Detection Algorithm Based on Improved General Likelihood Ratio Test Detector," in IEEE Sensors Journal, vol. 22, no. 24, pp. 24479-24492, 2022.
[2] B. Wagstaff, V. Peretroukhin and J. Kelly, "Improving foot-mounted inertial navigation through real-time motion classification," in 2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Sapporo, Japan, 2017, pp. 1-8.
[3] E. Sangenis, C. -S. Jao and A. M. Shkel, "SVM-based Motion Classification Using Foot-mounted IMU for ZUPT-aided INS," in 2022 IEEE Sensors, Dallas, TX, USA, 2022, pp. 1-4.
[4] Y. Wang and A. M. Shkel, "Adaptive Threshold for Zero-Velocity Detector in ZUPT-Aided Pedestrian Inertial Navigation," in IEEE Sensors Letters, vol. 3, no. 11, pp. 1-4, 2019.
[5] Y. Wang and A. M. Shkel, "A Review on ZUPT-Aided Pedestrian Inertial Navigation," in 2020 27th Saint Petersburg International Conference on Integrated Navigation Systems (ICINS), St. Petersburg, Russia, 2020.
[6] J. Wahlström, I. Skog, F. Gustafsson, A. Markham and N. Trigoni, "Zero-Velocity Detection—A Bayesian Approach to Adaptive Thresholding," in IEEE Sensors Letters, vol. 3, no. 6, pp. 1-4, June 2019.
[7] Z. Meng, Z. Deng, P. Zhang and Z. Li, "Adaptive Mid-Stance Phase Observer-Aided Pedestrian Inertial Navigation System for Varying Gait Speeds," in IEEE Internet of Things Journal, vol. 11, no. 12, pp. 21904-21915, 15 June, 2024.
[8] C. . -S. Jao, K. Stewart, J. Conradt, E. Neftci and A. M. Shkel, "Zero Velocity Detector for Foot-mounted Inertial Navigation System Assisted by a Dynamic Vision Sensor," in 2020 DGON Inertial Sensors and Systems (ISS), Braunschweig, Germany, 2020, pp. 1-18.
[9] C. -S. Jao, Y. Wang and A. M. Shkel, "A Zero Velocity Detector for Foot-mounted Inertial Navigation Systems Aided by Downward-facing Range Sensor," in 2020 IEEE SENSORS, Rotterdam, Netherlands, 2020, pp. 1-4.
[10] University of Toronto STARS Lab. Foot-Mounted Inertial Navigation Dataset
[11] Brandon Wagstaff, Valentin Peretroukhin, Jonathan Kelly, July 20, 2021, "University of Toronto Foot-Mounted Inertial Navigation Dataset", IEEE Dataport, doi: https://dx.doi.org/10.21227/v1z6-9z84.
[12] J. Wahlström and I. Skog, "Fifteen Years of Progress at Zero Velocity: A Review," in IEEE Sensors Journal, vol. 21, no. 2, pp. 1139-1151, 15 Jan., 2021.
[13] Guimarães, V.; Sousa, I.; Correia, M.V. Orientation-Invariant Spatio-Temporal Gait Analysis Using Foot-Worn Inertial Sensors. Sensors 2021, 21, 3940.
[14] Guimarães, V.; Sousa, I.; Correia, M.V. A Deep Learning Approach for Foot Trajectory Estimation in Gait Analysis Using Inertial Sensors. Sensors 2021, 21, 7517.
[15] J. Li et al., "Learning-Based Stance Phase Detection and Multisensor Data Fusion for ZUPT-Aided Pedestrian Dead Reckoning System," in IEEE Internet of Things Journal, vol. 11, no. 4, pp. 5899-5911, 15 Feb., 2024.