Software Report

GitHub

When it comes to development, we work in this GitHub repository. The main branch is directly deployed to our URL. To maintain the integrity of our deployment, we established a CI/CD pipeline. Every new feature starts as a GitHub issue and is developed on a separate branch from main. Once the feature was ready, the developer submits a pull request (PR) and assigned reviewers. Each PR triggered the following automated GitHub Actions: a GitHub CodeQL scan for potential security vulnerabilities, and a Firebase preview deployment so reviewers could test the update without pulling or checking out the branch locally. Only once a PR passes all checks and reviews, do we then merge the branch into main.

Set-up Instructions

1. Clone the repository
2. Make sure you are in the app directory via a cd command or something similar.
3. Set-up a .gitignore file. We provide an example here.
4. Create a .env file with necessary keys. We provide an example here. This will depend on your Firebase setup.
5. Run npm install to install necessary packages.
6. Run npm start to run a local deployment of your current code.
7. Happy developing!

Project Architecture

ExerSights is a web application built with three main technologies: React.js (React), MediaPipe Solutions, and Firebase. The web app can be accessed via a standard web browser or downloaded as a Progressive Web App (PWA) to use offline if desired.

• The entire frontend user interface of ExerSights is built with React, a popular JavaScript library known for its component-based architecture and efficient Virtual DOM rendering, which updates only necessary UI elements for improved responsiveness. We leverage the Material UI (MUI) component library, implementing Google’s Material Design with production-ready components like Button, Typography, and Card.

• ExerSights uses the WebRTC getUserMedia API for camera access or a file API for uploaded videos, feeding these to Google’s MediaPipe Solutions for real-time human pose detection with limb and joint landmarks. We employ MediaPipe's lightweight tasks-vision landmark detection model (which uses OpenCV, an open-source computer vision model) to minimize client-side compute and latency, receiving X-Y coordinates of detected landmarks for angle calculations to track user posture.

• ExerSights does not have a dedicated backend. Instead, ExerSights is a frontend-focused application with pose processing and exercise logic handled client-side in JavaScript to reduce latency. For user cloud features and hosting, we utilize Firebase, Google’s BaaS. Specifically, we use App Hosting for streamlined deployment via GitHub CI/CD, Authentication for easy Google sign-on, Cloud Firestore as our NoSQL database for intuitive user data storage and customization, and Analytics to monitor user activity and app performance.

Directory Structure

app

The app directory cotains a few key files for Firebase configuration and node management that can be ignored. This is also where your .env file and installed node packages should be stored locally as well.

app/src

This is where a majority of our assests and code is stored.

App.js is the central file that defines high-level page metadata and the navigation to different pages such as the home page, catalog page, etc.

The other .js files are configuration files for Javascript or the PWA.

app/src/assets

This directory serves as the repository for the static assets and media that help with the application's presentation. It essentially encompasses non-dynamic content.

• about-profiles: This subdirectory contains the image files of the development team, intended for display on the "About" page.
• exercise-cards: This subdirectory containsthe representative images for each exercise, utilized as a display image in the "Catalog" page.
• instructions stores demonstrative images for each exercise, accessed via their respective help pages.
• logos stores a few variations of the ExerSights logo used throughout the app.
• The other files are audio files or textual data used throughout the application.

app/src/components

This directory contains reusable frontend components, used in the other pages. Generalizing certain components and calling them makes it easier to repeatedly use them. React function components are JavaScript functions that accept paramters and return front-end components. When data changes, React calls these functions again. Based on the function's internal logic (conditionals, loops) and the updated data, it returns the updated front-end. We will discuss the two most notable components and how they depend on other components.

• ExerciseBox.js is the main visual component for exercise attempts. On the left, it shows the video feed, using VideoCanvas.js for uploads and WebcamCanvas.js for live camera input, enabling pose tracking. On the right, it displays the FeedbackPanel.js for real-time exercise guidance and controls.
• FeedbackPanel.js is a significant component that dynamically displays real-time exercise information and feedback received through function parameters. Crucially, it also integrates interactive buttons that trigger the display of the HelpModal.js popup for detailed instructions and the SettingsModal.js popup for adjusting exercise parameters. Furthermore, it incorporates the Timer.js component for tracking exercise duration and a VoiceButton.js component for enabling voice-based interactions.
• The other components listed here are all used throughout the other pages.

app/src/pages

This directory contains the fully defined front-end components that each page displays. Each page listed corresponds to a page in ExerSights that can be navigated to. These pages load all the necessary components and pass the necessary parameters to the components via their function calls to render all the components correctly.

The exercises directory contains our ExercisePage.js (which primarly ultizes the ExerciseBox.jscomponent described above), along with a helper file (ExercisePageData.js) and an archived page (SquatPage.js).

app/src/shared

This directory is meant for storing fundamental resources utilized across different parts of the application, particularly external integrations.

Currently, it only contains the two MediaPipe computer vision models used for detecting human body movements from visual input.

app/src/utils

This directory contains a significant portion of the application's core logic, specifically concerning the processing of exercise data and the generation of user guidance. While this logic runs on the client-side, it is separated for organization and clarity.

The exercises subdirectory contains individual JavaScript files, each defining a specific exercise through a structured data format.

• helpers is a subdirectory that comprises a collection of utility functions, including those for:
  • Calculating body angles from tracking data.
  • Managing audio output.
  • Interfacing with user accounts via Firebase
  • Determining user visibility within the camera's view.
• models is a subdirectory containing the code responsible for loading and setting up the MediaPipe models.
• GenFeedback.js is the core code that defines a generalized system for evaluating exercise repetitions using body landmarks and the exercise data passed to it. The genCheck function takes exercise information and returns the relevant objects and datas for the pages to load.

Exercise Logic

With landmarks and location data of the user limbs provided by MediaPipe, our main challenge was to translate these different exercises into code. To do so, we decided to represent exercises as an finite state machine (FSM), where each exercise is broken down into different states and the triggers for transitions between states. The defined transitions often require a specific sequence of movements and reaching certain angular thresholds to progress through the different phases of the exercise and complete a valid repetition. We realized that most exercises can be reasonably defined with this FSM and would allow us to generalize pages to add many exercises easily.

Let's consider the Standing Toe Touch exercise, which, as illustrated, uses states like ‘DESCENDING,’ ‘FOLDING,’ ‘TOUCHING,’ and ‘RETURNING.’ A full repetition typically involves moving from the initial ‘STANDING’ state to the ‘DESCENDING’ state as the user begins to bend. To progress towards the toe touch, the user transitions to the ‘FOLDING’ state, bending further at the hips. The ‘TOUCHING’ state is reached when the user extends downwards to touch their toes or reach a target point. To complete the repetition, the user must then move through the ‘RETURNING’ state, gradually straightening back up until they reach the initial ‘STANDING’ state. This completes one full cycle. For instance, if a user has reached the ‘FOLDING’ state but doesn't extend downwards sufficiently to meet the criteria for the ‘TOUCHING’ state, they will not be able to transition directly back to the ‘RETURNING’ state.

To represent this FSM, we use a JSON data structure. Our current exercises can be found in this directory.

To add an exercise, developers can follow these steps:

1. Create a new exercise file in the exercise directory.

2. Define the FSM JSON for your exercise. Your FSM should include:

   Field	Purpose	Example
   states	Defines feedback, color, and repetition behavior for each posture state	INIT, FOLDING, TOUCHING, etc.
   transitions	Defines allowed state changes based on events	INIT ➔ FOLDING on "folding"
   jointInfo	Lists key joints and angles to monitor (based on landmark indices)	hipAngle, kneeAngle, etc.
   conditions	Describes what angle thresholds trigger which transitions	hipAngle ≤ targetHipAngle ➔ "hitTarget"
   targets	Sets the default angles you want the user to hit	e.g., targetHipAngle = 75 degrees
   angleSetters	Names of joint angles you want to continuously update during tracking	setHipAngle, setKneeAngle, etc.
   title	Display name for the exercise	"Standing Toe Touch"

   Try to use clear and reasonable names for the states, transitions, and angles.

3. Write a check function like the one shown below. Ensure camel case is used for the function name. This should be fairly straightforward once the exercise FSM JSON has been defined.

   export const checkYourExercise = (landmarks, onFeedbackUpdate, setColor, setCurrAngle1, setRepCount, yourTarget = 75) => {
       yourExerciseInfo.targets["targetAngleName"] = yourTarget;
   
       currState = genCheck(
           yourExerciseInfo,
           (...args) => getTransitionType(...args, yourExerciseInfo, currState),
           currState,
           landmarks,
           onFeedbackUpdate,
           setColor,
           setRepCount,
           { AngleName: setCurrAngle1 }
       );
   };
   

4. Find a suitable exercise image for the display and put in in exercise-cards. Put a demonstration image for the ideal camera angle in instructions.

5. Register your exercise in content.json and content.js.