Skip to content

Commit 2c5574f

Browse files
RaidakarimRaidakarim
authored
Visualize robot motion (#54)
* add initial stop icon in header * modify lock icon * add non-moving states lock * edit readme * add lock for paused state * initial circle code * add circle progress bar * addressed pr comments * add text style * robot motion time add * robot motion time add * style fix and pause text
1 parent a170ed5 commit 2c5574f

19 files changed

+2424
-73
lines changed

‎README.md

+2
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,5 @@ This repository contains code for the feeding web app. The app itself is in `fee
99
- [web_video_server (branch: `ros2`)](https://github.com/RobotWebTools/web_video_server/tree/ros2)
1010
- Dependency: [async_web_server_cpp (branch: `ros2-develop`)](https://github.com/fkie/async_web_server_cpp)
1111
- Dependency: [vision_opencv (branch: `humble` or your ROS2 version)](https://github.com/ros-perception/vision_opencv/tree/humble)
12+
13+
All these repositories of `feeding_web_interface`, `ada_feeding`, `async_web_server_cpp`, `vision_opencv`, `web_video_server`, and `PRL fork of rosbridge_suite` as mentioned above should be downloaded using `git clone ...` in the "src" folder inside the ROS2 workspace. Please follow the [Ubuntu (Debian) tutorial](https://docs.ros.org/en/humble/Installation/Ubuntu-Install-Debians.html) for ROS2 installation in Ubuntu 22, which will automatically lead to creation of "src" folder in ROS2 workspace. To access hidden `.env` file in Ubuntu to change the debug flag's value, press Ctrl + H in the `feeding_web_interface` folder.

‎feedingwebapp/.eslintignore

+2
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44
/node_modules
55
/.pnp
66
.pnp.js
7+
# don't lint progressbar.js because it is a third-party
8+
/src/Pages/Home/MealStates/progressbar.js
79

810
# testing
911
/coverage

‎feedingwebapp/Acknowledgements.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
**Acknowledgement for all the icons we used:**
22

3-
“[Bowl with fruits](https://thenounproject.com/icon/bowl-with-fruits-5039233/)” icon by Langtik, “[Fork](https://thenounproject.com/icon/fork-10962/)” icon by Dmitry Baranovskiy, “[Human head mouth](https://thenounproject.com/icon/open-mouth-2885042/)” icon by Wuppdidu, “[Human head](https://thenounproject.com/icon/head-2288243/)” icon by Wildson Rafalski, “[Strawberry](https://thenounproject.com/icon/strawberry-3487721/)”icon by Ashabul Kahfi, “[Robot arm](https://thenounproject.com/icon/robot-arm-4466147/)” icon by Peter Lakenbrink, and “[Stop](https://thenounproject.com/icon/stop-34715/)” icon by AS Design from Noun Project CC BY 3.0.
3+
“[Bowl with fruits](https://thenounproject.com/icon/bowl-with-fruits-5039233/)” icon by Langtik, “[Fork](https://thenounproject.com/icon/fork-10962/)” icon by Dmitry Baranovskiy, “[Human head mouth](https://thenounproject.com/icon/open-mouth-2885042/)” icon by Wuppdidu, “[Human head](https://thenounproject.com/icon/head-2288243/)” icon by Wildson Rafalski, “[Strawberry](https://thenounproject.com/icon/strawberry-3487721/)”icon by Ashabul Kahfi, “[Robot arm](https://thenounproject.com/icon/robot-arm-4466147/)” icon by Peter Lakenbrink, “[Lock](https://thenounproject.com/icon/lock-3107451/)” icon by IcoMoon and “[Stop](https://thenounproject.com/icon/stop-34715/)” icon by AS Design from Noun Project CC BY 3.0.

‎feedingwebapp/README.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,8 @@ The overall user flow for this robot can be seen below.
2222
4. Source the directory: `source install/setup.bash`
2323
5. Navigate to the web app folder: `cd feeding_web_interface/feedingwebapp`
2424
6. Install web app dependencies: `npm install --legacy-peer-deps`
25-
* Consider checking out the [Troubleshooting](## Troubleshooting) section if there are errors in this process.
25+
* Consider checking out the Troubleshooting section if there are errors in this process.
26+
* Always run `source /opt/ros/humble/setup.bash` in the ROS2 workspace when a new terminal is opened.
2627

2728
### Usage (Web App)
2829
1. Navigate to the web app folder: `cd {path/to/feeding_web_interface}/feedingwebapp`

‎feedingwebapp/package-lock.json

+16
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎feedingwebapp/package.json

+1
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
"react": "^18.1.0",
1313
"react-bootstrap": "^2.3.1",
1414
"react-dom": "^18.1.0",
15+
"react-hook-form": "^7.43.9",
1516
"react-native-web": "^0.19.4",
1617
"react-router-dom": "^6.3.0",
1718
"react-script-tag": "^1.1.2",

‎feedingwebapp/public/robot_state_imgs/lock_icon_image.svg

+135
Loading

‎feedingwebapp/src/Pages/Constants.js

+22-8
Original file line numberDiff line numberDiff line change
@@ -14,17 +14,31 @@ export const REALSENSE_HEIGHT = 480
1414
export const TIME_TO_RESET_MS = 3600000 // 1 hour in milliseconds
1515

1616
/**
17-
* A dictionary containing the icon associated with each "robot motion" state.
17+
* A dictionary containing the icon associated with each "robot moving" state
18+
* except the Bite Acquisition state which has only a back button in footer.
19+
* This dictionary is used to populate buttons with the icon images for states
20+
* that those buttons will transition to.
1821
* The keys are the meal states where the robot moves.
1922
* Each key is paired with a value of an svg icon image of that meal state.
2023
*/
21-
let FOOTER_STATE_ICON_DICT = {}
22-
FOOTER_STATE_ICON_DICT[MEAL_STATE.R_MovingAbovePlate] = '/robot_state_imgs/move_above_plate_position.svg'
23-
FOOTER_STATE_ICON_DICT[MEAL_STATE.R_BiteAcquisition] = '/robot_state_imgs/move_to_bite_acquisition_position.svg'
24-
FOOTER_STATE_ICON_DICT[MEAL_STATE.R_MovingToStagingLocation] = '/robot_state_imgs/move_to_staging_position.svg'
25-
FOOTER_STATE_ICON_DICT[MEAL_STATE.R_MovingToMouth] = '/robot_state_imgs/move_to_mouth_position.svg'
26-
FOOTER_STATE_ICON_DICT[MEAL_STATE.R_StowingArm] = '/robot_state_imgs/stowing_arm_position.svg'
27-
export { FOOTER_STATE_ICON_DICT }
24+
let MOVING_STATE_ICON_DICT = {}
25+
MOVING_STATE_ICON_DICT[MEAL_STATE.R_MovingAbovePlate] = '/robot_state_imgs/move_above_plate_position.svg'
26+
MOVING_STATE_ICON_DICT[MEAL_STATE.R_MovingToStagingLocation] = '/robot_state_imgs/move_to_staging_position.svg'
27+
MOVING_STATE_ICON_DICT[MEAL_STATE.R_MovingToMouth] = '/robot_state_imgs/move_to_mouth_position.svg'
28+
MOVING_STATE_ICON_DICT[MEAL_STATE.R_StowingArm] = '/robot_state_imgs/stowing_arm_position.svg'
29+
export { MOVING_STATE_ICON_DICT }
30+
31+
/**
32+
* A set containing the states where the robot does not move.
33+
*/
34+
let NON_MOVING_STATES = new Set()
35+
NON_MOVING_STATES.add(MEAL_STATE.U_PreMeal)
36+
NON_MOVING_STATES.add(MEAL_STATE.U_BiteSelection)
37+
NON_MOVING_STATES.add(MEAL_STATE.U_BiteAcquisitionCheck)
38+
NON_MOVING_STATES.add(MEAL_STATE.U_BiteInitiation)
39+
NON_MOVING_STATES.add(MEAL_STATE.U_BiteDone)
40+
NON_MOVING_STATES.add(MEAL_STATE.U_PostMeal)
41+
export { NON_MOVING_STATES }
2842

2943
// The names of the ROS topic(s)
3044
export const CAMERA_FEED_TOPIC = '/camera/color/image_raw'

‎feedingwebapp/src/Pages/Footer/Footer.jsx

+3-3
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ import Row from 'react-bootstrap/Row'
88
import PropTypes from 'prop-types'
99

1010
// Local imports
11-
import { FOOTER_STATE_ICON_DICT } from '../Constants'
11+
import { MOVING_STATE_ICON_DICT } from '../Constants'
1212
import { useGlobalState } from '../GlobalState'
1313

1414
/**
@@ -32,8 +32,8 @@ const Footer = (props) => {
3232

3333
// Icons and other parameters for the footer buttons
3434
let pauseIcon = '/robot_state_imgs/pause_button_icon.svg'
35-
let backIcon = props.backMealState ? FOOTER_STATE_ICON_DICT[props.backMealState] : ''
36-
let resumeIcon = FOOTER_STATE_ICON_DICT[mealState]
35+
let backIcon = props.backMealState ? MOVING_STATE_ICON_DICT[props.backMealState] : ''
36+
let resumeIcon = MOVING_STATE_ICON_DICT[mealState]
3737
let phantomButtonIcon = '/robot_state_imgs/phantom_view_image.svg'
3838
// Width of Back and Resume buttons
3939
let backResumeButtonWidth = '150px'

‎feedingwebapp/src/Pages/GlobalState.jsx

+11-1
Original file line numberDiff line numberDiff line change
@@ -87,12 +87,18 @@ export const SETTINGS = {
8787
export const useGlobalState = create(
8888
persist(
8989
(set) => ({
90-
// Values stored in global state
90+
// The app's current meal state
9191
mealState: MEAL_STATE.U_PreMeal,
92+
// The timestamp when the robot transitioned to its current meal state
9293
mealStateTransitionTime: Date.now(),
94+
// The current app page
9395
appPage: APP_PAGE.Home,
96+
// The most recent food item that the user selected in "bite selection"
9497
desiredFoodItem: null,
98+
// The center of the mouth as detected by face detection
9599
detectedMouthCenter: null,
100+
// Whether or not the currently-executing robot motion was paused by the user
101+
paused: false,
96102
// Settings values
97103
stagingPosition: SETTINGS.stagingPosition[0],
98104
biteInitiation: SETTINGS.biteInitiation[0],
@@ -116,6 +122,10 @@ export const useGlobalState = create(
116122
set(() => ({
117123
detectedMouthCenter: detectedMouthCenter
118124
})),
125+
setPaused: (paused) =>
126+
set(() => ({
127+
paused: paused
128+
})),
119129
setStagingPosition: (stagingPosition) =>
120130
set(() => ({
121131
stagingPosition: stagingPosition

‎feedingwebapp/src/Pages/Header/Header.jsx

+33-4
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ import React, { useCallback, useEffect, useState } from 'react'
33
// The NavBar is the navigation toolbar at the top
44
import Navbar from 'react-bootstrap/Navbar'
55
import Nav from 'react-bootstrap/Nav'
6+
// The Button is used for stop icon at the top
7+
import Button from 'react-bootstrap/Button'
68
// PropTypes is used to validate that the used props are in fact passed to this
79
// Component
810
import PropTypes from 'prop-types'
@@ -13,7 +15,7 @@ import 'react-toastify/dist/ReactToastify.css'
1315
import { useROS } from '../../ros/ros_helpers'
1416

1517
// Local imports
16-
import { ROS_CHECK_INTERVAL_MS } from '../Constants'
18+
import { ROS_CHECK_INTERVAL_MS, NON_MOVING_STATES } from '../Constants'
1719
import { useGlobalState, APP_PAGE, MEAL_STATE } from '../GlobalState'
1820
import LiveVideoModal from './LiveVideoModal'
1921

@@ -44,6 +46,7 @@ const Header = (props) => {
4446
// Get the relevant global state variables
4547
const mealState = useGlobalState((state) => state.mealState)
4648
const setAppPage = useGlobalState((state) => state.setAppPage)
49+
const paused = useGlobalState((state) => state.paused)
4750

4851
/**
4952
* When the Home button in the header is clicked, return to the Home page.
@@ -79,7 +82,7 @@ const Header = (props) => {
7982
* of the robot is placed in between Settings and Video.
8083
*/}
8184
<Navbar collapseOnSelect expand='lg' bg='dark' variant='dark' sticky='top'>
82-
<Navbar id='responsive-navbar-nav'>
85+
<Navbar id='responsive-navbar-nav' bg='dark' variant='dark'>
8386
<Nav className='me-auto'>
8487
<Nav.Link
8588
onClick={homeClicked}
@@ -96,15 +99,41 @@ const Header = (props) => {
9699
Settings
97100
</Nav.Link>
98101
</Nav>
102+
{NON_MOVING_STATES.has(mealState) || paused ? (
103+
<div>
104+
<Button
105+
variant='danger'
106+
disabled={true}
107+
style={{
108+
marginLeft: 3,
109+
marginRight: 3,
110+
width: '44px',
111+
height: '56px',
112+
opacity: 1,
113+
'--bs-btn-padding-y': '0rem',
114+
'--bs-btn-padding-x': '0rem'
115+
}}
116+
>
117+
<img
118+
style={{ width: '44px', height: '50px' }}
119+
src='/robot_state_imgs/lock_icon_image.svg'
120+
alt='lock_icon_img'
121+
className='center'
122+
/>
123+
</Button>
124+
</div>
125+
) : (
126+
<></>
127+
)}
99128
{isConnected ? (
100129
<div>
101-
<p className='connectedDiv' style={{ fontSize: '24px' }}>
130+
<p className='connectedDiv' style={{ fontSize: '24px', margin: 3 }}>
102131
🔌
103132
</p>
104133
</div>
105134
) : (
106135
<div>
107-
<p className='notConnectedDiv' style={{ fontSize: '24px' }}>
136+
<p className='notConnectedDiv' style={{ fontSize: '24px', marginLeft: 3, marginRight: 3 }}>
108137
⛔
109138
</p>
110139
</div>

‎feedingwebapp/src/Pages/Home/Home.jsx

+3-1
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ function Home(props) {
2929
const mealState = useGlobalState((state) => state.mealState)
3030
const mealStateTransitionTime = useGlobalState((state) => state.mealStateTransitionTime)
3131
const setMealState = useGlobalState((state) => state.setMealState)
32+
const setPaused = useGlobalState((state) => state.setPaused)
3233

3334
/**
3435
* Implement time-based transition of states. This is so that after the user
@@ -40,8 +41,9 @@ function Home(props) {
4041
if (Date.now() - mealStateTransitionTime >= TIME_TO_RESET_MS) {
4142
console.log('Reverting to PreMeal due to too much elapsed time in one state.')
4243
setMealState(MEAL_STATE.U_PreMeal)
44+
setPaused(false)
4345
}
44-
}, [mealStateTransitionTime, setMealState])
46+
}, [mealStateTransitionTime, setMealState, setPaused])
4547

4648
// Get the relevant global variables
4749
const desiredFoodItem = useGlobalState((state) => state.desiredFoodItem)

0 commit comments

Comments
 (0)