Skip to content

Commit 63e4cd4

Browse files
minkyu-choi07minkyu-choi07
committed
refactor
1 parent 69d64b3 commit 63e4cd4

39 files changed

+923
-2240
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ artifacts/
44
store/
55
libs
66
ns_vfs/output/
7+
_dev_/
78
# Byte-compiled / optimized / DLL files
89
__pycache__/
910
*.py[cod]

‎README.md

Lines changed: 30 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@
99
<br />
1010
<div align="center">
1111
<a href="https://github.com/UTAustin-SwarmLab/Neuro-Symbolic-Video-Search-Temploral-Logic">
12-
<img src="images/logo.png" alt="Logo" width="340" height="340">
1312
</a>
1413

1514
<h3 align="center">Neuro Symbolic Video Search with Temporal Logic</h3>
@@ -28,12 +27,17 @@
2827

2928
## Table of Contents
3029

31-
- [TL;DR](#TL;DR)
32-
- [Abstract](#abstract)
33-
- [Installation Guide](#installation-guide)
34-
- [System Setup and Execution Guide](#system-setup-and-execution-guide)
35-
- [Running the System](#running-the-system)
36-
- [Citation](#citation)
30+
- [Neuro Symbolic Video Search with Temporal Logic (NSVS-TL)](#neuro-symbolic-video-search-with-temporal-logic-nsvs-tl)
31+
- [Table of Contents](#table-of-contents)
32+
- [TL;DR](#tldr)
33+
- [Abstract](#abstract)
34+
- [System Overview](#system-overview)
35+
- [Installation Guide](#installation-guide)
36+
- [System Setup and Execution Guide](#system-setup-and-execution-guide)
37+
- [Running the System](#running-the-system)
38+
- [FAQ](#faq)
39+
- [Connect with Me](#connect-with-me)
40+
- [Citation](#citation)
3741

3842
## TL;DR
3943

@@ -84,50 +88,32 @@ Please avoid stopping and removing the container, as you will need to reinstall
8488

8589
## System Setup and Execution Guide
8690

87-
### Configuration Management
88-
89-
This system utilizes the Hydra Python package to manage configurations effectively. Ensure you configure the system correctly to use it as intended.
90-
91-
#### Example Configurations
92-
93-
We provide default configurations for different use cases:
94-
95-
1. **Real-Video Configuration** - `real_video.yaml`
96-
2. **TLV Dataset Configuration** - `tlv_dataset.yaml`
97-
98-
#### Required Configuration Fields
99-
100-
Update the following fields in your configuration file based on your specific needs:
101-
102-
- `save_result_dir`: Directory to save the results
103-
- `video_file_path`: Path to the video file
104-
- `ltl_formula`: LTL formula specifications.
105-
- `proposition_set`: Set of propositions
106-
107-
Example configuration for the ltl_formula and proposition_set:
108-
```
109-
ltl_formula: "\"prop1\" U \"prop2\""
110-
proposition_set:
111-
- prop1
112-
- prop2
113-
```
91+
**Note that we've opted out the system run with the TLV dataset in the current version. If you're interested in running the code with TLV dataset, please use `v1.0.0` version. All versions after `v1.0.0` only supports for the real video.**
11492

11593
## Running the System
11694

117-
To launch the system, execute the `main.py` script from the `ns_vfs` directory. You will need to specify which configuration to use by setting the `config_name` parameter.
118-
119-
#### Example Command
120-
121-
To run the system with the `real_video` configuration:
122-
123-
```bash
124-
python3 main.py +config_name=real_video
95+
```python
96+
from ns_vfs import run_nsvs_yolo
97+
from cvias.image.detection.object.yolo import Yolo
98+
99+
model_list = Yolo.available_models()
100+
101+
run_nsvs_yolo(
102+
video_path="video_path",
103+
desired_fps="fps", # optional: desired_interval_in_sec="You can get the frame by sec",
104+
proposition_set=["prop1", "prop2"], # e.g.["car", "truck"],
105+
ltl_formula='"prop1" U "prop2"' # e.g.'"car" U "truck"'
106+
yolo_model_name="model from model_list" # e.g. "YOLOv8x",
107+
output_path="output dir",
108+
threshold_satisfaction_probability=0.80,
109+
)
125110
```
126111

112+
127113
## FAQ
128-
Q: We are receiving a smaller number of frames. For instance, we need frame 81 to be included in the frames of interest, but we have been provided with [...,21].
114+
Q: We are receiving a smaller number of frames. For instance, we need frame 81 to be included in the frames of interest, but nsvs-tl results in [...,21].
129115

130-
A: This issue involves video processing. You are looking at frames 0 to 81. However, the last frame index is 21 if you didn’t change the video processor's parameters (`/ns_vfs/config/video_processor/real_video.yaml`).
116+
A: This issue involves video processing. You are looking at frames 0 to 81. However, the last frame index is 21 if you didn’t change the video processor's parameters. Make sure to use correct parameters for `desired_fps` or `desired_interval_in_sec`.
131117

132118

133119
## Connect with Me

‎ns_vfs/__init__.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
11
"""Package containing your_project name."""
22

3+
from .api import run_nsvs_yolo
4+
5+
__all__ = ["run_nsvs_yolo"]
36
__version__ = "0.0.1"

‎ns_vfs/api/__init__.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
from .run_with_yolo import run_nsvs_yolo
2+
3+
__all__ = ["run_nsvs_yolo"]

‎ns_vfs/api/run_nsvs.py

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
from __future__ import annotations
2+
3+
import vflow
4+
5+
from ns_vfs.automaton._base import Automaton
6+
from ns_vfs.data.frame import FramesofInterest, VideoFrame
7+
from ns_vfs.model_checking.stormpy import StormModelChecker
8+
from ns_vfs.percepter._base import VisionPercepter
9+
from ns_vfs.validator import FrameValidator
10+
11+
12+
def run_nsvs(
13+
video_path: str,
14+
vision_percepter: VisionPercepter,
15+
automaton: Automaton,
16+
frame_of_interest: FramesofInterest,
17+
model_checker: StormModelChecker,
18+
proposition_set: list[str],
19+
ltl_formula: str,
20+
output_path: str,
21+
model_checker_verbose: bool = False,
22+
model_checker_is_filter: bool = False,
23+
model_checker_type: str = "sparse_ma",
24+
desired_interval_in_sec: float | None = None,
25+
desired_fps: int | None = None,
26+
) -> list:
27+
frame_validator = FrameValidator(ltl_formula=ltl_formula)
28+
video_processor = vflow.read_video(video_path)
29+
frame_idx = 0
30+
while True:
31+
frame_img = video_processor.get_next_frame(
32+
return_format="ndarray",
33+
desired_interval_in_sec=desired_interval_in_sec,
34+
desired_fps=desired_fps,
35+
)
36+
if frame_img is None:
37+
break # No more frames or end of video
38+
detected_objects: dict = vision_percepter.perceive(
39+
image=frame_img,
40+
object_of_interest=proposition_set,
41+
)
42+
activity_of_interest = None
43+
44+
frame = VideoFrame(
45+
frame_idx=frame_idx,
46+
timestamp=video_processor.current_frame_index,
47+
frame_image=frame_img,
48+
object_of_interest=detected_objects,
49+
activity_of_interest=activity_of_interest,
50+
)
51+
frame_idx += 1
52+
53+
# 1. frame validation
54+
if frame_validator.validate_frame(frame=frame):
55+
# 2. dynamic automaton construction
56+
automaton.add_frame(frame=frame)
57+
frame_of_interest.frame_buffer.append(frame)
58+
# 3. model checking
59+
model_checking_result = model_checker.check_automaton(
60+
transitions=automaton.transitions,
61+
states=automaton.states,
62+
model_type=model_checker_type,
63+
use_filter=model_checker_is_filter,
64+
)
65+
if model_checking_result:
66+
# specification satisfied
67+
frame_of_interest.flush_frame_buffer()
68+
automaton.reset()
69+
70+
print("--------------------------------")
71+
print("Detected frames of interest:")
72+
print(frame_of_interest.foi_list)
73+
# save result
74+
if output_path:
75+
frame_of_interest.save(path=output_path)
76+
print(f"\nResults saved in {output_path}")
77+
78+
return frame_of_interest.foi_list

‎ns_vfs/api/run_with_yolo.py

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
from __future__ import annotations
2+
3+
from cvias.image.detection.object.yolo import Yolo
4+
5+
from ns_vfs.api.run_nsvs import run_nsvs
6+
from ns_vfs.automaton.video_automaton import VideoAutomaton
7+
from ns_vfs.data.frame import FramesofInterest
8+
from ns_vfs.model_checking.stormpy import StormModelChecker
9+
from ns_vfs.percepter.single_vision_percepter import SingleVisionPercepter
10+
11+
12+
def run_nsvs_yolo(
13+
video_path: str,
14+
proposition_set: list[str],
15+
ltl_formula: str,
16+
output_path: str,
17+
yolo_model_name: str,
18+
yolo_model_path: str | None = None,
19+
gpu_number: int = 0,
20+
threshold_satisfaction_probability: float = 0.80,
21+
frame_scale: int | None = None,
22+
calibration_method: str = "temperature_scaling",
23+
desired_interval_in_sec: float | None = None,
24+
desired_fps: int | None = None,
25+
) -> None:
26+
# Yolo model initialization
27+
yolo_model = Yolo(
28+
model_name="YOLOv8x",
29+
explicit_checkpoint_path=yolo_model_path,
30+
gpu_number=gpu_number,
31+
calibration_method=calibration_method,
32+
)
33+
# Video automaton initialization
34+
ltl_formula = f"P>={threshold_satisfaction_probability} [{ltl_formula}]"
35+
automaton = VideoAutomaton()
36+
automaton.set_up(proposition_set=proposition_set)
37+
# Model checker initialization
38+
model_checker = StormModelChecker(
39+
proposition_set=proposition_set, ltl_formula=ltl_formula
40+
)
41+
# Frame of interest initialization
42+
frame_of_interest = FramesofInterest(ltl_formula=ltl_formula)
43+
# Video processor initialization
44+
45+
# Vision percepter initialization
46+
vision_percepter = SingleVisionPercepter(
47+
cv_models=yolo_model,
48+
)
49+
run_nsvs(
50+
video_path=video_path,
51+
vision_percepter=vision_percepter,
52+
automaton=automaton,
53+
frame_of_interest=frame_of_interest,
54+
model_checker=model_checker,
55+
proposition_set=proposition_set,
56+
ltl_formula=ltl_formula,
57+
output_path=output_path,
58+
desired_interval_in_sec=desired_interval_in_sec,
59+
desired_fps=desired_fps,
60+
)
61+
62+
63+
if __name__ == "__main__":
64+
run_nsvs_yolo(
65+
video_path="/home/mc76728/repo/Coargus/Neuro-Symbolic-Video-Search-Temporal-Logic/_dev_/toyota.mp4",
66+
desired_interval_in_sec=None,
67+
desired_fps=30,
68+
proposition_set=["car", "truck"],
69+
ltl_formula='"car" U "truck"',
70+
yolo_model_name="YOLOv8x",
71+
output_path="/home/mc76728/repo/Coargus/Neuro-Symbolic-Video-Search-Temporal-Logic/_dev_",
72+
threshold_satisfaction_probability=0.80,
73+
frame_scale=None,
74+
calibration_method="temperature_scaling",
75+
)

‎ns_vfs/automaton/_base.py

Lines changed: 0 additions & 8 deletions
This file was deleted.

0 commit comments

Comments
 (0)