Course Project for CS4187 Computer Vision for Interactivity, Semester A 2024/25, City University of Hong Kong.
Project Lephon is an interactive rhythm game developed using openFrameworks. It combines hand tracking with engaging gameplay to provide a dynamic and immersive experience. Players interact with the game by moving their hands in front of a webcam to tap and hold notes in sync with music videos.
- Hand Tracking: Utilizes a pre-trained neural network for real-time hand detection using the system's webcam.
- Rhythm-Based Gameplay: Offers two types of notes—tap notes and hold notes—that players must interact with at precise timings.
- Dynamic Visuals: Displays shrinking circles and images for notes to provide visual cues aligned with the music.
- Scoring System: Rewards players with "GREAT!" or "MISS" feedback based on their accuracy.
- Chart Creation Mode: Allows users to create custom note charts by recording their hand movements while watching a video.
demo.mp4
- Hardware: A computer with a webcam.
- Software:
- OS: Tested on macOS Sequoia (version 15.2), but should work on other platforms as well.
- openFrameworks version 0.12.0.
- Addons:
ofxOpenCv
,ofxCv
,ofxGui
.
- Addons:
- Clone the Repository: Download or clone the project to your local machine.
- Data Files: Ensure the following files are placed in the
bin/data
directory:cross-hands-tiny-prn.cfg
andcross-hands-tiny-prn.weights
for hand detection.- Images:
hand.png
,tap.png
,hold.png
for visual elements.
- Build the Project: Open the project in your IDE and compile it using openFrameworks.
- Start the Application: Run the compiled executable.
- Main Menu: Upon launch, you'll see the main menu where you can choose to:
- Play Game: Start playing with an existing note chart.
- Create Chart: Create a new note chart for a video.
- Selecting a Video: Choose an
.mp4
video file when prompted. - Gameplay:
- Make sure your hand is visible to the webcam.
- Hand Icon: A hand icon will follow your hand movements on the screen.
- Tap Notes: Quickly move your hand over the tap notes as they appear.
- Hold Notes: Hold your hand over the hold notes for their duration.
- Feedback:
- A "GREAT!" message indicates a successful hit.
- A "MISS" message appears if you miss a note.
- Enter Chart Creation Mode: Select "Create Chart" from the main menu.
- Recording Notes:
- Tap Notes: Press any key (except the spacebar) to place a tap note at the current time and hand position.
- Hold Notes:
- Press and hold the spacebar to start a hold note.
- Release the spacebar to end the hold note.
- Saving the Chart: The note chart will be saved as a
_gen.txt
file alongside your video.
- OpenCV DNN Module: The application uses OpenCV's DNN module to load a pre-trained YOLO model (
cross-hands-tiny-prn
) for hand detection. - Processing:
- Captures frames from the webcam and processes them to detect hands.
- Updates the position of the hand icon based on detection results.
-
Note Base Class:
- Contains common attributes like
x
,y
,time
, and methods for updating and drawing. - Abstract methods
update
,draw
,isGreat
,isFinished
, andisStarted
are implemented in derived classes.
- Contains common attributes like
-
TapNote:
- Represents a single tap note.
- Displays
tap.png
when not hit. - Draws a circle that shrinks as the note's time approaches, providing a visual countdown.
-
HoldNote:
- Represents a note that must be held over a duration.
- Displays
hold.png
and a progress indicator showing how long the note has been held. - Tracks the
holdTime
to determine if the note was successfully completed.
-
PlayGame:
- Manages the main gameplay.
- Loads notes from a chart file (
_gen.txt
or.txt
). - Updates and draws active notes based on the current time.
- Checks for note hits and provides feedback.
-
CreateChart:
- Allows players to create custom note charts.
- Records the timing and position of tap and hold notes based on key presses and hand position.
- Saves the recorded notes to a
_gen.txt
file for future gameplay.
- Main Application:
ofApp.cpp
andofApp.h
: Handle the overall application setup, including webcam initialization and hand detection.
- Modes:
BaseMode
: An abstract class that other modes inherit from.MainMenu
,SelectSong
,PlayGame
,CreateChart
,EndScreen
: Classes representing different states of the application.
- Notes:
Note.h
: Contains theNote
,TapNote
, andHoldNote
class definitions.
- Resources:
- Models: Neural network configuration and weights files for hand detection.
- Images: Visual assets like
hand.png
,tap.png
, andhold.png
.
- Improve hand detection accuracy and robustness.
- Add support for multiple hand detection.
- Allow users to adjust audio offset.
-
The hand detection model was obtained from the cansik/yolo-hand-detection repository.
-
The two music videos used in the game are:
-
The icons for hand, tap note, and hold note were sourced from Flaticon.