A real-time perception-and-response system running on NVIDIA Jetson, built as a ROS 2 node graph.
USFRapidCam ties computer vision, voice, and an LLM into a set of cooperating ROS 2 nodes on a Jetson. It watches a live camera feed with YOLO11 pose estimation, listens to and transcribes speech, reasons about what is happening with an LLM, and can place an automated phone call when it needs to reach someone.
run_all_nodes.py launches four ROS 2 nodes:
| Node | Role |
|---|---|
videoNode |
Camera capture + YOLO11 pose detection |
voiceNode |
Audio capture and speech streaming |
llmNode |
LLM reasoning over the video and voice context |
callNode |
Places automated phone calls via Twilio |
Nodes communicate over standard ROS 2 topics (std_msgs), so each stage can run and be tested independently.
- Robotics: ROS 2 (rclpy)
- Vision: YOLO11-pose (Ultralytics)
- Edge: NVIDIA Jetson
- Telephony: Twilio
- Interface: web demo frontend + backend server
Designed to run on a Jetson with ROS 2 installed.
# build the ROS 2 workspace
colcon build
# launch every node
python3 run_all_nodes.pySet TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN, and TWILIO_FROM_NUMBER in a .env file for the call node.