Event-Triggered Audio Replay Agent

An event-triggered replay stack for replaceable audio nodes, connected by a stable uplink contract to PC-side buffering and ASR.

中文说明见：README.zh-CN.md

Architecture

Overall

flowchart LR
  node["Audio Uplink Node<br/>any hardware implementing the contract"] -->|"UDP PCM<br/>node_uuid / node_id / sample metadata"| hub["PC Audio Hub"]
  mqtt["Control Plane<br/>MQTT-compatible topics"] <-->|"status / commands"| node
  hub --> ring["Rolling Buffer"]
  ring --> jobs["Async STT Jobs"]
  jobs --> asr["ASR Worker<br/>Bailian Qwen ASR by default"]
  mcp["MCP Client"] --> hub

Loading

This diagram shows the system contract: the software side expects an upstream node that can send protocol-compatible audio packets and expose a basic status/control plane. It does not require that node to be an ESP32-S3.

Reference Hardware Architecture

flowchart LR
  mic["I2S Microphone<br/>INMP441"] --> capture["Capture<br/>16 kHz / 16-bit / mono"]
  capture --> packetizer["Packetizer<br/>20 ms PCM frames"]
  identity["ESP32-S3 STA MAC"] --> uuid["node_uuid"]
  uuid --> packetizer
  packetizer --> udp["UDP Uplink<br/>contract packet format"]
  cfg["Runtime Config<br/>Wi-Fi / MQTT / UDP / node_id"] --> udp
  cfg --> mqtt_hw["MQTT Telemetry and Control"]
  setup["AP/STA Setup Page"] --> cfg

Loading

The Hardware/Mic-ESP32 firmware is one reference implementation of that contract, not the only hardware assumption of the PC hub.

What This Repo Contains

• Hardware/Mic-ESP32 reference ESP32-S3 firmware for a microphone node.
• Software/pc_hub PC-side UDP ingest, rolling buffer, MCP server, and cloud-first ASR worker.

Today the system does this:

• ingests audio nodes through a stable UDP PCM contract
• captures 16 kHz / 16-bit / mono PCM on ESP32-S3 in the reference firmware
• tracks nodes by node_uuid
• buffers recent audio on the PC
• submits async STT jobs to Alibaba Cloud Model Studio qwen3-asr-flash by default
• exposes MCP as the preferred AI-facing interface

Fastest Path

Hardware

Use the supported ESP-IDF 5.5.3 commands in:

• Hardware/Mic-ESP32/README.md

If the node has no runtime config, it boots into setup mode:

• connect to MicSetup-<last6>
• open http://192.168.4.1/
• fill Wi-Fi, MQTT, UDP, and node_id

Software

The recommended runtime path is:

1. install the Python package
2. start worker.main
3. start mcp_adapter.main
4. use MCP as the primary interface

Minimal example:

cd Software/pc_hub
uv sync

$env:DASHSCOPE_API_KEY="your-api-key"
$env:PC_HUB_ASR_PROVIDER="bailian"
$env:PC_HUB_ASR_LANGUAGE="zh"
uv run python -m worker.main

cd Software/pc_hub
$env:PC_HUB_MCP_BIND_HOST="127.0.0.1"
$env:PC_HUB_MCP_PORT="8767"
$env:PC_HUB_MCP_PATH="/mcp"
uv run python -m mcp_adapter.main

MCP endpoint:

http://127.0.0.1:8767/mcp

Legacy HTTP remains available for compatibility and debugging, but it is optional and disabled by default.

Where To Read Next

• Hardware/Mic-ESP32/README.md Firmware setup, provisioning flow, build, flash, and node behavior.
• Software/pc_hub/README.md Runtime model, configuration, recommended startup path, Docker, and legacy API.
• docs/verification.md Worker smoke checks, MCP validation notes, simulated uplink status, and legacy HTTP verification.
• docs/protocols.md Audio uplink format, MQTT topics, timebase, and public integration contracts.

Notes

• node_uuid is the stable backend key; the current reference firmware derives it from the ESP32-S3 STA MAC.
• node_id is the human-readable label configured locally.
• Query windows use pc_receive_time, not the embedded packet timestamp.
• The project is audio-first right now; video ingestion is future work.

Star History