Real-Time Text-to-Speech on RP2040

ECE 4760 Final Project, Fall 2025

Team: Yayun Zhao (yz3545), Qingyin Zhong (qz425), Hongming Yang (hy665)

Project Summary: This project demonstrates a real-time text-to-speech (TTS) system running entirely on the RP2040 microcontroller. The system converts input text into intelligible speech audio with low latency, leveraging dual-core parallelism and efficient embedded audio processing. The design is tailored for resource-constrained environments, making it ideal for embedded, IoT, and audio applications.

• End-to-end TTS: text input → speech output
• Dual-core (RP2040) real-time audio synthesis
• Multiple voice profiles, adjustable pitch, speed, and timbre
• SD card dictionary and phoneme storage
• Hardware-accelerated DAC output via SPI (MCP4822)

What you'll learn: How to architect and optimize a real-time TTS pipeline on embedded hardware, and the trade-offs between algorithmic complexity, memory, and audio quality.

System Overview

• Input: ASCII text (via serial or SD card)
• Processing: Text parsing → phoneme mapping → audio synthesis
• Output: Real-time audio via DAC (SPI to MCP4822), speaker

The system supports two main implementation pathways, each with distinct trade-offs in latency, memory, and audio quality.
[System Block Diagram Placeholder]

What you'll learn: The overall data flow and hardware-software co-design for embedded TTS.

Implementation Pathways Comparison

Key takeaway: Pathway A is ideal for extreme resource constraints. Pathway B offers a good balance of quality and performance. Pathway C provides the best audio quality at the cost of storage and complexity.

Program Design

The software is modular, with clear separation between text parsing, phoneme conversion, prosody generation, and audio synthesis. Each module is optimized for real-time performance and minimal memory usage.

• Text Parser: Tokenizes input, handles punctuation and case.
• Phoneme Mapper: Looks up or generates phoneme sequences for each word.
• Prosody Engine: Assigns pitch, duration, and stress to each phoneme.
• Audio Synthesizer: Generates or streams PCM data for output.

What you'll learn: How to structure a TTS pipeline for embedded systems, and the importance of modularity for debugging and extension.

Architecture & Data Flow

The system leverages the RP2040's dual-core architecture for parallel processing:

• Core 0: Handles text parsing, phoneme mapping, and SD card I/O.
• Core 1: Manages real-time audio synthesis and DAC output.

Data is exchanged via a ring buffer to ensure smooth, uninterrupted audio playback. The DAC (MCP4822) is driven via SPI at a recommended sampling rate of 16–22 kHz, with a buffer size of 256–512 samples for optimal latency and stability.

What you'll learn: How to exploit hardware parallelism and buffer management for real-time audio on microcontrollers.

Implementation Details

Pathway A: Dual-Core Architecture

• In offline_tts.c, the main() function calls multicore_launch_core1(core1_entry);, which starts the second core (core1) and runs the core1_entry function.
• The project links the pico_multicore library (see CMakeLists.txt), the official RP2040 SDK for multicore programming.
• The header pt_cornell_rp2040_v1_4.h defines FIFO macros (e.g., PT_FIFO_WRITE, PT_FIFO_READ) for inter-core communication, and provides two scheduler structs: pt_sched (core0) and pt_sched1 (core1), enabling separate task scheduling for each core.

• Core 0 (Primary):
  • Handles all system initialization (hardware, SD card, SPI, GPIO, LED, etc.).
  • Runs the main loop, which is mainly a heartbeat LED blink (no user input or synthesis).
  • Manages audio output: a timer interrupt (audio_timer_cb) continuously reads from the audio buffer and sends data to the DAC via SPI, also controlling the audio status LED.
  • Responsible for all hardware management.
• Core 1 (Secondary):
  • Runs core1_entry, which loops to monitor serial input (e.g., '1', '2', '3') for switching voice styles (male, female, child).
  • Updates current_voice and prints status when a new style is selected.
  • After switching, calls synthesize_demo_word() to generate a demo audio segment (calls a series of synthesize_formant_phoneme_fixed).
  • Handles all speech synthesis and audio data generation, writing samples to the buffer.
• Inter-core Collaboration:
  • Global variables and the spinlock audio_lock protect the audio buffer for safe dual-core access.
  • Core 0 focuses on audio output and hardware, ensuring smooth playback.
  • Core 1 focuses on user interaction and speech data generation, offloading computation from the primary core and improving responsiveness.

Pathways B & C: Single-Core + Interrupt Model

• Both pathways use a single core with a cooperative protothread (protothread_console) for user command handling and synthesis.
• When the user sends a command like say <word> via UART, the synthesis is performed synchronously in the protothread.

• synthesize_speech(const char *text) is called in the protothread, parsing the input, looking up the dictionary, and converting text to a phoneme sequence.
• For each phoneme of type PHONEME_FORMANT (vowel), synthesize_formant_vowel_espeak(phoneme_t *ph) is called:
  • Generates a sawtooth source signal.
  • Applies low-pass filtering.
  • Passes through three fixed-point IIR resonators (F1/F2/F3).
  • Applies envelope and DC blocking.
  • Writes each sample to the audio ring buffer using audio_buffer_write_safe(uint16_t sample), which uses atomic operations for safety.

• Playback is not handled in the thread. Instead, a hardware timer interrupt (audio_timer_cb) at 22,050 Hz reads samples from the buffer and sends them to the DAC via SPI.

• Vowel synthesis is a synchronous, blocking operation performed entirely in the user command thread, with the resulting samples written to the buffer.
• Real-time audio output is guaranteed by the interrupt-driven playback, ensuring deterministic synthesis and uninterrupted audio even during computation.

Thread and interrupt flow diagram for Pathway C: The main thread handles synthesis and SD card reading, while a hardware timer interrupt ensures real-time audio output to the DAC. This separation allows uninterrupted playback even during SD card access or synthesis computation.

End-to-End "hello" Demo Flow

1. User inputs "hello" via serial or SD card.
2. Text parser splits into phonemes: h, e, l, o.
3. Phoneme mapper finds or generates PCM for each phoneme.
4. Vowels are synthesized; consonants are loaded from SD card.
5. PCM data is buffered and streamed to DAC in real time.
6. Speaker outputs the word "hello" with selected voice profile, pitch, and speed.

Hardware Design

This section details the hardware architecture and key configurations of the real-time text-to-speech system running on the RP2040 microcontroller. Below is an overview diagram illustrating the main components.

Hardware overview of the RP2040 TTS system, showing key connections and components.

A. Libraries and Runtime Environment

• Pico SDK
• Cornell PT library
• SD card stub
• Formant interface

B. Audio System Configuration

• Sample rate: 22050 Hz (enough, better for real time)
• Ring buffer: 4096 samples
• Sample format: 12-bit DAC value stored in uint16_t
• Playback timing: $\approx$ 22.2 kHz timer interrupt, close to the target 22.05 kHz.

C. Hardware Pins and SPI / DAC Interface

• The system uses SPI port zero running at twenty megahertz.
• Pin assignments: SCK, MOSI, MISO, CS, LDAC, LED.

1. SD Card Module (SPI1 Bus)
   
   • microSD Vin (5 volt) --> VBUS on PICO (pin 40)
   • microSD GND --> PICO gnd
   • microSD CLK --> gpio 10 SPI1_SCK
   • microSD DI --> gpio 11 SPI1_TX
   • microSD DO--> gpio 12 SPI1_RX
   • microSD CS --> gpio 13 software chip select
2. MCP4822 DAC (SPI0 Bus)
   

   Pico GPIO	Function	DAC Pin
   GPIO 5	CS	Chip Select
   GPIO 6	SCK	Clock
   GPIO 7	MOSI	Data
   GPIO 8	LDAC	Latch/Ground

3. Status Indicator Lights
   

   Pico GPIO	Function	Description
   GPIO 25	LED	On-board LED

4. Power and Ground
   

   Pico GPIO	Function	Description
   5V/3.3V	Power	DAC/SD card power supply
   GND	Ground	DAC/SD card ground

Buffer and DAC Output Design

1. Buffer Design
   Ring Buffer Size:
   The RP2040 has 264 KB of internal SRAM. The audio buffer must be allocated without impacting other tasks.
   
   16 KB ÷ 2 bytes/sample = 8192 samples
   At a 16 kHz sample rate, this buffer stores about 0.5 seconds of audio, enough to absorb synthesis/output jitter.
   Double Buffering Strategy:
   A producer-consumer model is used: Core1 fills the buffer, Core0 outputs via DMA, avoiding blocking.
   Buffer Management:
   • Ring queue with atomic pointer updates to prevent dual-core contention.
   • Buffer state monitoring (low watermark alert) to prevent underrun.
2. DAC Configuration Details
   Device: MCP4822 (12-bit dual-channel DAC, SPI interface).
   Output Parameters:
   • Resolution: 12-bit, output range 0~4.096 V
   • Sample rate: 16 kHz (up to 22.05 kHz, SPI bandwidth permitting)
   • SPI clock: at least 8 MHz to ensure data transfer per sample period
   DMA Configuration:
   • RP2040 DMA channels push buffer data directly to SPI FIFO, reducing CPU load
   • DMA trigger: timer interrupt (PWM or PIO), ensuring precise sample intervals
3. Input/Output Pathways
   Input:
   
   USB Serial (text) → eSpeak NG (phoneme sequence) → Parameter lookup → Core1 waveform synthesis → Write to ring buffer
   Output:
   
   Core0 buffer monitor → DMA to SPI → MCP4822 DAC output → External low-pass filter → Speaker
   Latency Control:
   Buffer size and DMA batch transfer strategy ensure total latency < 50 ms.
4. 

   System Data Flow Diagram.

Challenges

• Vowel Synthesis Parameter Tuning
  The filter (formant) and synthesis parameters (such as fundamental frequency, formant frequencies, bandwidth, etc.) require constant tuning. Although we referenced parameters from eSpeak, direct adoption did not yield ideal results due to differences in synthesis algorithm implementation. All phoneme parameters had to be repeatedly adjusted and optimized specifically for our algorithm to achieve more natural speech quality.
• Limitations of Integer Arithmetic
  To match the int16 DAC output, the entire synthesis pipeline uses int types for intermediate values. This improves efficiency and compatibility, but also limits computational precision and flexibility. With more resources and time, it would be beneficial to use float for intermediate calculations, then shift and clip to int16 at the output stage, improving audio quality and algorithm performance.
• Buffering and SD Card Access Latency
  The limited RAM of the RP2040 restricts the size of the audio buffer. Without proper preloading and caching mechanisms, SD card file access can introduce tens of milliseconds or more of perceptible latency, affecting speech fluency. Since our project stores and accesses files by phoneme, each must be read in real time. Buffering strategy and buffer design are therefore critical. The prefetch and buffering scheme must be carefully designed based on the actual buffer size to meet the requirements of real-time speech synthesis.

Appendices

A. Code Download

• Pathway A (Real-Time Synthesis): Download Code (tts_syn.zip)
• Pathway B (Hybrid Synthesis): Download Code (syn+sd.zip)
• Pathway C (Sample-Based TTS): Download Code (tts_sd.zip)

Note: Each code package includes the "tts_data" folder, which should be placed on the SD card.

B. Team Member Contributions

• Yayun Zhao: Buffer and synthesis algorithm research; contributed to Pathway A implementation.
  
  
• Qingyin Zhong: SD card system setup, pitch algorithm research; contributed to Pathways B and C implementation.
  
  
• Hongming Yang: Audio file construction, hardware setup; contributed to Pathways B and C implementation.
  
  

Project presentation and report were completed collaboratively by all three members. (Order does not indicate ranking.)

The group approves this report for inclusion on the course website.

The group approves the video for inclusion on the course youtube channel.

C. References

• ECE 4760/5730 Course Website
• ECE4760 microSD Card Reader Guide
• eSpeak NG - Synthesis & Phoneme Reference
• FatFs - Generic FAT Filesystem Module
• RP2040 Datasheet