Coqui TTS GPU Project - TTS Humanize

A personal Text-to-Speech (TTS) training project using Coqui TTS with VITS model for generating natural, personalized voice synthesis. This project converts text to speech with GPU acceleration (CUDA), enabling high-quality inference and fine-tuning on custom voice datasets.

Project Overview

TTS Humanize is designed to train and deploy single-speaker VITS models for generating speech in a specific voice. The project includes:

• Pre-processing and cleaning of audio datasets
• Training VITS models on custom voice data
• Inference from trained checkpoints
• Dataset analysis and quality validation
• Real-time training monitoring and management

Key Features

• GPU-Accelerated Training: Optimized for NVIDIA RTX 3050 (6GB VRAM) and similar GPUs
• Dataset Cleaning: Automated detection and removal of noisy/low-quality samples using text patterns and audio metrics (RMS, duration)
• Model Checkpointing: Saves best models and regular checkpoints during training
• Real-time Inference: Generate speech from trained models with custom text
• Training Management: Easy start/stop/monitor commands for training workflows
• Audio Quality Metrics: RMS analysis, duration filtering, silence detection

Project Structure

tts_humanize/
├── README.md                               # This file
├── requirements.txt                        # Python dependencies
├── main.py                                 # Simple TTS inference with pre-trained models
├── train_vits.py                          # VITS model training script
├── infer_trained.py                       # Inference from trained checkpoints
├── generate_checkpoint_samples.py         # Generate samples from checkpoints during training
├── analyze_dataset.py                     # Dataset analysis and validation
├── clean_dataset.py                       # Dataset cleaning and filtering
├── prepare_dataset.py                     # Create train/validation splits
├── start_training.ps1                     # PowerShell script to start training
├── stop_training.ps1                      # PowerShell script to stop training
├── start_sample_watcher_refine.ps1        # Watch training with sample generation
├── watch_training_refine.ps1              # Monitor training logs
├── processed_dataset/                     # Audio dataset and metadata
│   ├── wavs/                             # Audio files (.wav)
│   ├── metadata_clean_train.csv          # Training metadata (path|text|normalized_text)
│   ├── metadata_clean_val.csv            # Validation metadata
│   ├── metadata_clean_v3.csv             # Latest cleaned dataset
│   ├── cleaning_report_v3.json           # Cleaning statistics
│   └── dataset_analysis_v3.json          # Audio quality analysis
└── training_runs/                         # Training artifacts
    └── vits_personal/                    # Personal voice models
        ├── male_refine_cleaned_lr5e5-April-05-2026_10+12PM-17d46f6/  # Training run directory
        │   ├── config.json               # VITS training configuration
        │   ├── best_model*.pth           # Best model checkpoints
        │   ├── checkpoint_*.pth          # Regular checkpoints
        │   └── trainer_0_log.txt         # Training log
        └── [other training runs]

Hardware & Requirements

• Tested Hardware: NVIDIA RTX 3050 (6GB VRAM)
• Python Version: 3.8+
• CUDA: 12.1+ (recommended)
• Storage: ~5GB for training artifacts and checkpoints

Setup

1. Create and activate a virtual environment:

python -m venv .venv
.venv\Scripts\Activate.ps1

2. Install PyTorch with CUDA support for your system.

Pick the correct command from the official PyTorch install page if your CUDA version differs. Example for CUDA 12.1:

pip install torch --index-url https://download.pytorch.org/whl/cu121

3. Install Coqui TTS:

pip install -r requirements.txt

4. For the clean project environment used here:

python -m venv .venv
.\.venv\Scripts\python -m pip install --upgrade pip
.\.venv\Scripts\python -m pip install -r requirements.txt

Quick Start

Run Pre-trained Model

Generate output.wav with the default text:

python main.py

Generate speech from custom text:

python main.py --text "Hello from Coqui TTS running on the GPU."

Write to a different file:

python main.py --text "Custom output file example." --output demo.wav

Notes on Pre-trained Inference

• The script prints whether it is using cuda or cpu.
• If CUDA is available, the model is moved to the GPU automatically.
• The default model is tts_models/en/ljspeech/tacotron2-DDC.

Personal Voice Training Pipeline

Your dataset lives in processed_dataset/ and is already validated for training:

• processed_dataset/wavs/ - Audio files
• processed_dataset/metadata_clean_v3.csv - Latest cleaned metadata

Step 1: Analyze Your Dataset

Analyze the dataset and write a JSON report:

python analyze_dataset.py --metadata processed_dataset/metadata_clean_v3.csv --report processed_dataset/dataset_analysis_v3.json

Output includes:

• Total sample count
• Audio duration statistics (min/avg/max)
• RMS levels (volume quality)
• Text statistics (length, duplicates, artifacts)
• Flagged problematic rows

Step 2: Clean Your Dataset

Create a cleaned metadata file that removes noisy/low-quality samples based on:

• Text patterns (chapter titles, system messages, OCR artifacts)
• Audio metrics (duration, RMS level)

python clean_dataset.py --min-duration 2.0 --max-duration 9.0 --min-rms 0.1 --output processed_dataset/metadata_clean_v3.csv --report processed_dataset/cleaning_report_v3.json

Parameters:

• --min-duration: Minimum clip length in seconds (default: 1.2)
• --max-duration: Maximum clip length in seconds (default: 10.5)
• --min-rms: Minimum RMS level to filter quiet clips (default: 0.0)

Step 3: Prepare Dataset Splits

Prepare deterministic train/validation splits:

python prepare_dataset.py --metadata processed_dataset/metadata_clean_v3.csv --train-out processed_dataset/metadata_clean_train.csv --val-out processed_dataset/metadata_clean_val.csv

This creates:

• processed_dataset/metadata_clean_train.csv - Training samples (~98%)
• processed_dataset/metadata_clean_val.csv - Validation samples (~2%)

Step 4: Validate Training Configuration

Generate and validate the VITS training config without starting training:

python train_vits.py --prepare-only

Step 5: Start Training

Start single-speaker VITS training on GPU:

python train_vits.py

The training script automatically prefers metadata_clean_train.csv and metadata_clean_val.csv when they exist.

Useful options for RTX 3050 6GB:

python train_vits.py --batch-size 2 --grad-accum 1 --workers 0 --epochs 500

Common Parameters:

• --batch-size: Batch size per step (default: 2, for 6GB VRAM)
• --epochs: Total training epochs (default: 500)
• --save-step: Save checkpoint every N steps (default: 500)
• --save-n-checkpoints: Keep N recent checkpoints (default: 5)
• --lr-gen: Generator learning rate (default: 2e-4)
• --lr-disc: Discriminator learning rate (default: 2e-4)
• --grad-clip: Gradient clipping value (default: 1.0)

Training artifacts are written under training_runs/vits_personal/.

Training Management Commands

Start Training on Cleaned Data

python train_vits.py --restore-path training_runs/vits_personal/male_refine_cleaned_lr5e5-April-05-2026_10+12PM-17d46f6/best_model_94516.pth --run-name male_refine_cleaned_v3 --device-index 0 --save-step 500 --save-n-checkpoints 5 --batch-size 2 --grad-accum 1 --workers 0 --grad-clip 1.0 --lr-gen 0.00005 --lr-disc 0.00005 --epochs 1000

Monitor Training Progress

Get-Content training_runs\vits_personal\male_refine_cleaned_v3*\trainer_0_log.txt -Wait -Tail 20

Stop Training

.\stop_training.ps1

Generate Preview Samples During Training

Generate checkpoint preview samples every 2000 steps from a run:

python generate_checkpoint_samples.py --run-dir training_runs\vits_personal\male_refine_cleaned_v3*\* --every-steps 2000

Keep watching a run and auto-generate new preview samples as training continues:

python generate_checkpoint_samples.py --run-dir training_runs\vits_personal\male_refine_cleaned_v3*\* --every-steps 2000 --watch

Inference From Your Trained Voice

Generate Speech with Newest Model

Generate speech with the newest trained run:

python infer_trained.py --text "This is my trained voice model speaking."

Save to Custom File

Write to a custom file:

python infer_trained.py --text "Custom output example." --output my_voice.wav

Use Specific Model Checkpoint

Use a specific run directory:

python infer_trained.py --run-dir training_runs\vits_personal\male_refine_cleaned_v3*\* --text "Inference from a chosen checkpoint."

Dataset Quality Metrics (Latest)

Current dataset statistics after cleaning:

• Total Samples: 1,432 (after filtering)
• Training Samples: 1,404
• Validation Samples: 28
• Audio Duration: 2.0-9.0 seconds (avg: 5.1 sec)
• RMS Level: 0.100-0.139 (avg: 0.113) - indicates good volume
• Text Length: 24-158 characters (avg: 80 chars)
• Issues Found: 0 flagged rows (fully cleaned)
• Duplicates: 4 (acceptable)

Training Configuration (VITS Model)

Audio Parameters

• Sample Rate: 24 kHz
• FFT Size: 1024
• Hop Length: 256
• Mel Bins: 80
• Mel Frequency Range: 0 - 12 kHz

Model Parameters

• Model: VITS (single speaker)
• Batch Size: 2 (optimized for RTX 3050)
• Precision: FP16 (mixed precision training)
• Optimizer: AdamW
• Learning Rate: 2e-4 (generator), 2e-4 (discriminator), refined to 5e-5 for fine-tuning

Training Details

• Current Status: Continuing from ~96.5k steps
• Target: 100k+ steps for optimal quality
• Latest Checkpoint: best_model_94516.pth (54k parameters)

Troubleshooting

CUDA Out of Memory

• Reduce --batch-size (try 1)
• Reduce --epochs per checkpoint
• Enable gradient accumulation with --grad-accum > 1

Training Not Starting

• Verify dataset: python analyze_dataset.py
• Check config with: python train_vits.py --prepare-only
• Ensure metadata files exist in processed_dataset/

Poor Voice Quality

• Increase training steps (run to 100k+)
• Re-clean dataset with stricter RMS threshold
• Verify audio quality: check RMS levels in analysis report
• Reduce learning rate for fine-tuning

Inference Produces Noise

• Use checkpoint from more training steps
• Verify dataset cleanliness with analysis report
• Try different checkpoints from training run

Dependencies

See requirements.txt for complete list:

• TTS >= 0.22, < 0.23
• PyTorch (with CUDA support)
• soundfile (for audio I/O)
• numpy (for numerical operations)

Contributing

This is a personal training project. Modifications are welcome for experimenting with:

• Different model architectures
• Custom hyperparameters
• Dataset preprocessing techniques
• Training optimization strategies