/projects/conditional-wavenet-audio/
Conditional WaveNet Audio Generation
How can raw audio be modeled one sample at a time while still controlling the instrument family and pitch it generates?
- Python
- PyTorch
- librosa
- NumPy
- NSynth
- μ-law companding
- autoregressive modeling
- signal processing
Overview
A set of PyTorch experiments in raw-waveform generation: WaveNet-style models learn next-token prediction over audio, with a conditional variant that uses NSynth instrument-family and pitch metadata to steer generation.
Input
- 16 kHz mono audio
- Instrument-family and pitch labels
Processing
- Trim and split audio into one-second chunks
- Encode samples into 256 μ-law classes
- Train and sample a conditioned dilated residual model
Output
- Decoded generated waveform
Problem
Raw audio sampled at 16 kHz produces a long sequence even for a one-second sound. Modeling it autoregressively requires a representation with manageable output classes, a receptive field that can capture temporal structure, and a way to inject labels without discarding the sample-level generation objective.
Approach
Audio is loaded as mono at 16 kHz, trimmed for leading and trailing silence, divided into one-second chunks, and compressed into 256 discrete values with μ-law encoding. The model predicts each next token from earlier tokens. The conditional architecture embeds instrument family and pitch, then injects that context into gated, dilated residual blocks.
Technical Implementation
The repository includes minimal and conditional WaveNet implementations, PyTorch Dataset wrappers, preprocessing utilities, and training loops. The conditional model combines one-by-one input and output projections with residual blocks using dilation rates 1, 2, 4, and 8; training uses cross-entropy, CUDA automatic mixed precision, checkpoints, loss curves, entropy diagnostics, and predicted-token histograms. Generation samples one token at a time from a temperature-scaled distribution and decodes the sequence back to a waveform.
Evaluation & Results
This is an experimental implementation rather than a benchmarked audio model, so the repository does not support a quantitative audio-quality claim. Its diagnostics track training loss, output entropy, and token distributions, while generated samples can be auditioned and inspected as spectrograms to identify collapse or poor conditioning behavior.
What I Learned
Sample-level generation makes computational cost and receptive-field design impossible to ignore: even one second requires thousands of sequential predictions. The project also made the boundary between preprocessing and modeling concrete, because companding, chunk length, conditioning labels, and sampling temperature all directly shape what the network can learn and produce.