From acoustic vocabularies to online musical inference.
Today, much of music AI is dominated by very large models trained on enormous datasets. In many cases, these systems treat audio primarily as a statistical prediction problem: enough data, enough parameters, enough compute, and useful behaviour eventually emerges.
But many of the core principles behind modern representation learning existed long before the current wave of large-scale generative models. Deep learning itself did not suddenly appear in the 2010s but evolved gradually over decades through earlier work on multilayer neural architectures, convolutional processing, distributed representations, and hierarchical feature extraction.
One early research milestone in the development of the system that eventually evolved into Algoriffix was a 2018 thesis exploring convolutional nonnegative matrix factorisation (CNMF) for speech and audio analysis.
In retrospect, the thesis can be viewed as an attempt to push sparse dictionary learning beyond static shallow representations by introducing convolutional structure, multilayer organisation, and online optimisation of activations during inference itself.
Structured Sound
Musical audio is not random.
Speech, musical tones, and acoustic events already possess partially understood physical and perceptual structure. Speech, for example, can often be approximated through the source-filter model. Musical tones, meanwhile, often exhibit harmonic structure even though the relative strengths of harmonics may evolve rapidly over time due to resonance effects, wave interference, excitation dynamics, and interactions between the strings and the instrument body.
This makes it possible to engineer parts of the acoustic vocabulary directly into the representation itself rather than learning every structure purely from massive datasets.
In sparse dictionary learning, the dictionary acts as a vocabulary of possible acoustic structures, while the activations dynamically combine those structures to explain the incoming signal. Much like words and sentences are formed from a limited vocabulary, complex audio structures can emerge from combinations of simpler acoustic components.
The convolutional variants explored in the thesis extended this idea further by allowing structures to evolve locally through both time and frequency. Instead of representing sound as static templates, the model could capture short temporal behaviours and local spectro-temporal patterns more naturally.
Nonnegative Representations
The nonnegativity constraints were also important.
Spectrograms are physically nonnegative because they represent magnitude or energy distributions. Negative acoustic energy does not exist in this representation. By constraining both the dictionary and the activations to remain nonnegative, the decomposition becomes additive rather than subtractive.
This often leads to representations that are substantially more interpretable than unconstrained decompositions. Individual activations begin to resemble positive evidence accumulation or local component activations rather than arbitrary signed coefficients in an abstract vector space.
Interestingly, modern deep neural networks often impose similar nonnegative constraints implicitly through activation functions such as ReLU, which suppress negative activations between layers.
The nonnegativity constraints also enable multiplicative optimisation updates. Unlike conventional additive gradient updates, multiplicative updates preserve nonnegativity automatically while effectively adapting step sizes elementwise during optimisation.
In practice, this often leads to substantially faster convergence and makes iterative online inference feasible for low-latency audio systems. These constraints become especially important in live systems where inference itself must happen causally and under strict latency constraints. We explore these trade-offs further in Time, Frequency, and the Limits of Real-Time Polyphonic Pitch Detection.
Online Inference Instead of Static Prediction
One of the more interesting aspects of the thesis was the emphasis on online optimisation during inference itself.
Rather than treating the representation as a fixed output of a purely feed-forward model, the system continuously refined its internal activations while the signal was arriving frame by frame in real time. Multiple competing constraints were balanced simultaneously:
- spectral reconstruction,
- sparsity,
- temporal consistency,
- and perceptual plausibility.
The goal was not merely to classify the signal, but to converge toward a musically meaningful explanation of it.
This creates a fundamentally different balance between representation and inference.
In large feed-forward systems, most of the intelligence is compressed into fixed model weights during offline training. If a relatively small pretrained model is trained on unrelated data, it may fail catastrophically because its statistical assumptions no longer match the incoming signal.
An optimisation-based system behaves differently. Even with a relatively compact dictionary, the system can still construct plausible representations dynamically during runtime, provided the underlying vocabulary remains meaningful and structurally aligned with the physics and perception of sound.
The dictionary provides the vocabulary; the activations construct words and sentences from it.
One practical application of this type of online inference is software-based guitar-to-MIDI conversion using ordinary instrument signals without specialised pickup hardware. A more accessible overview can be found in How to Convert Your Guitar to MIDI in Real Time — Without Special Hardware.
Multiple Layers and Hierarchical Audio Structure
The thesis also explored multilayer variants of convolutional NMF.
Lower layers captured local spectro-temporal structures, while higher layers organised increasingly abstract patterns emerging from those activations. In retrospect, this resembles several ideas that later became central in hierarchical representation learning.
But there was also an important philosophical difference.
The intermediate representations remained relatively interpretable. Activations still corresponded to meaningful structures inside the signal itself rather than to opaque latent spaces learned purely through large-scale statistical training.
This created a system that behaved less like a static predictor and more like a structured inference process operating continuously on incoming audio.
From Research Prototype to Algoriffix
The system used today by Algoriffix has evolved substantially since that thesis was written. Some ideas were abandoned, others refined, and many components became significantly more sophisticated over time.
But several core principles remained:
- structured representations,
- iterative inference,
- online optimisation,
- perceptually informed constraints,
- and an architecture designed specifically for live musical interaction.
In that sense, the thesis is best understood not as a finished system, but as an early snapshot of a longer research direction.
Why Structure Still Matters
Large-scale machine learning has unquestionably transformed audio technology. But music remains deeply structured, highly temporal, and tightly connected to human perception.
For live musical systems especially, intelligence is not purely a matter of larger datasets or larger parameter counts. Responsiveness, causality, interpretability, and perceptual stability still matter.
As music AI increasingly becomes commoditised, many of these older ideas around structured inference, sparse representations, online optimisation, and physically informed vocabularies may become relevant once again — particularly for systems designed not merely to generate audio, but to interact with musicians in real time.
Leave a Reply