Deep convolutional architectures with multiplicative factor updates.
Deep learning did not emerge from a single breakthrough. Long before today’s large-scale AI systems became dominant, researchers were already exploring multilayer representations, convolutions, sparse coding, and hierarchical feature extraction across several overlapping research traditions (see also Before Music AI Became a Commodity).
One of these directions emerged from non-negative matrix factorisation (NMF) and sparse dictionary learning.
A 2020 master’s thesis supervised by the founder of Algoriffix explored what happens when these ideas are extended into deep convolutional architectures while preserving non-negative representations throughout the network. The result was a mathematically grounded framework for deep convolutional non-negative autoencoders whose principles later influenced parts of the system used by Algoriffix today.
From Matrix Factorisation to Deep Architectures
At its core, NMF factorises a non-negative observation matrix into a dictionary and an activation matrix.
In audio analysis, the dictionary behaves like a vocabulary of possible acoustic structures such as harmonics, transients, local spectro-temporal patterns, and typical frequency spectra. The activations then combine these structures into an additive interpretation of the observed signal.
Because spectrograms themselves are non-negative, the resulting decompositions remain additive. This often leads to representations that remain surprisingly interpretable even as the structures become increasingly hierarchical.
NMF can also be interpreted through the probabilistic framework of latent semantic analysis, where the observed signal corresponds to visible variables, the activations behave as latent representations, and the dictionary reconstructs the signal through additive combinations of learned components.
The thesis explored how these principles could be generalised into non-negative autoencoders, convolutional non-negative autoencoders, and eventually deep convolutional non-negative autoencoders.
Importantly, positivity constraints were preserved throughout the architecture itself, including inputs, activations, and weights.
Deep Convolutional Non-Negative Representations
The convolutional structure allowed the representations to evolve locally through time and frequency, while deeper layers gradually organised increasingly abstract structures emerging from the signal itself.
This introduced hierarchical organisation and end-to-end optimisation across multiple layers.
But the representations remained additive, non-negative, and relatively interpretable throughout the hierarchy. Rather than compressing information into arbitrary latent embeddings, the system continuously decomposed the signal into additive combinations of meaningful components.
In that sense, the work occupied an unusual position between sparse dictionary learning, matrix factorisation, and deep neural architectures.
Gradient Descent with Multiplicative Factor Updates
One of the elegant mathematical properties of NMF is that non-negativity allows gradient descent to be reformulated using multiplicative factor updates.
Ordinary gradient descent is additive. But once variables are constrained to remain positive, the optimisation updates can instead be expressed multiplicatively while preserving non-negativity automatically.
The thesis generalised these optimisation principles into deep convolutional non-negative autoencoder architectures trainable end-to-end through forward propagation.
This produced optimisation dynamics that remained stable, non-negative, and structurally consistent throughout the hierarchy. The multiplicative factor updates also reduced sensitivity to fixed learning rates by adapting the effective step size during optimisation itself. If additive updates become too large, the optimisation may oscillate around a saddle point without converging.
From Layer-by-Layer Optimisation to End-to-End Training
Earlier multilayer convolutional NMF systems often relied on sequential layer-by-layer optimisation in which each layer was trained independently.
The thesis instead generalised the framework into unified end-to-end optimisation. This moved structured non-negative factorisation architectures conceptually closer to deep neural networks while still preserving several important characteristics that distinguished them from conventional deep learning, including additive representations, positivity constraints, structured decompositions, and relatively interpretable intermediate layers.
The result was a mathematically coherent deep architecture built around structured non-negative representations rather than unconstrained latent spaces.
Feedforward vs Feedback Inference
The thesis also highlighted an important conceptual distinction between conventional feedforward neural networks and NMF-like inference systems.
In ordinary feedforward networks, activations are typically produced through a single forward pass using fixed learned weights.
In NMF-like systems, however, the latent activations are iteratively refined to interpret the observed signal itself. Rather than retrieving predictions directly from frozen parameters, the internal representation continuously adapts during inference.
This creates a fundamentally different relationship between representation, optimisation, and inference. Similar constraints emerge in real-time audio systems discussed in Time, Frequency, and the Limits of Real-Time Polyphonic Pitch Detection.
Towards Non-Negative Autoencoders
Rather than treating matrix factorisation and deep neural networks as fundamentally separate paradigms, the work demonstrated that several principles associated with hierarchical neural architectures could emerge naturally from structured non-negative representations and multiplicative factor updates.
In that sense, deep convolutional non-negative autoencoders were an attempt to unify sparse dictionary learning, convolutional representations, and deep hierarchical inference inside a single mathematically coherent framework. Some of these ideas later evolved further in systems for real-time musical interaction explored in How to Convert Your Guitar to MIDI in Real Time — Without Special Hardware.
Leave a Reply