Why latency remains one of the hardest problems in modern audio-to-MIDI systems.
Latency remains one of the central challenges in real-time audio-to-MIDI systems, not because modern computers are too slow, but because musical information itself takes time to emerge from sound.
Windows, Context, and Causality
A real-time system cannot analyse an entire signal at once because the future has not happened yet. Incoming audio arrives sample by sample and must therefore be processed in small blocks. These blocks are typically added to a slightly longer buffer, often referred to as a window.
The idea is somewhat analogous to looking through a physical window. Only a limited portion of the surrounding world is visible at any given moment, and the size and shape of that window determine how much can be observed simultaneously. A similar concept exists in convolutional neural networks, where the receptive field determines how much local structure can be perceived by the model. The role of the analysis window in classical signal processing is therefore conceptually similar to the receptive field in neural architectures. In both cases, the system can only make decisions based on the information available within a limited observation range.
The Trade-Off Between Time and Frequency
In signal processing, the shape of the analysis window is defined by a window function. This function weights the incoming signal over time and determines how strongly neighbouring samples contribute to the analysis. Many commonly used window functions are designed to satisfy overlap-add conditions, allowing smooth reconstruction without discontinuities. This forms the basis of spectral or time-frequency analysis, including the short-time Fourier transform used in many modern audio systems.
The difficulty begins with the fundamental trade-off between temporal and frequency resolution. According to Heisenberg’s uncertainty principle, one cannot simultaneously achieve arbitrarily high precision in both time and frequency. Improving one necessarily degrades the other. Short windows or short receptive fields improve responsiveness because less future information is required before producing an estimate. Longer contexts improve stability and allow more reliable interpretation of harmonic structure, but they also increase delay.
For onset detection, high temporal resolution is essential. The system must respond quickly to the transient attacks produced by the interaction between the string and the pick or finger. This favours shorter windows because they localise events more precisely in time.
Pitch detection presents the opposite problem. Estimating pitch accurately requires fine frequency resolution, which in turn favours longer windows. The required window length depends largely on the lowest frequency that must be resolved and its spectral distance from neighbouring notes. Lower notes require longer observation times because their oscillatory periods are physically longer and more difficult to distinguish spectrally.
In practice, however, the situation is more complicated because the first milliseconds of a plucked string rarely contain stable pitch information. Instead, the onset behaves more like a short burst of broadband noise mixed with unstable harmonics generated by the interaction between the string and the finger. Humans intuitively perceive this as the attack of the note, but for a pitch-detection system it creates considerable ambiguity.
During this early transient phase, short-window pitch estimates are often unstable or incorrect. If the system reacts immediately using these preliminary estimates and later corrects them once more reliable information becomes available, the resulting pitch shifts can produce audible artefacts and unnatural behaviour.
These limitations become especially relevant in software-based guitar-to-MIDI systems that attempt real-time polyphonic analysis using only standard instrument signals and no specialised hardware. A more practical overview of this problem can be found in How to Convert Your Guitar to MIDI in Real Time — Without Special Hardware.
Symmetric and Low-Delay Windows
Waiting for a longer window introduces a different problem. Symmetric windows possess several mathematically desirable properties, including good sidelobe attenuation, smooth spectral behaviour, and strong frequency discrimination. These characteristics improve the frequency selectivity of spectral analysis.
The centre of the window effectively acts as a balance point in time. For symmetric windows, that balance point lies near the middle, meaning the analysis becomes temporally aligned around the centre of the window rather than the present moment. As a result, the system effectively incurs a delay of roughly half the window length.
For pitch detection, one would ideally like a window that is simultaneously highly frequency-selective and strongly suppressive of sidelobes. In practice, these goals compete with one another. Improving frequency selectivity generally narrows the main lobe but worsens sidelobe behaviour, while stronger sidelobe attenuation often broadens the main lobe and reduces spectral precision.
Many pitch-detection systems therefore prioritise main-lobe selectivity because accurate frequency discrimination is usually more important than maximal sidelobe suppression.
The situation becomes more complicated when low latency is required. A long symmetric window may provide good spectral behaviour, but its group delay quickly becomes perceptible during performance. Various approaches exist for designing low-delay analysis windows, although a full discussion lies beyond the scope of this article.
What these approaches generally share is asymmetry. Low-delay windows shift more of their analytical weight towards the present moment, reducing latency while still retaining enough temporal support for reliable pitch estimation.
This asymmetry introduces new compromises. Compared to symmetric windows, low-delay windows typically exhibit weaker sidelobe attenuation and less smooth spectral behaviour, increasing spectral leakage from neighbouring tones. In musical signals containing dense harmonics or closely spaced pitches, this leakage can make stable pitch estimation considerably more difficult.
No single window therefore solves the problem completely.
Multi-Resolution and Time-Domain Approaches
One practical compromise is multi-resolution analysis, in which shorter windows are used to detect note onsets rapidly while longer windows refine the pitch estimate afterwards. Time-aligning data between multiple resolutions is the main challenge here, aside from the increased computational burden.
Polyphonic pitch detection can be designed directly in the time domain using predefined waveform templates representing notes or harmonic structures. Conceptually, this resembles a wavelet or dictionary-based representation in which each template acts as a localised musical atom. Correlating these templates with the incoming signal yields coefficients that can be interpreted as activations of notes or chords, much in the same way that Fourier or wavelet transforms project signals onto basis functions.
Sparse approximation techniques such as matching pursuit can then be used to represent the signal efficiently using only a subset of the available dictionary elements. These approaches can work remarkably well because the templates may implicitly encode musical structure that is difficult to capture using simpler spectral methods alone.
The downside is computational cost. Unlike the FFT, which benefits from highly optimised fast algorithms, arbitrary template dictionaries generally require explicit convolution or correlation operations across large numbers of candidate patterns. For real-time polyphonic analysis, especially at low latency, this quickly becomes expensive.
Spectral Compression and Neural Inference
One way to simplify the problem is to move into the frequency domain early using a short-time Fourier transform. Spectral representations are often more convenient for subsequent processing than raw waveform data, particularly when neural networks or convolutional operations are involved.
A critically sampled FFT does not magically compress the signal; it mainly reorganises the same information into a different representation. The computational advantage emerges when the spectral representation is reduced to a more compact form with non-uniform frequency resolution. Rather than preserving uniformly spaced frequency bins across the entire spectrum, the representation can be compressed to reflect human auditory perception more closely, allocating higher resolution where it matters and lower resolution elsewhere.
This becomes especially useful for machine learning when neural inference must happen in real time.
Neural architectures introduce two different constraints in live systems. The first is receptive-field size, which determines how much signal context must be accumulated before a reliable prediction can be made. Longer temporal contexts may improve accuracy or stability, but they also increase unavoidable causal delay because the required information has not yet fully unfolded in time.
The second constraint is computational depth. Every additional network layer increases the amount of computation required before an output can be produced. Unlike receptive-field size, this delay is not fundamentally tied to the structure of the signal itself, but to processing cost and hardware limitations.
Although modern processors can execute very large networks efficiently in offline settings, live musical interaction places much stricter limits on acceptable inference time. Low-latency systems therefore tend to favour relatively shallow architectures with limited temporal context.
In practice, the problem of low-latency polyphonic pitch detection is therefore not simply a matter of faster hardware or deeper neural networks. It is fundamentally constrained by the structure of sound itself and by the trade-offs imposed by time-frequency analysis. Real-time musical interaction requires systems that balance responsiveness, spectral precision, perceptual stability, and computational efficiency simultaneously.
Real-Time Computing vs Live Computing
This also highlights an important distinction between real-time computing and live computing. In conventional real-time systems, the objective is often deterministic scheduling: computations must complete before a specified deadline. Faster hardware can usually reduce the likelihood of missing those deadlines.
Live musical systems operate under additional constraints. Some delays arise from computational complexity, such as deep neural architectures or expensive signal-processing pipelines. Others are fundamentally causal. Onset detection, pitch estimation, and harmonic interpretation all depend on observing temporal structure that unfolds over time. Even an infinitely fast processor cannot estimate information that has not yet emerged in the signal itself.
In that sense, live audio systems are constrained not only by processing speed, but also by causality. The challenge is therefore not simply to compute faster, but to design systems that behave intelligently under conditions of incomplete information. That balancing act remains one of the more interesting problems in modern music technology.
These constraints also help explain why some modern music-AI systems increasingly favour structured inference, compact representations, and online optimisation over purely large-scale feed-forward architectures. We discuss this design philosophy further in Beyond Big Data: A Different Philosophy for Intelligent Music Technology.
Leave a Reply