By Tania Alwan

Sing "ah" and then "ee" on the same note and anyone can hear the difference, even though the pitch never changed. Sing "ah" on two different notes and it is still clearly the same vowel. Speech and music software needs a way to capture that difference in numbers, and for decades the go-to answer has been MFCCs: mel-frequency cepstral coefficients. This post walks through what they are, how they are computed, and ends with a short Python script that shows MFCCs caring more about the vowel than the note.

What an MFCC actually describes

An MFCC vector is a compact summary of the shape of the spectrum in one short frame of audio, usually 13 to 20 numbers per frame.

For a voice, that shape is mostly set by the vocal tract: the throat, mouth and lips act like a filter that boosts some frequency regions (the formants) and damps others. That filter is what makes an "ah" an "ah" and gives each singer part of their recognisable colour. The pitch, on the other hand, shows up as the fine spacing of the harmonics, and MFCCs are designed to largely smooth that detail away.

So, roughly:

  • Spectral envelope (timbre, vowel, instrument colour): kept
  • Harmonic fine structure (exact pitch): mostly discarded
  • Overall loudness: packed into the first coefficient, c0

The pipeline, step by step

The classic recipe, popularised for speech recognition by Davis and Mermelstein in 1980, looks like this:

  1. Frame and window the signal into short overlapping chunks (often 20 to 40 ms), so each chunk is roughly stationary
  2. Take the FFT of each frame and keep the power spectrum
  3. Apply a mel filterbank: a set of overlapping triangular filters spaced on the mel scale, which is roughly linear at low frequencies and logarithmic higher up, a bit like human pitch perception
  4. Take the logarithm of each filter's energy, because loudness perception is closer to logarithmic than linear
  5. Apply a discrete cosine transform (DCT) to the log-mel energies and keep the first coefficients

The last step is the "cepstral" part. The DCT re-expresses the smooth log-mel curve as a sum of cosine shapes. Low coefficients describe the broad outline of the spectrum (where the energy sits, how fast it tilts off), higher ones describe finer wiggles. Keeping only the first dozen or so acts as a smoothing filter: the envelope survives, the harmonic ripple caused by pitch mostly does not.

Some pipelines also add a pre-emphasis filter before framing, and delta and delta-delta features afterwards, which describe how the coefficients change over time. In librosa those are librosa.feature.delta(mfcc) and librosa.feature.delta(mfcc, order=2).

The Python example

The script below builds three synthetic "vowels" from harmonics shaped by two Gaussian formant peaks, using rough textbook formant positions for "ah" and "ee". It then compares the average MFCC vectors (without c0, so loudness does not count).

import numpy as np import librosa sr = 22050 t = np.arange(int(sr * 1.0)) / sr def vowel(f0, formants): """Harmonic tone at f0 shaped by a simple formant envelope.""" y = np.zeros_like(t) for k in range(1, int(5000 / f0)): f = k * f0 amp = sum(np.exp(-0.5 * ((f - fc) / 80) ** 2) for fc in formants) + 0.01 y += amp * np.sin(2 * np.pi * f * t) return y / np.max(np.abs(y)) def mfcc_profile(y): m = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=13) return m[1:].mean(axis=1) # drop c0 (overall loudness), average over time AH, EE = (700, 1200), (300, 2300) # rough formants of "ah" and "ee" sounds = { "ah @ 220 Hz": vowel(220, AH), "ah @ 330 Hz": vowel(330, AH), "ee @ 220 Hz": vowel(220, EE), } ref = mfcc_profile(sounds["ah @ 220 Hz"]) for name, y in sounds.items(): d = np.linalg.norm(mfcc_profile(y) - ref) print(f"{name}: distance to 'ah @ 220 Hz' = {d:.1f}") 

Output (tested with librosa 1.0.1 and numpy 2.5.4):

ah @ 220 Hz: distance to 'ah @ 220 Hz' = 0.0 ah @ 330 Hz: distance to 'ah @ 220 Hz' = 59.9 ee @ 220 Hz: distance to 'ah @ 220 Hz' = 166.7 

Moving the same vowel up a perfect fifth changes the MFCC profile a little. Changing the vowel on the same note changes it almost three times as much. That is the whole point of the feature: it tracks the filter (the vowel) more than the source (the pitch). The distance is not zero for the higher note because the harmonics sample the envelope at different points, which is also why MFCCs are not perfectly pitch-independent on real voices.

To try it on a recording, replace the synthetic signal with y, sr = librosa.load("vocal.wav") and plot librosa.feature.mfcc(y=y, sr=sr) with librosa.display.specshow.

Where MFCCs are used

  • Speech recognition: they were the standard input to classic HMM-GMM recognisers for decades
  • Speaker identification and verification: comparing MFCC statistics is a classic baseline for "who is speaking"
  • Music information retrieval: genre, instrument and timbre similarity tasks often start with MFCCs
  • Audio fingerprinting of sound events: quick, compact features for classifiers on small datasets

Limitations worth knowing

  • Pitch is thrown away on purpose. For melody, key or vibrato analysis, use a pitch tracker such as librosa.pyin instead.
  • Noise and reverb shift the numbers. Background hum or a roomy recording changes the spectral envelope, so the features no longer match a clean reference.
  • Mixtures confuse them. A frame with voice, guitar and drums gives one blended envelope. Separating stems first gives far cleaner features.
  • Neural models often skip the DCT. Many modern speech and music models take log-mel spectrograms directly and learn their own representation, since the decorrelation the DCT provides matters less for deep networks.

The noise point is very practical. Any system that learns a voice, from a speaker-ID baseline to a modern voice-cloning model, does better with clean, dry, solo recordings. That is why services that train a model of your singing voice, such as Voicestars, ask for clear recordings with minimal background noise, and why open-source projects give similar advice for their training data.

Takeaways

  • MFCCs summarise the spectral envelope of a short frame: mel filterbank, log, then DCT.
  • They capture timbre and vowels well and largely ignore exact pitch, as the demo shows.
  • Drop or normalise c0 when you care about colour rather than loudness, and add deltas for dynamics.
  • Clean, isolated input matters more than any parameter tweak.

Disclosure: this article was written in partnership with Voicestars.