Listening Tests by Nawhead and bAdDuDeX
Notes by David Robinson
Additional notes by ff123
Revised 6/20/01: David Robinson clarified how MAD 0.13.0b behaves with clipped signals.
MAD is a 24-bit MPEG audio decoder. There has been debate, sometimes heated, on the claimed benefits (or flaws) of this decoder for mp3 files compared with Fraunhofer's decoder (for example, as found in Winamp 2.666 and later). Summary of this page: Nawhead satisfies me that he hears a difference between the 24-bit output of MAD versus the default Fraunhofer 16-bit decoder in Winamp 2.7. bAdDuDeX hears more clipping distortion from the MAD decoder (using an earlier version without clipping attenuation) than with the default Winamp 2.666 decoder. Nawhead subsequently verifies this in an independent listening test. David Robinson verifies that MAD 0.13.0b produces more clipping than other decoders when dithering is enabled (which is all the time in the Winamp plugin) and when auto-attenuation is disabled. If dithering is disabled (only possible with the command line version of MAD, madplay.exe), clipping is no worse than other decoders. If dithering is enabled, but auto-attenuation is also enabled, playback clipping distortion is avoided.
Bottom line: For MAD version 0.13.0b, If you use the MAD Winamp plugin, and you listen to mp3's which clip, you should have the auto-attenuation feature enabled. If auto-attenuation is not enabled, audible clipping distortion will be worse than with standard decoders. On the other hand, if auto-attenuation is enabled, there will be less audible clipping distortion than with standard decoders (I test this assertion on this page).
Nawhead listens to MAD at 24 bits versus the Fraunhofer decoder in Winamp at 16 bits
Nawhead (12/8/00), in the headwize.com Audio Technologies Forum, performed a listening test, the results of which follow (original posts are here):
ff123, though I can't satisfy you with an ABX test, I can give you the results of another simple blind test I did just now. I was comparing the 24-bit output of MAD versus the default Fraunhofer decoder in Winamp 2.7. You might be asking just how blind the test was. Well, I just opened up two instances of Winamp, set one to use MAD and the other to use FhG, loaded up the same playlist, and put them both in windowshade mode. I turned off all visualizations and hid my taskbar. After each test run, I scrambled up the order of the two Winamp windowshade bars as much as possible. I guess you'll just have to take my word that I didn't cheat myself out of my own "blind" test.
For equipment. I was using my Audiophile 2496 sound card connected to Monsoon MH-500 speakers.
On regular pop and electronic music, I got my guesses right 50% of the time. I thought one sounded better than the other and assumed it was MAD, but I wasn't too consistent. So basically I was just guessing.
However, on classical and jazz, I was right about 92% of the time. When I stopped, I was at 13/14 correct guesses. On some pieces, all it took was just a few seconds of the opening strings to tell right away. MAD's 24-bit output gives real instruments a weight, resonance and natural decay you expect. In comparison, the Fraunhofer decoder sounds quite artificial. It's too snappy, almost MIDI-like. You can't notice it as much on pop and electronic instruments, and in some instances, it almost sounds better. But with unsynthesized music, MAD truly shines. Hope my little blind test helps.
--
I have a M Audio Audiophile 24/96 sound card. Also, I don't know which particular MP3 encoder I used for my test files, but it was either Fraunhofer (Opticom .mp3 producer), Proteron encoder found in EasyMP3, or LAME 3.87. But the files varied from 128kbps to 192kbps. So it's not relegated to any particular codec or bitrate.
Feel free to quote me all you want. MAD's the best thing in MP3 since MP3 itself! :)
In my opinion, the classical/jazz test shows that Nawhead reliably detected a difference between the two codecs (p = 0.001). Or to put it another way, there is only a 1-in-a-thousand chance that random guessing could have produced the same results. I should add the caveat that this assumes that Nawhead wasn't somehow biasing his results unconsiously. The benefits of the MAD decoder with 16-bit audio cards has not been evaluated with this test.
bAdDuDeX hears more clipping distortion from MAD
bAdDuDeX has also performed listening tests. The following is an excerpt from a Usenet post (11/5/00) to alt.binaries.sounds.mp3.d (Google Groups archive is here). bAdDuDeX is responding to a post by Nawhead and is referring to the song "Liberate" by Slipknot.
Nevermind. I didn't test a file encoded with MP3Enc before and it turns out there's many more differences with a song encoded with it than one with LAME. However, the results aren't good. So you don't try to accuse me of looking at graphs and hearing things (or whatever), let me tell you the story.
After I saw Liberate with all those differences I was just listening to random files with MAD and Nitrane 2.666 just to see if maybe some files caused that. I was listening to Crowbar - Broken Glass (encoded with MP3Enc) and heard a good amount more clipping with MAD than Nitrane. So I mix pasted them and it turned out similar to how your graph of Liberate did. Then I looked at the possibly clipped samples between Nitrane 2.666 and MAD (which I don't really trust normally because it's not a fact. that's why it says "possibly") and sure enough, MAD had about 20% more clipping than Nitrane 2.666. I then listened to more files encoded with MP3Enc that I had and heard more clipping with MAD on each one. So I'm pretty much decided on Nitrane 2.666 now.
bAdDuDeX made his test clip (Crowbar - Broken Glass, 30 second excerpt encoded with MP3Enc, 128kbit/s) temporarily available as lbg.mp3 (no longer available). He followed up with a blind listening test (12/14/00, original post is on the VQF forum here):
OK, I just did the test and picked out MAD 9/10 times. The only reason I missed one of them was because blind listening tests aren't good for testing clipping actually... They're good for testing if an encoded file sounds transparent. Reason is because both files have clipping, and I have to try and remember in my brain which amount of clipping sounds like that particular decoder. It's quite hard to remember it exactly. Well, hopefully I've proved it now how MAD induces more clipping. By the way, I first found this out before generating possibly clipped samples in Cool Edit Pro anyways... I was just listening to both of them on files encoded with MP3Enc and notice how MAD produced more clipping. So I looked at what CEP generated, and sure enough it said so too.
Nawhead then confirmed bAdDuDeX's results (12/15/00, original post is on the VQF forum here):
Alright bAdDuDeX, I downloaded the clip your put up, lbg.mp3, ran through the ABX Comparator with it, and got 10/10 looking for clipping. You were right, MAD does add more clipping, and it's pretty evident with this one little clip. I could tell in the first few seconds, as the FhG decoded file has one slight series of clipped samples, but the MAD decoded file has two such series in a row (in the left channel). I guess I have to swallow all the things I said to you before and become your bitch now (doh!).
However, upon further analysis of this MP3 file, it doesn't sound all that great with either FhG *or* MAD. As you've probably noticed as well. But using the MAD pre-release 3 (given to the people on the MAD-User mailing list), and using its automatic attenuation function, I went from 4221 to 28 clipped samples at -1.5dB attenuation (as reported by MAD). Or from 1933+2292 (left+right) possibly clipped samples (as reported by Cool Edit Pro) to 7+21 possibly clipped samples. Quite an improvement. Has virtually no clipping, and sounds even better than FhG in this regard with the auto clipping attenuation turned on.
[Link to pre-release version shown at bottom of page]
David Robinson (12/13/00) explains the possible mechanisms for MAD sounding different from FhG in terms of clipping (original post is on the VQF forum here):
In Cool Edit Pro, a "possibly clipped sample" is any sample at full scale. It might not be clipped, but it probably is. If we didn't have a "my record's louder than yours" obsessed industry, there would never be a problem - no one would ever drive the levels anywhere near full scale. However, in the real world, it's a problem that we have to live with.
The best option is to normalize before encoding to prevent the problem. Another option would be to reduce the volume after decoding, assuming the decoding arrithmetic correctly handled overflows, so that the final result was quieter than the original, but incurred no more clipping than was on the original CD.
The reason we have decoding clipping is because the hard digital dynamic compression used on modern pop records means that you can filter (remove!) some sound components, and an unclipped, but very loud record will suddenly clip all over the place. CD players include oversampling fitler and do just this - and they do introduce clipping on compressed pop records. It's recognized in the industry (see AES preprint 5019 from the Sept 1999 convention in New York - Level Control in Digital Mastering - S0REN H. NIELSEN AND THOMAS LUND) but loudness is still king, and producers/mastering engineers still compress the life out of much music.
Let's assume we have an mp3 made from a loud CD, and it's going to clip. If you have a number that may "clip" (ie it may be higher than a largest possible value in a given system), if you choose to dither this number before rounding to a pre-set accuracy (typically 16-bits), then you increase the likelihood of detecting a clipped sample. However, does this increase clipping? Let's see...
Consider a sample at one less than full scale after it's been decoded. Next it's dithered, which pushes it to full scale. Cool Edit says it's clipped, but actually, it's just where it should be, plus a bit of dither. Now consider a sample at exactly full scale. It's dithered, and goes above full scale, but as this is impossible, it stays at full scale. Cool Edit Pro says it's clipped, but it's actually just where it should be again. Now consider a sample above full scale after it's been decoded. This couldn't have existed on the original CD, but now we have it, we've got to do something with it. Whether we dither it or not, it gets put at full scale, cool edit pro says it's clipped, and in truth, it is!
So, dithering increases the number of reported clipped samples, but doesn't actually cause any sample to clip that wouldn't have done anyway. You could argue that any sample dead on full scale will not be correctly dithered - but in this particular case, the dither, 96dB quieter than the signal, has momentarily vanished - That's very very subtle distortion. In effect, the truncation rounding distortion that would be present at each and every sample without dithering is only present at full scale samples when the signal has been dithered.
There are two other reasons that MAD may sound different (worse to some ears) from the FhG decoder. First, it may scale signals differently, and so cause genuine clipping. I've test this possibility, and it doesn't do this. You can't tell decisively in 16-bit mode because of the dithering, but by taking the actual 24-bit output, you can show that the output is not scaled to be louder than l3dec - or by taking the truncated (not dithered) 16-bit output you can show that the output is never more than 1-bit away from l3dec.
The other possibility is that the dithering is causing the audible difference. I've found that the dithering is increasing the dynamic range (ie letting you hear what would otherwise have been discarded below the 16th bit), but it's not ideal (Rob is aware of this). Whilst I'd personally be happier if the dither was correctly done, it makes the decoder no worse than FhG at 16-bits, and obviously gives much better results at 24-bits. l3dec at 24-bits is slightly better - though anyone who can hear the difference in an mp3 that originated from a good oldfashioned 16-bit CD gets a golden ears award!
Hope this is of some interest. I'm trying to get a decoder test update up as soon as possible!
veremont writes (12/12/00, original post on the HeadWize Audio Technologies Forum):
I hope the next release of MAD can put the clipping issue to rest. Not only will it tell you exactly how much clipping is going on, it will let you attenuate the signal to avoid clipping entirely.
[The latest version of MAD is here, complete with attenuation feature]
I test the auto-attenuation feature in the Winamp plugin version 0.13.0b on this page.
David Robinson adds (6/20/01, original post on the r3mix.net Forum):
The latest version I have is the Winamp plug-in v 0.13.0b. I don't have a recent compiled MADPLAY (in which the dither can be switched on and off), but I do have an old 0.11.4b accuracy tuned version. In the Winamp plug-in, the dither is always on.
ff123, you'll remember many moons ago I wrote a long post in the VQF forum explaining why dither could cause slightly more clipping than not dithering, if levels were just on the verge of clipping. Whilst this is true, this is not what is happening here.
The 0.11.4b MADPLAY and the Winamp plugin 0.13.0b both give identical clipping problems. When you get a row of clipped samples, MAD gets stuck in clipping for a few extra samples after the actual signal has come out of clipping (as decoded by FhG). This only ocurrs with dither enabled, With dither disabled, as people have found here, there's no problem.
So, this is a dither problem. Not dither per se, because dither is a good thing - but the dither in MAD currently isn't right. Not only is it non ideal (see http://mp3decoders.org 24-bit test), but guessing from what Rob (MAD author) has said, I think he used a short-cut and maybe made the dither dependent on previous samples - not only does this make the dither non-ideal, but it somehow gives this clipping bug.
So, I've (belatedly) accepted Rob's offer to test his new dither algorithm, and hope this will fix it. Then MAD with dither should give, on average, about the same amount of clipping as any other compliant decoder.