The Great MP3 Bitrate Experiment

blog
123

I think you stacked the deck alot with the choice of the clip you chose. I’m no audiophile, but I can hear the difference between MP3s, CDs, and records. For me it’s a question of range. I can hear that the top and bottom are clipped. Maybe it’s my age but I miss real bass - which you don’t hear much of anymore.

Part of this is I feel that artists are producing for the medium that dominates (iToones, MP3s, etc.) MP3s don’t sound much different from AM radio at times. The other part is folks can’t tell the difference because they’ve never heard anything else - or not for a very long time or very often anymore.

I considered a number of songs to illustrate my points, but the one that won out was ‘Rust Never Sleeps’ by Neil Young. It’s not that I’m such a big Young fan, but I can’t think of a better example of extremes in a single record - the open base cords should move your furniture across the room (without cranking the volume too much)and Youngs voice should nearly set your teeth on edge. If they don’t…well you’re not getting the full experience.

124

@Adolfojp and @Gsuberland

I agree that you both makes a very good point here. However, I’d like to point out that the experiment is aimed to prove the “lossless vs lossy vs uncompressed” myth. We’re looking for the best compressed format which, when played (= uncompressed on-the-fly ultimately to a speaker), gives the nearest-to-original quality. That is, INCLUDING the artifacts of old ! Tape hiss, low-bitrate drum samples, microphone clipping etc.

Although, I won’t be surprised if some, or maybe even a lot, younger audiences who live in the “HD era” could not appreciate the imperfections of older recordings. But personally, when speaking about compression quality, I’d like the playback come as close as possible to the original.

A nice footnote I’d like to add: I piggyback this test for my personal purpose, namely to know whether I could perceive the difference between the 5 using various earphones, headphones, and speakers.

The conclusion:

A lot of difference are only beginning to be seen (heard) using HiFi or reference-class headphones or speakers. Namely, the lower end of the bass spectrum; the vocal definition; the upper frequency natural rings (cymbals, screams/high-pitched vocals); and the times when all the frequencies are mashed-up together.

Earphones do not help, btw.

In short: in the world of audio-listening, you want quality = prepare to burn a bit of a hole in your pocket. All the way in the audio reproduction chain:

  • have a big harddrive, store your digital collection at least in FLAC.
  • have a good player/soundcard
  • have a decent amp
  • have at least HiFi or reference-class speakers or headphones.

Then re-encode them in 160kbps’ for your mobile phone/PMP speakers/earphones.

125

To my surprise, I discovered I’m no dog. Based on my post-analysis, my worst is actually #2 (where #1 is worst), and my best is actually #3.

However, when ripping from original source I’ll go the archival route and rip to flac. Then I’ll just re-encode for the device later, but most of the time I listen on my computer with fancy headphones anyway, so flac is used.

126

The problem with your test is that you are using highly processed and compressed rock music that is filled with intentional distortion as artistic expression.

Over the years I have helped musicians design instruments, speakers, amplifiers and processing electronics to suit their needs. Invariably I am asked “what is the best kind” of speaker, amplifier, pickup, etc. My answer is always “whatever pleases you.” The artist’s choice, almost without exception, is something that significantly colors the sound with linear and non-linear distortion; after all, they are seeking something unique and expressive. While this is an excellent choice for performance, it is a terrible choice for reproduction! The responsibility of a performance system is to express the artist’s vision, while the responsibility of a reproduction system is to render the original performance as faithfully as possible.

The recordings that you have posted for your test are reproductions and should be judged according to their faithfulness to the original performance, but how are we to do that? Do the recordings sound different? On my AKG headphones the answer is absolutely yes. Can I tell which is a more faithful reproduction of the original performance? The answer is absolutely no. In judging recordings of an electronic performance, there is simply no way to determine which reproduction is the most faithful without comparing it to the live performance.

This is why high quality acoustic recordings, be they rock, jazz, classical or folk, are more useful for judging reproduction quality. The reason is that our ears are already calibrated to what a guitar, xylophone, marimba, tabla or cello sound like in a live performance, which gives us a basis for comparison. Furthermore, a recording made in a natural acoustic environment allows us to judge whether the individual instruments and vocalists are precisely defined and well positioned on the stereo stage, or muddled together and muddy sounding (an artifact of non-linear distortions; the sum and difference frequencies referred to by another writer). This is simply not possible with studio recordings that have been electronically processed and mixed.

If your music library consists solely of electric rock and pop music then I agree that the differences between your samples are probably inconsequential. An electric guitar can have literally millions of different sonic personalities. How are we to know if it is reproduced correctly or not? But if you listen to well recorded acoustic music it is immediately obvious if a flute or violin or sitar “just doesn’t sound quite right.”

127

I have perfect pitch and studied music from an early age. Using sennheiser hd headphones the difference between lossy (mp3 256 bitrate) and lossless (flac) is huge.

While doing music people couldn’t clearly differentiate between the instruments on a track while using crappy headphones, after switching to good ones the instruments could be separated much more clearly by the ear. It is the same with lossy vs lossless.

The only way I can describe what is happening is the following: Have you ever been in the same room with a live drummer? Only you and him, to hear how he hits the bare metal? Now record him and listen to the difference. The lossless format is much much closer to the live sound. You hear it like the metal is hit before you. You do not need a trained ear to hear this difference.

Try to focus on only one particular instrument, see how easy it is in loseless vs lossy, hear it during the entire piece, do not let your mind fade from the guitar to the bass guitar for example. In loseless the instruments are ‘clearer’, not ‘stepping on one another’… It’s easier to do this, just like they are playing in the same room with you.

Making the difference between 256kbps and 320kbps mp3 is much harder of course, even for a trained ear, but between 256 mp3 and flac, it’s really obvious.

The experiment you are making is inconclusive, at least you should have people tested on the same headphone set. Even then it is difficult to judge any results because of different hearing finesse of the subjects.

You said “nobody could really hear the difference between a raw CD track and an MP3 using a decent encoder at a variable bit rate averaging around 160kbps”. This is wrong.
If this was true, then we would be using 160kbps for high quality audio. We do not use lossless formats out of masochism, just for the computers to store the sounds without us being able to hear them.

So, in the end, if you can’t hear a difference it doesn’t mean others can’t. Go to a recording studio near you and ask the guys there what they think.

128

Hello Mr. Horror. I have one problem with your experiment. You are only testing for accuracy and not for precision. It would be better if you had 3 songs with each song at 3 bitrates. This way you can measure whether people are judging wrong or their criteria is wrong. Personally I like songs encoded at 96 and think they are lossless. Don’t like lossless CD quality and would rate it as 96.

129

In striving to get rid of physical stuff, you tend to gather lots of data instead. Data that just gathers dust…

I’d go for Spotify instead when it comes to music. Stream to the phone or computer when you need it.

130

If you’re so concerned with efficiency, why not go with AAC rather than with MP3?

131

The German computer magazine “c’t” did an experiment back in 2000. They invited sound and music specialists (among them a famous German music producer and a blind person) into a studio and did a double-blind test to see if they could differentiate between MP3 and CD audio.

Bottom-line: “Our music-trained test listeners could tell the the poorer MP3 quality (128 kbit s) is quite accurately from the other two samples; between MP3, 256 kbit/s and the original CD however, they could not - on average over all the music pieces - tell the difference.”

Because the first test created such an outrage among the readers of the magazine, c’t released the samples and let the readers repeat the tests themselves. The result:

“The fact that some of the 128-kBit/s-recording […] were consistently rated better than the originals from the CD, however, stunned even the involved editor …”

The quotes are taken from the follow-up article (c’t 6/2000) on the original test article which appeared in c’t 3/2000.

http://www.heise.de/ct/artikel/Kreuzverhoertest-287592.html

The original article is only available behind a paywall.

132

@Julian: AAC is not efficient use of your time if you’re deploying to portable units (radios, car stereos, DVD players etc) outside of the Apple ecosystem. WMA is the most commonly found compressed format after MP3 on most devices. Even expensive devices I’ve bought in the last year or two either don’t support AAC at all, or require AAC-LC.

133

It might be difficult to hear on your laptop’s built-in audio card.

134

First off… GREAT EXPERIMENT.

Two key points I wanted to throw into the pile here…

  1. Quality (bit rate) of the audio track is moot if you have shitty (apple headphone) output quality… you will indeed NEVER notice a difference. Audiophile grade recordings are only worthwhile and of value if the output (speakers) are reference grade. Believe me, if you love music as much as many of us do, you will indeed notice a difference.

  2. Recording 192k FLAC from a CD is a true waste of space. You are not GAINING anything from the extended bit rate… since it has already been taken away when it was put down to “CD quality” bit rates. It’s like drinking shite wine from crystal goblets… it’s still going to taste like shite. Audiophiles will go to sites that sell original master recordings, RECORDED at 192k… and trust me, there is a massive difference. The voice isn’t muddy, the bass is rich and punchy and the high end is clear and not brassy or sharp.

For those who are happy now, rock on…

135

Unfortunately, I’m just a normal guy… So I couldn’t really hear the difference.

But I thank you for the song I didn’t know and which was pretty good actually.

136

Looks like the survey is closed, so I’ll add my thoughts here.

Limburger and Feta both sound awful. I’m guessing Feta is the 128bkps one, and Limburger isn’t too far off.

Brie is a bit better, although the compression artifacts are still noticeable on the sibilants.

Cheddar and Gouda are the tricky ones. I can’t tell if any of them are compressed on their own, and comparing them against each other gets very subjective. My bet is on Gouda being the uncompressed one, as Cheddar sounds slightly duller in the high frequency domain.

I listened on AKG K271 mkIIs plugged straight into a MacBook Pro, no amp or external DAC. These are studio headphones with well defined upper highs, which is where MP3 compression is most noticeable.

I think anything above 256kbps is fine. I’ve settled on the V0 preset for my own library, harddrives are cheap.

137

It is quite hard. I think they should be ordered like this:

5 Feta = Worst
4 Gouda
3 Limburger
2 Cheddar
1 Brie = Best

Is it possible for you to list with the results the different kbps for each sample and the encoder use. I would also like to test it with other types of music, as other comments suggested.

Cheerio
Cartero33

Cartero33

138

Three of them were so painfully bad that I had to stop after 10-15 seconds. One was acceptable, but not as good as the copy I have on my PC, while the other was clearly the genuine article. Then again, I do have sensitive ears.

The best policy, if you have the space and local law permits, to keep a reference copy of as high a quality as possible on your archive machine, then downgrade it according to the equipment available and likely listeners on the target device. For example, my desktop machine has the best quality I can get. This is often 192 kbps due to upstream decisions on the quality/download speed compromise or the music being just plain old, but I take 320 or lossless where possible. Even then I often prefer CDs because the CD-speaker connection is slightly cleaner than the hard drive-speaker route on my particular machine.

My PDA, on the other hand, uses an indifferent sound card to produce audio and has a bad speaker. The headphones I have are not much better because the good ones require a converter which reduces the sound to the point where the improved headphone quality is irrelevant. Music on that is either 8 kbps or 16 kbps, depending on which produces artefacts that mask the hardware imperfections better. For some really awkward tracks, I’ve even resorted to converting to MIDI, though that has major downsides (including losing the words sung).

139

I must echo the complaints about the choice of material. Not only are their a lot of samples in there, which are low quality from the 80s, there is also a lot of digital reverb which has a smeary "digital sound. In addition, if this was “mastered” to CD in the 80s or 90s, a lot of those masters were done very poorly, and in general there is all sorts of processing applied during that stage. A lot of those early CDs sound downright horrible. Mastering techniques have improved a lot, but more recent masters of classic rock have to battle the decay of the original analog tapes.

So there is a vast range in “sound quality” of commercially released music. A good sounding recording at 128kbit mp3 will sound way better than a bad sounding recording at full bandwidth.

That said, I had a hard time hearing differences. One problem is that hearing subtle differences in recordings is hard. It takes critical listening skills. Also, remembering differences in sound if you have to go through a processing of switching sources that takes a few seconds is pretty much impossible. Our brains are constantly processing how we hear things, every time you listen to the same recording closely it will actually sound different!

I doubt anyone can rank the samples correctly in this case. On the other hand, in a good listening environment, I am sure I could rank the original against the 160kbit if I had a quick A/B comparison. At least I could tell a difference. Deciding which one sounds “better” is entirely subjective.

On Jeff’s original point, well if he thinks encoding at 160kbit is good enough for his own listening, then it is!

140

Maybe of interest for some.

Comparing FLAC vs WAV: http://www.computeraudiophile.com/blogs/mitchco/flac-vs-wav-part-2-final-results-155/

Comparing 16/44 to 24/192: http://www.computeraudiophile.com/blogs/mitchco/16-44-vs-24-192-experiment-163/

And if you are a full-on audio/software geek and like playing with sound and DSP: http://www.computeraudiophile.com/blogs/mitchco/importance-timbre-sound-reproduction-systems-222/

142

Results please?

143

theres a few rules when it comes to audio,

  1. Audiophiles are insane and is basically a bunch of Placebo marketing bullshit, who like listening to their gear more than the actual music HeadFi forums have been known no ban any science, and one particular video I saw on youtube of a Headfier meet-up had some guy with a pair of $150 headphones hooked up to some Rackmount DSP-EQ and another rackmount Headphone amp, these electronics must be worth about $50k when yet he was using some seriously cheap headphones on it, and I’m sure this was his “portable” setup.

  2. Placebo placebo placebo ( http://www.youtube.com/watch?v=G-lN8vWm3m0 )

  3. Audio artifacts can be more audible on bad quality headphones as codecs are designed for a flat signal response and only the good quality reference ones actually have a good flat response (sites like Soundcloud and Bandcamp (128mp3) are actually far more enjoyable on my AGK Q701’s compared to a cheap pair of headphones)

  4. trust your ears, just make sure you remove the placebo effect and bias by doing blind listening tests, I was recently talking to some guy who thought 128kbps AAC encoded with lib_voaac sounded better than 128 mp3 encoded with LAME because his test was so far from reality that it was meaningless, if only he had used his ears he would have realised his testing method was stupid.

  5. 128kbps is not equal to 128kbps, codec, encoder, and source all play a factor in quality thats why VBR is superior, and why all of youtube sounds like crap even though it using a superior codec to MP3, and another reason why you should buy lossless files so you can trust the MP3 is truly the best sounding one possible as you encoded it yourself.

  6. Lossless is the ‘better safe than sorry’ option. mp3/vorbis/aac have some inherent mathematically flaws or just have buggy encoders that cause certain sounds “Killer samples” to artifact at any bitrate these usually don’t appear in real world applications but can, so either blind test everything or be lazy and just use lossless ( http://www.hydrogenaudio.org/forums/index.php?showforum=35 ) and this allows you to convert to Vorbis, MP3 or AAC depending on what the device can play later.