Tag Archives: compression

PMPs – Why do People Ignore Compression?

One thing I notice is that many portable devices, companies sell higher capacity versions for exorbitant premiums, when flash memory really isn’t that expensive.  Seems to be less of an issue for players which do include an (mini/micro)SDHC expansion slot, as you can effectively increase capacity with a cheap add-on card.

But despite this, it seems that many people really do pay these excessive premiums for this increased storage.  I sometimes do wonder how people fill up so much space, eg getting a 32GB player over a 16GB one.  Surely these people have lots of videos and music, probably more than they need, and obviously, a higher capacity player allows them to carry more on the same device.

Whilst this is fine for the majority who aren’t so technically inclined, I do wonder about the people who are more technically inclined, and them overlooking the other side of the equation.  For example:

Amount of music that can be stored = Storage capacity ÷ per song size

Now we want to be able to store more music (again, even if it’s a lot more than we need), but the general approach of simply upping storage capacity is only one part of the equation – most people, even more technically inclined people, seem to ignore the fact that you can also store more stuff by reducing the file sizes of media!

Admittedly, compressing stuff can take effort.  In fact, I’ve had a number of motivations that most probably never had, including the old days of me trying to fit MP3s on floppies, squish as much as I could out of my 4GB harddrive, squeeze music on a 256MB MP3 player, and packing videos onto my 1GB PSP memory stick.  However, with a bit of reading, it’s mostly sticking your music/videos into a batch converter and then copying everything across.  It’s slightly less convenient when you add stuff (you probably need to pass these through a converter too), though, personally, I’m used to doing this, so I don’t mind.

But does compression really yield much benefit?  From what I’ve seen, I’d say so.  It seems most people just dump their 128/192/256/320kbps MP3s (usually more 320kbps as this is a popular size in P2P) on the device and that’s all they care about.  From the fact that most people cannot tell defects in 128kbps MP3s (let’s just say it’s LAME encoded), and my own listening tests, I’d say that most people cannot hear defects in 56-64kbps HE-AAC (encoded with NeroAAC).  Support for this format is limited though (difficulty of implementing SBR on embedded devices), though I believe Rockbox supports it, along with the latest iDevices (pre-late-2009 do not support HE-AAC).  Next in line would be 80-96kbps OGG Vorbis, if your player supports it.  In fact, I cannot personally hear defects in 128kbps Vorbis, so even audiophiles could use a big space saving by using higher bitrate Vorbis.  But support for Vorbis is surprisingly low, considering that this is a royalty free codec.

For an audio format with a fair bit of support, would be LC-AAC (aka “AAC”) which achieves similar quality to 128kbps MP3 at around 96-112kbps (using NeroAAC or iTunes).  Failing that, using LAME to encode MP3s with a variable bitrate can yield decent quality with average bitrates around 112kbps.

Now if we assume that the average song is a 320kbps MP3 and the listener really can’t hear defects in 128kbps MP3s, and the underlying player supports HE-AAC, we could get a massive 320-56 = 264kbps saving (82.5% smaller!) by being a bit smarter in storing our music.  This equates to being able to store over 5 times more music in the same amount of space.  But of course, this is an optimal situation, and may not always work.  Even if we’re more conservative, and say that the average MP3 is 192kbps, and the underlying player only supports LC-AAC, we can still get a 50% reduction in size by converting the 192kbps MP3 to 96kbps LC-AAC, which equates to a doubling in storage space.

Videos are perhaps more difficult to get right as the parameters involved in video encoding is significantly more complex than audio encoding (also note that videos often include audio).  But from what I’ve seen, significant space savings can be gained by encoding videos more intelligently, but it’s hard to provide rough figures as most people do convert videos for their portable devices, but use a wide variety of applications and settings.  For reference, I see a lot of >100MB PSP encoded anime episodes, however, I can personally get them to around 30-40MB using a x264 crf of 25 and ~8MB audio stream (allowing me to easily store a 12 episode anime series on a 1GB stick, with plenty of space to spare).

So for those who don’t compress their media, maybe give it a bit of a shot and see what space savings you can get.  You may be surprised at how much 16GB can really store.


Compressing PSP ISOs with 7z deflate

Many applications which compress data, do so using the free zlib library.  It’s relatively fast and provides a good speed/compression ratio for most applications, but above anything else, I imagine it’s huge popularity is due to it being licensed under a very liberal license.

zlib implements the deflate compression algorithm (same algorithm used in the popular ZIP file format, as well as GZip, PNG, PDF etc), and supports 9 compression levels (10, if you include “no compression” as a level), 9 being the highest compression level (at the expense of compression speed).

It may be somewhat known that 7-Zip implements its own deflate algorithms (not to be confused with the .7z format or its default LZMA algorithm) which tend to outperform zlib in terms of compression, at the expense of speed.  Some applications, such as AdvanceCOMP, have, thus, leveraged this to allow some files to have slightly higher compression ratios than that created with most applications.

Now, all CISO/CSO (compressed ISO) makers, for compressing PSP ISOs use zlib, so I imagined that replacing this with a 7z implementation of deflate would allow CSOs of smaller size to be made.

The CSO format compresses the ISO in blocks for 2048 bytes each (probably to match the ISO sector size) to allow the format to actually be read on the fly and playable on the PSP.  This does, unfortunately, mean that there’s a hit to compression, and therefore, CSOs will larger than an ISO compressed with GZip (assuming same deflate algorithm and settings).

But due to the above structure, it also means that compression can easily be made multi-threaded, simply by passing blocks around to different threads to compress.  So I decided to make such an application, not only to get the benefits of smaller CSOs, but also as an exercise in multi-threaded programming (haven’t really done this before).  The latter was somewhat a bit of a challenge, and I may write a separate blog post on it, but I have managed to compress some ISOs without it corrupting the thing, or just deadlocking during the process.

Unfortunately, the results seem to be somewhat disappointing.  I recall trying to compress the Europe release of Loco Roco and I only got about a 500KB smaller CSO compared to zlib level 9.  Just tried DJ Max Portable 2 ISO, using 7z’s deflate, 5 passes, fastbytes = 255, NC ratio = 98% (if compression ratio is above this, it will store the block uncompressed; the default ciso compressor doesn’t do this):

Uncompressed ISO: 1,778,319,360 bytes
Compressed with zlib 9: 1,637,928,332 bytes
Compressed with 7z: 1,636,999,514 bytes

So a disappointing gain of less than 1MB or 0.057%.  Granted, more can be done to achieve higher ratios (test with various fastbytes sizes, check with kzip, apply DeflOpt, maybe even reference duplicate blocks) but I doubt any of these would give even close to the gain I’ve gotten here, not to mention that they’d increase compression time by a lot.  The above process, with 6 threads, already took around 18 minutes on my Core 2 Quad Q6600 CPU @2.4GHz (was consuming 90%+ CPU throughout the process; I haven’t implemented I/O threading, which may allow it to get closer to 100%).  Compare this to the standard ciso compressor which (okay, I didn’t actually measure this) only took a few minutes to do, and this is only a single threaded app.

Oh well, at least I got some experience with multi-threaded programming, and I guess 1MB is still 1MB…  Maybe it’ll work better for popstation, as it seems to be using a larger block size of 37,632 bytes (=18*2048 or 16*2352?).