Many applications which compress data, do so using the free zlib library.  It’s relatively fast and provides a good speed/compression ratio for most applications, but above anything else, I imagine it’s huge popularity is due to it being licensed under a very liberal license.

zlib implements the deflate compression algorithm (same algorithm used in the popular ZIP file format, as well as GZip, PNG, PDF etc), and supports 9 compression levels (10, if you include “no compression” as a level), 9 being the highest compression level (at the expense of compression speed).

It may be somewhat known that 7-Zip implements its own deflate algorithms (not to be confused with the .7z format or its default LZMA algorithm) which tend to outperform zlib in terms of compression, at the expense of speed.  Some applications, such as AdvanceCOMP, have, thus, leveraged this to allow some files to have slightly higher compression ratios than that created with most applications.

Now, all CISO/CSO (compressed ISO) makers, for compressing PSP ISOs use zlib, so I imagined that replacing this with a 7z implementation of deflate would allow CSOs of smaller size to be made.

The CSO format compresses the ISO in blocks for 2048 bytes each (probably to match the ISO sector size) to allow the format to actually be read on the fly and playable on the PSP.  This does, unfortunately, mean that there’s a hit to compression, and therefore, CSOs will larger than an ISO compressed with GZip (assuming same deflate algorithm and settings).

But due to the above structure, it also means that compression can easily be made multi-threaded, simply by passing blocks around to different threads to compress.  So I decided to make such an application, not only to get the benefits of smaller CSOs, but also as an exercise in multi-threaded programming (haven’t really done this before).  The latter was somewhat a bit of a challenge, and I may write a separate blog post on it, but I have managed to compress some ISOs without it corrupting the thing, or just deadlocking during the process.

Unfortunately, the results seem to be somewhat disappointing.  I recall trying to compress the Europe release of Loco Roco and I only got about a 500KB smaller CSO compared to zlib level 9.  Just tried DJ Max Portable 2 ISO, using 7z’s deflate, 5 passes, fastbytes = 255, NC ratio = 98% (if compression ratio is above this, it will store the block uncompressed; the default ciso compressor doesn’t do this):

Uncompressed ISO: 1,778,319,360 bytes
Compressed with zlib 9: 1,637,928,332 bytes
Compressed with 7z: 1,636,999,514 bytes

So a disappointing gain of less than 1MB or 0.057%.  Granted, more can be done to achieve higher ratios (test with various fastbytes sizes, check with kzip, apply DeflOpt, maybe even reference duplicate blocks) but I doubt any of these would give even close to the gain I’ve gotten here, not to mention that they’d increase compression time by a lot.  The above process, with 6 threads, already took around 18 minutes on my Core 2 Quad Q6600 CPU @2.4GHz (was consuming 90%+ CPU throughout the process; I haven’t implemented I/O threading, which may allow it to get closer to 100%).  Compare this to the standard ciso compressor which (okay, I didn’t actually measure this) only took a few minutes to do, and this is only a single threaded app.

Oh well, at least I got some experience with multi-threaded programming, and I guess 1MB is still 1MB…  Maybe it’ll work better for popstation, as it seems to be using a larger block size of 37,632 bytes (=18*2048 or 16*2352?).