[Beowulf] Accelerator for data compressing
diep at xs4all.nl
Fri Oct 3 03:53:49 PDT 2008
Ah you googled 2 seconds and found some oldie homepage.
Try this homepage www.maximumcompression.com
Far better testing over there. Note that it's the same testset there
that gets compressed a lot.
In real life, database type data is having all kind of patterns which
PPM type compressors find.
My experience is that at terabyte level the better compressors at
are a bit too slow (PAQ) and not so good like simple things like 7-zip.
Look especially at compressed sizes and decompression times.
The only thing you want to limit over your network is the amount of
bandwidth over your network.
A real good compression is very helpful then. How long compression
time takes is nearly not relevant,
as long as it doesn't take infinite amounts of time (i remember a new
zealand compressor which took 24
hours to compress a 100MB data). Note that we are already at a phase
that compression time hardly
matters, you can buy a GPU for that to offload your servers for that.
Query time (so decompression time) is important though.
If we look to graphics there:
026 7-Zip 4.60b -m0=ppmd:o=4 764420 81.58 1.4738
94 BZIP2 1.0.5 -9 890163 78.55
158 PKZIP 2.50 -exx 1250536 69.86
159 HIT 2.10 -x 1250601 69.86
160 GZIP 1.3.5 -9 1254351 69.77 2.4184
161 ZIP 2.2 -9 1254444 69.77
162 WINZIP 8.0 (Max Compression) 1254444 69.77 2.4185
Note a real supercompressor is getting it even tinier:
003 WinRK 3.0.3 PWCM 912MB 568919 86.29 1.0969
Again all these tests are at microlevel. Just a few megabtes of data
that gets compressed.
You don't build a big infrastructure just for a few megabytes, it's
not so relevant.
The traffic over your network dominates there, plenty of idle server
cores there is, in fact there is
so many companies now that buy dual cores, as they do not know how to
keep the cores in quad cores
This is all microlevel. Things really change when you have terabytes
to compress and HUGE files.
Bzip2 is ugly slow for files in gigabyte size, 7-zip is totally
beating it there.
On Oct 3, 2008, at 11:27 AM, Carsten Aulbert wrote:
> Hi all
> Bill Broadley wrote:
>> Another example:
>> 7zip compress: 19:41
>> Bzip2 compress: 8:56
>> Gzip compress: 3:00
>> Again 7zip is a factor of 6 and change slower than gzip.
> Have you looked into threaded/parallel bzip2?
> freshmeat has a few of those, e.g.
> (with the usual disclaimer that I haven't tested them myself).
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
More information about the Beowulf