availability of Memory compression routine

[szii at sziisoft.com]

----- Original Message ----- 
From: "Christoph Best" <cbest at tigertiger.de>
To: "Kwan Wing Keung" <hcxckwk at hkucc.hku.hk>
Cc: <beowulf at beowulf.org>
Sent: Wednesday, July 17, 2002 8:28 PM
Subject: Re: availability of Memory compression routine


> Kwan Wing Keung writes:
>  > My question is now whether we have a generic memory compression routine
>  > that allow the compression of a big memory chunk to a much smaller one
>  > like that used in "zip" or "compress".  Of course we are talking about
>  > compression for memory variable inside a standard Fortran program BUT
>  > NOT the compression in a disk file.
>  > 
>  > In this case we can first compress the huge array and then use
>  > mpi_broadcast to send the compressed data.  Upon receiving the compressed
>  > data, each slave can decompress it to retrieve the original data.
>  > In simple word, we are sacrifying local computation vs communication.
> 
> Compression routines usually do no good in numerical problems since
> the data tends to look nearly random. Most of it are the
> less-significant digits of the mantissa (decimal fraction), and these
> usually do not exhibit any pattern or repetition that can be utilized
> by a loss-less compression routine. 
> 
> However, you should check whether your application really requires
> double-precision arithmetic - most numerical algorithms don't (there
> are exceptions, especially when delicate subtractions are
> involved). Going to single-precision saves half of the memory and
> bandwidth. An easy test is to run the code both in single and in
> double precision with the same input and make sure that the results
> do not differ significantly.
> 
> -Christoph
> -- 
> Christoph Best                                        cbest at tigertiger.de
> MIT Center for Theoretical Physics                http://tigertiger.de/cb
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf