availability of Memory compression routine

[Christoph Best]

Kwan Wing Keung writes:
 > My question is now whether we have a generic memory compression routine
 > that allow the compression of a big memory chunk to a much smaller one
 > like that used in "zip" or "compress".  Of course we are talking about
 > compression for memory variable inside a standard Fortran program BUT
 > NOT the compression in a disk file.
 > 
 > In this case we can first compress the huge array and then use
 > mpi_broadcast to send the compressed data.  Upon receiving the compressed
 > data, each slave can decompress it to retrieve the original data.
 > In simple word, we are sacrifying local computation vs communication.

Compression routines usually do no good in numerical problems since
the data tends to look nearly random. Most of it are the
less-significant digits of the mantissa (decimal fraction), and these
usually do not exhibit any pattern or repetition that can be utilized
by a loss-less compression routine. 

However, you should check whether your application really requires
double-precision arithmetic - most numerical algorithms don't (there
are exceptions, especially when delicate subtractions are
involved). Going to single-precision saves half of the memory and
bandwidth. An easy test is to run the code both in single and in
double precision with the same input and make sure that the results
do not differ significantly.

-Christoph
-- 
Christoph Best                                        cbest at tigertiger.de
MIT Center for Theoretical Physics                http://tigertiger.de/cb