[Beowulf] Accelerator for data compressing

[Greg Lindahl]

On Thu, Oct 02, 2008 at 05:40:31PM -0400, Joe Landman wrote:

> I have heard of some "xml accelerators" in the  
> past (back when XML was considered a good buzzword) that did on-the-fly  
> compression.

Well, given how wordy the tags are, simply compressing those is
inexpensive and is a big win, if they're a large % of the data.

To get back to the question that was asked, (1) no hardare compresser
compresses smaller than a good software compresser (hardware
compressors tend to be faster but can't compress as well), and (2)
sounds like your data can be compressed in an embarrassingly parallel
fashion, so on a quad-core box you might find that you can keep up
with your link with software compression in parallel.

-- greg