[Beowulf] Accelerator for data compressing

[Joe Landman]

Xu, Jerry wrote:
> Hello, 
> 
>  Currently I generate nearly one TB data every few days and I need to pass it
> along enterprise network to the storage center attached to my HPC system, I am
> thinking about compressing it (most tiff format image data) as much as I can, as
> fast as I can before I send it crossing network ... So, I am wondering whether
> anyone is familiar with any hardware based accelerator, which can dramatically
> improve the compressing procedure.. suggestion for any file system architecture
> will be appreciated too..  I have couple of contacts from some vendors but not
> sure whether it works as I expected, so if anyone has experience about it and
> want to share, it will be really appreciated !

Hi Jerry:

   Sounds like a bunch of sequencers or arrays going, and dumping data 
to a file system.  I am not sure you can get deterministic compression 
run times from a compressor, or even deterministic compression ratios on 
random (binary) data.  I have heard of some "xml accelerators" in the 
past (back when XML was considered a good buzzword) that did on-the-fly 
compression.

   I guess it boils down to if

	T(compression) + T(comppressed transfer) <<
				T(uncompressed_transfer)

   And the cost benefit analysis would focus upon the cost of 
T(compression) as compared to faster networks.   That is, if you spent 
$1000/node more to get a faster fabric, which dropped your transfer time 
to 20%, is this better/more cost effective than spending $10,000 or so 
on an accelerate that may get 70% file compression, and double the 
overall time?

   Obviously the above numbers are made up, but you get the idea.

   Will look around.  If you want to talk to a group doing FPGA stuff in 
other markets, let me know and I can hook you up.  Just be aware that 
this might not be cost/time effective.

Joe

> 
> 
> 
> Thanks,
> 
> Jerry
> 
> Jerry  Xu  PhD
> HPC Scientific Computing Specialist 
> Enterprise Research Infrastructure Systems (ERIS) 
> Partners Healthcare, Harvard Medical School
> http://www.partners.org
> 
> The information transmitted in this electronic communication is intended only
> for the person or entity to whom it is addressed and may contain confidential
> and/or privileged material. Any review, retransmission, dissemination or other
> use of or taking of any action in reliance upon this information by persons or
> entities other than the intended recipient is prohibited. If you received this
> information in error, please contact the Compliance HelpLine at 800-856-1983 and
> properly dispose of this information.
> 
> 
> 
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615