IO and compression on the fly

[Gerry Creager N5JXS]

We have  an application we've just gotten funded where the computational
aspects will almost certainly be handled by a smallish (32-64
processors) cluster.  The big gotcha we have is data acquisition: We're
going to be capturing megabits/sec of scanning data from a laser-based
physical tomography device, and need to find some way to rationally
scan, compress and store all the data without losing it.  

Knowing the propensity of this group to find elegant solutions to
problems, I decided it was time to place this problem here.  

Our current thought is to do data acquisition on a box with a PCI-2
interface to take advantage of the faster PCI bus speed.  We're
currently thinking in terms of on-the-fly compression to reduce the
dataload on disk I/O.  to quote the PI, "Then you can process it on
whatever old supercomputer you have lying about."  

I don't know if anyone has faced (or conquered) the dataoverload problem
already, but we'd love to hear thoughts on alternatives.

Thanks,
gerry
--
Gerry Creager		gerry at cs.tamu.edu, gerry at page4.cs.tamu.edu
Network Engineering			|Research focusing on
Academy for Advanced Telecommunications |Satellite Geodesy and 
and Learning Technologies		|Geodetic Control
Texas A&M University	979.458.4020  (Phone) -- 979.847.8578  (Fax)