[Beowulf] SSD caching for parallel filesystems
Ellis H. Wilson III
ellis at cse.psu.edu
Sun Feb 10 05:09:48 PST 2013
On 02/10/13 04:41, Vincent Diepeveen wrote:
> SSD's are not about bandwidth, they're about latency.
This is a bit aggressive of a vantage point -- let's tone it back:
"SSD's aren't always the cheapest way to achieve bandwidth, but they are
critical for latency-sensitive applications that are too large for main
In any event, your original statement used to be wholly correct. It has
changed to a certain degree to "SSDs are about IOPs," which isn't quite
the same thing. However, more pointedly, with modern HDDs barely
approaching 200MB/s and SSD solutions approaching 2-4GB/s, this is an
increasingly limited viewpoint. We have to start considering their use
> With a raid array of cheapo disks we can also get 3GB/s bandwidth,
> more than most 2 socket nodes effectively can handle.
3GB/s divided by 200MB/s gives me something like 15 drives, unless my
math is wrong, which will be something like $2-$3K, and that's really
only possible in RAID0, so you're only going to get the capacity of one
drive. If all I'm looking for is bandwidth I'd rather spend that 3k on
an expensive SSD (or RAID a bunch of cheaper SSDs) and get it for far
less power, wire complexity, space consumption, and risk of failure.
Moreover, it'll have better latency. This gap will continue to widen,
so while we can talk about 15 disks reasonably right now, in a year
we'll be talking more like 25-30 and then it just becomes absurd. Just
buy the SSD(s) at that point.
> Only theoretically a higher bandwidth will be possible (benchmarks huh).
> However getting 20 bytes from a SSD is in the few dozens of
> microseconds, versus several milliseconds for the cheapskate disks.
> That factor of 50-100 difference roughly in latency difference is the
> reason SSD's exist.
> Any bandwidth test of a SSD is total nonsense.
(I wish you'd put [In my personal opinion] in front of all of your
sentences. It would make them less nails on a chalkboard.)
So what happened to "perfectly parallel"? Seems to me like a perfectly
parallel device would be well tuned to deliver good bandwidth.
More information about the Beowulf