[Beowulf] scaling

[Jim Lux]

At 05:03 PM 1/16/2006, Joe Landman wrote:
>Hi Craig:
>
>Craig Tierney wrote:
>>Mike Davis wrote:
>>
>>>geeks out there, what happens when a folder has 50,000 files in it. Can 
>>>you say SLOOOOOOOOOWWWW?
>>First, pick the right filesystem.
>>Second, rewrite your code so you don't have 50k+ files in one directory.
>>There must be some straightforward way to solve the problem if
>>you have too many files in one directory.
>
>Lots of the informatics codes were not written with such input (or 
>database) scaling in mind.  For them, 10-100 files in a directory isn't 
>much of a problem.  Its when you start to scale up that the bugs and 
>surprises start.

I think that unforeseen scaling problems are probably the single biggest 
challenge with HPC.  If it's not the plethora of files, or crippling 
interprocessor comm needs, it's something like timing races and implied 
barriers (in a brute force master doling out identical work packets to all 
the slaces.. they all finish at the same time).


>_______________________________________________
>Beowulf mailing list, Beowulf at beowulf.org
>To change your subscription (digest mode or unsubscribe) visit 
>http://www.beowulf.org/mailman/listinfo/beowulf

James Lux, P.E.
Spacecraft Radio Frequency Subsystems Group
Flight Communications Systems Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875