[Beowulf] scaling

Jim Lux James.P.Lux at jpl.nasa.gov
Mon Jan 16 18:13:35 PST 2006

At 05:03 PM 1/16/2006, Joe Landman wrote:
>Hi Craig:
>Craig Tierney wrote:
>>Mike Davis wrote:
>>>geeks out there, what happens when a folder has 50,000 files in it. Can 
>>First, pick the right filesystem.
>>Second, rewrite your code so you don't have 50k+ files in one directory.
>>There must be some straightforward way to solve the problem if
>>you have too many files in one directory.
>Lots of the informatics codes were not written with such input (or 
>database) scaling in mind.  For them, 10-100 files in a directory isn't 
>much of a problem.  Its when you start to scale up that the bugs and 
>surprises start.

I think that unforeseen scaling problems are probably the single biggest 
challenge with HPC.  If it's not the plethora of files, or crippling 
interprocessor comm needs, it's something like timing races and implied 
barriers (in a brute force master doling out identical work packets to all 
the slaces.. they all finish at the same time).

>Beowulf mailing list, Beowulf at beowulf.org
>To change your subscription (digest mode or unsubscribe) visit 

James Lux, P.E.
Spacecraft Radio Frequency Subsystems Group
Flight Communications Systems Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875

More information about the Beowulf mailing list