[Beowulf] since we are talking about file systems ...

[Robert G. Brown]

On Tue, 17 Jan 2006, Douglas Eadline wrote:

>> A well designed and implemented file system shouldn't bog you down as
>> you scale out in size, even if you shouldn't.  Its sort of like your
>> car.  If you go beyond 70 MPH somewhere in the US that supports such
>> speeds, your transmission shouldn't just drop out because you hit 71 MPH.
>>
>
> Joe,
>
> You obviously have not seen some of the cars I used to drive.

Just for the heck of it, I wrote the C benchmark.  On my 800 MHz laptop,
it creates 50000 files with 1024 bytes each (in fact, integers from 0 to
255 in binary format).  I used open/write/close instead of line based
I/O for maximal efficiency.  I get:

rgb at lilith|B:1180>disktest
000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff
50000 files created in 3.804779 seconds
rgb at lilith|B:1181>time ls file.* | wc
bash: /bin/ls: Argument list too long
       0       0       0

real    0m0.999s
user    0m0.928s
sys     0m0.068s

rgb at lilith|B:1182>llth
total 201516
drwxr-xr-x    2 rgb rgb 1421312 Jan 17 18:07 .
-rwx------    1 rgb rgb    1024 Jan 17 18:07 file.41592
-rwx------    1 rgb rgb    1024 Jan 17 18:07 file.41593
-rwx------    1 rgb rgb    1024 Jan 17 18:07 file.41594
-rwx------    1 rgb rgb    1024 Jan 17 18:07 file.41595
-rwx------    1 rgb rgb    1024 Jan 17 18:07 file.41596
...

This seems to be pretty fast on an ext3 filesystem.  Creating all the
files and writing them is less than 4 seconds, listing them takes about
a second.

Very shortly I'll reverse this and read in all these files in C.  I
expect this to take a lot longer because of the requisite stats and
overhead.

     rgb

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu