[Beowulf] since we are talking about file systems ...

Mark Hahn hahn at physics.mcmaster.ca
Wed Jan 18 08:12:17 PST 2006


> > No, because an one-byte-size file wastes a block of harddisk.
> 
> That's file system dependent. Solaris UFS can subdivide blocks into
> fragments. A file can use as little as an eighth of a block.

sure all derivatives of the BSD FFS do that (including NTFS and 
most native "Unix" filesystems.)  ReiserFS even goes further.

but IMO, if you find yourself worrying about this, you have committed
bad design.  no filesystem will give you truely low-overhead access 
to tiny files, and you should be using some higher-level container 
(ie, a DB of some kind).  consider that regardless of what OS/FS 
you use, you have inode and directory entry space overheads, as well 
as pretty nasty systemcall overheads.  with an appropriate container,
you eliminate quite a lot of useless work in the kernel (permissions,
etc), obtain optimal packing, and can trade library calls for syscalls.

> little files in a flat directory without too much pain. Just use
> the correct operating system and file system.

most linux uses ext3, which now defaults to hashed directories.
yes, blocks are by default 4k, but if your files average <2k, 
you're just being silly.




More information about the Beowulf mailing list