[Beowulf] NFS over RDMA small block DIRECT_IO corruption

[Joe Landman]

On 09/04/2012 06:57 AM, holway at th.physik.uni-frankfurt.de wrote:
> FYI
>
> https://bugzilla.linux-nfs.org/show_bug.cgi?id=228
>
> Careful with your block sizes when doing DIRECT_IO people :)

Direct IO should work only with 4k and multiple of 4k block sizes.  It 
does page-at-a-time IO, and doesn't do a read-modify-write on a per page 
basis.  It assumes you are doing this in your own app.

It looks lie you got it to "work" below 4k, but I would call that 
"undefined".  I am pretty sure it doesn't work in sector-sized increments.


c.f. http://www.titov.net/2006/01/02/using-o_largefile-or-o_direct-on-linux/

or you can google other sources.

Technically if you try to use it incorrectly, you should be getting a 
signal back.  dd may be trying to do something "smart" and hide this 
from you by "helping" you (either fake opening it as direct IO for less 
than page size, or really open it as direct IO with the 4k page size, 
and then do something wrong on the write side).  That is,

     dd if=file1 of=file2 bs=1k oflag=direct iflag=direct

may be silently changing the if/of to 4k.  I'd recommend an strace to be 
sure.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615