[Beowulf] Mixing 32-bit compute nodes with 64-bit head nodes

[Mark Hahn]

> > I was lead to believe that PAE on 32-bit systems was prone to causing
> > performance  problems.  If that isn't the case, you are certainly right.

well, on 32b chips it's better than nothing.  it also doesn't actually
expand the address space visible to a single process.

> PAE is an Intel developed (hacked) method to increase the address space 
> of the 32 bit processors.  It is slow due to the way it works.  It 

I'm not sure I agree - afaik-remember, PAE just has some extra physical
addr bits stuck in the PTE.  procs still have just 32b of addressibility
(and given the usual kernel config, 2-ish GB of real space).  but PAE
lets the kernel address up to something like 16 GB.

using PAE probably means you're using >4G physical ram, which _can_
cause inefficiencies with IO devices that are unable to generate 
>4G addresses.  again iirc, the kernel can wind up copying buffers 
around, but this is not necessarily a problem with all PAE systems.

> basically adds another bit of segmentation to the address space.  Anyone 
> remember EMS/XMS from DOS days?  Accessing pages through a 64kB window? 
>   This is not identical in design to that, but it is close.

very different, IMO.

> If you have the option, AMD64 is a better route than PAE.

well, that's certainly true ;)

but I think the main point here is that there are may still reasons to use
32b environments, even if pretty much all interesting hardware supports 64b.
a significant subset of the ~1300 users of my organization, for instance,
run quite small programs, sometimes only ~1M.  they have no need for 64b
addressibility, though I'd love to know whether the MUCH better registers
available in AMD64 would help them.  I'm hoping to get some of them to 
compile their code both ways and run on a bi-width system to test.

I don't know whether there's any way to use the extra registers
without paying the cost of all addrs going to 64b.  I suspect this 
could be made to work, but would be painful, ABI-wise.