[Beowulf] multi-threading vs. MPI

[Mark Hahn]

> with a grain of salt. However, it seems to me that a sufficiently
> intelligent operating system would be able to attach the thread local
> store to the local memory pool.

yes, everyone does numa-local allocation, to some extent or other.
but it's far from obvious which node should actually be the home of 
particular data - most of the time, a "first touch" policy is used
(whichever node touches the page first gets to host it.)  attempting
to dynamically adjust this affinity later sounds very tricky to me,
since afaikt it would require messing with TLB entries (slow).

> the Windows and UNIX world. Using the complexity of MPI isn't very
> popular when a simpler method works okay.

if you just want a quick hack and are satisfied with modest speedup,
threading is great!  I wouldn't agree that MPI is more complex - 
compared to an application which has had the same level of tuning
and refactoring to reach a high level of scalability.  in other words,
once you undertake to scale a code to hundreds of CPUs, using threads 
won't give you greater simplicity.

> However, it seems like threads have taken a little beating lately in
> favor of discrete address spaces in security conscious system software
> (most famously, DJBs stuff) which communicate with pipes and shm. And

threads, of course, are antithetical to security, since the whole point
is freedom to read/write anything.  I wouldn't say threads are antithetical
to The Unix Way, necessarily, though TUW obviously emphasizes fast fork/exec,
pipes, etc.  sendmail is less unixy than postfix, for instance.

and bear in mind that there are clever optimizations (splice) to optimize
the performance of pipes.

> lessons from QNX and BeOS tell us we might see some resurgence in the
> use of message queues and other means of low level message passing in
> the standard library.

I don't think so.  doing RPC-like message-passing over untyped channels
would be the modern trend (SOA).