[Beowulf] clusters in gaming

[Eugen Leitl]

On Fri, Feb 02, 2007 at 01:42:53AM +0800, Dean Michael Berris wrote:

> I don't see why a usual C/C++ MPI approach wouldn't work, though the

In theory, there is no reason why these (or even Fortran) wouldn't be adequate either,
but in practice it would be very difficult to accomodate user-contributed
scripted objects into a rigid array/pointer framework. Adding new
methods in C to a brand new object instantiated at runtime is certainly
possible, but it sounds intensely painful. 

>From the point of view of running a massively parallel realtime
(fake) physics simulation with many 10 k simultaneous viewers/
points of input it looks as if requires a massive numerical
performance, which suggests C (less C++). Common Lisp now has
very good compilers, but I wonder how well that translates
into numerics, and similiar to C++ the unwary programmer can
produce very slow code (CONSing, GC, etc).

> scaling issues of adding a new node to the cluster is certainly a

There are two types of regions, isolated islands, and addition to the
main "continent". Both look quite suitable for geometric problem tesselation
(one node, one region) and incremental node addition as the terrain
grows.

> problem that may be a hindrance from the implementation -- but one that
> can be remedied by having local clusters "gridded" together using some
> protocol.

As far as I know SL is run on one local cluster, which is why I thought
of how a Beowulf approach would help. They're complicating it by using
virtual machines, and packing several VMs on one physical server.
After (frequent) upgrades servers are restarted in a rolling fashion,
and I presume snapshot/resume migration is a useful function here.
But then, there are cluster-wide process migration available,
which are a lot more fine-grained.
 
> As for throwing hardware at it, I don't think that's a problem -- that's
> actually a good solution. That being said, if the implementation was

I thought the cluster had some 1000 nodes, but 
http://gwynethllewelyn.net/article119visual1layout1.html
claims there are just 5000 virtual servers. Maybe they
just run 5 VServers/node, and there are really 1 kNodes,
which is a reasonably large cluster for just 16 kUsers
at peak (not for your garden-variety Beowulf, but
for a game server).

> already good to start with then adding more hardware would have
> (supposedly) better effect on the overall performance/experience.

It would be really interesting to learn how current SL scales.
 
> Short answer is yes: it can even be done in C++. However what I would
> rather use in these situations would be a dynamic language like as you
> mention Lisp or things like Python (embedded in C++ or the other way
> around, see Boost.Python).

Thanks for the link.
 
> I think it's an architecture problem more than anything as far as the SL
> server side is concerned. But then when you're faced with a problem like
> full-3D physics engine in the server side, that's not something "as easy
> as Hello, World" to implement (or fix for the matter).

OpenCroquet uses a deterministic computation model, which replicates
worlds to the end unser nodes a la P2P, and synchronizes differing inputs 
so that each simulation instance doesn't diverge. It can also do a master/slave
type of state replication, if I understand it correctly, so in
theory it could use physics accelerators, and clone state to slower
nodes. SL in comparison does about anything but primitive rendering
cluster-side. Given current assymetric broadband, this seems
to be a superior approach to do everything P2P. (And I would imagine
OpenCroquet hasn't even begun to deal with the nasty problem of NAT
penetration).
 
> Though it certainly is not "impossible", "hard" would be an
> understatement especially now that it's in full-deployment with
> thousands of people getting on it at any given time.

It's really interesting. I wish there was more information flow out
of Linden Labs, on how they're doing it.

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820            http://www.ativel.com
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE