[Beowulf] Erlang as a language for Beowulf applications (and
gbulmer at gmail.com
Wed Sep 5 17:59:00 PDT 2007
On 5 Sep 2007, at 20:00, Eric Shook wrote:
> Subject: Re: [Beowulf] Erlang as a language for Beowulf applications
> To: "H.Vidal, Jr." <hvidal at tesseract-tech.com>
> I am engaged in similar explorations and I too have found the Erlang
> language as a possible solution to scaling issues. I have been
> experimenting with the language for almost a month so my evidence
> is not by any means conclusive. With that said in my opinion the
> language has a fatal flaw and that is data handling. According to the
> documentation and my brief experience there are certain instances
> the data being passed to and from processes / functions is copied and
> not referenced. You can imagine the problems if you are manipulating a
> multigig data set.
You are ahead of me, but, there are claims in the HiPE (High
Performance Erlang) papers (e.g. http://www.fantasi.se/jesper/
publications/TOPLAS06.pdf) that copying can be minimised with their
more sophisticated approach to memory management. I have not
confirmed this, though.
> There are certain ways to handle bigger data sets
> (mnesia, et al.) but I have not fully explored these.
> If large data handling is not a problem for you I believe you will be
> pleasantly surprised by Erlang. As you stated error handling and fault
> tolerance are mature within the language and the lightweight processes
> (300 bytes) are great for multicore machines, particularly
I agree strongly. This has to be experienced. It is very easy to
write, and great fun.
> In addition to these great features Erlang has built-in
> functionality for distributed (e.g. across the network) communication.
> The distributed communication is also platform independent (e.g.
> Windows, Linux, etc) By combining fault-tolerance, distributed
> communication, and lightweight processes one could easily enable an
> application to adapt to multiple heterogeneous clusters, desktop
> computers, and grid environments without modifications.
There are also interesting features to support 'live evolution' (i.e.
updating specific modules of the system without stopping it), which
may be handy for long running codes.
> I feel this language is very promising. It will simply take an
> individual or group to fully exploit the languages potential.
I haven't got any useful numbers yet, and I am still concerned that
raw Erlang performance may be too far short of its competition.
So, I am investigating using Erlang for distribution, fault
tolerance, management, and maybe live evolution. My approach is to
build a simple bridge to bits that need performance (which may
cripple the 'live evolution').
I would like to add a plug for Haskell, which is a statically typed
Functional language. It is beginning to show real promise on data-
parallel applications (likely still a year away, but much closer than
I had expected a year ago). Also, a guy (in the UK) has built some
neat Haskell optimisation transformations (for simple problems), with
the result that his *auto-magically transformed* Haskell code is
faster than the original C (and it is horrible to implement the
transformed algorithm in C).
Microsoft are supporting some of this effort, so it may come faster
than I think.
I'd like to understand what experience others may have of functional
programming languages on clusters, if any?
More information about the Beowulf