[Beowulf] Erlang as a language for Beowulf applications (and Haskell)

[G Bulmer]

On 5 Sep 2007, at 20:00, Eric Shook wrote:
> Subject: Re: [Beowulf] Erlang as a language for Beowulf applications
> To: "H.Vidal, Jr." <hvidal at tesseract-tech.com>
>
> I am engaged in similar explorations and I too have found the Erlang
> language as a possible solution to scaling issues. I have been
> experimenting with the language for almost a month so my evidence  
> below
> is not by any means conclusive. With that said in my opinion the
> language has a fatal flaw and that is data handling. According to the
> documentation and my brief experience there are certain instances  
> where
> the data being passed to and from processes / functions is copied and
> not referenced. You can imagine the problems if you are manipulating a
> multigig data set.
You are ahead of me, but, there are claims in the HiPE (High  
Performance Erlang) papers (e.g. http://www.fantasi.se/jesper/ 
publications/TOPLAS06.pdf) that copying can be minimised with their  
more sophisticated approach to memory management. I have not  
confirmed this, though.

> There are certain ways to handle bigger data sets
> (mnesia, et al.) but I have not fully explored these.
>
> If large data handling is not a problem for you I believe you will be
> pleasantly surprised by Erlang. As you stated error handling and fault
> tolerance are mature within the language and the lightweight processes
> (300 bytes) are great for multicore machines, particularly  
> heterogeneous
> machines.
I agree strongly. This has to be experienced. It is very easy to  
write, and great fun.
> In addition to these great features Erlang has built-in
> functionality for distributed (e.g. across the network) communication.
> The distributed communication is also platform independent (e.g.
> Windows, Linux, etc) By combining fault-tolerance, distributed
> communication, and lightweight processes one could easily enable an
> application to adapt to multiple heterogeneous clusters, desktop
> computers, and grid environments without modifications.
There are also interesting features to support 'live evolution' (i.e.  
updating specific modules of the system without stopping it), which  
may be handy for long running codes.
>
> I feel this language is very promising. It will simply take an
> individual or group to fully exploit the languages potential.

I haven't got any useful numbers yet, and I am still concerned that  
raw Erlang performance may be too far short of its competition.
So, I am investigating using Erlang for distribution, fault  
tolerance, management, and maybe live evolution. My approach is to  
build a simple bridge to bits that need performance (which may  
cripple the 'live evolution').

I would like to add a plug for Haskell, which is a statically typed  
Functional language. It is beginning to show real promise on data- 
parallel applications (likely still a year away, but much closer than  
I had expected a year ago). Also, a guy (in the UK) has built some  
neat Haskell optimisation transformations (for simple problems), with  
the result that his *auto-magically transformed* Haskell code is  
faster than the original C (and it is horrible to implement the  
transformed algorithm in C).

Microsoft are supporting some of this effort, so it may come faster  
than I think.

I'd like to understand what experience others may have of functional  
programming languages on clusters, if any?

Garry